Electronic device for adjusting volume using sound signal output from external object and method thereof

Abstract

According to one embodiment, a processor of a wearable device may link to an external object visible through a display, and may control the display to display a visual object. When the visual object is displayed, in response to receiving a sound signal having a first volume through the microphone, the processor may identify whether the sound signal has been output from an external object based on an image acquired by the camera. Based on identifying that the sound signal has been output from the external object, the processor may adjust the first volume based on a positional relationship between the external object and the visual object so that a second volume corresponding to the visual object may be acquired. Based on the acquired second volume, the processor may output an audio signal corresponding to the visual object through the speaker.

Description

Technical Field

[0001] The present disclosure relates to an electronic device and method for adjusting volume using an acoustic signal output from an external object. Background Art

[0002] In order to provide an enhanced user experience, electronic devices that provide augmented reality (AR) services are being developed, which display computer-generated information associated with external objects in the real world. The electronic device may be a wearable device worn by a user. For example, the electronic device may be AR glasses and / or a head mounted device (HMD). Summary of the invention

[0003] Technical Solution According to an example embodiment, a wearable device may include: a speaker, a microphone, a display, a camera, and a processor. The display may be configured to allow at least a portion of the light incident on a first surface to pass through the display to a second surface opposite to the first surface. The camera may be configured to capture an image. The processor may be configured to control the display to display a visual object associated with an external object visible through the display. The processor may be configured to receive an acoustic signal through a microphone when displaying the visual object. The processor may be configured to identify whether the acoustic signal is output from an external object based on at least an image obtained by the camera in response to receiving an acoustic signal with a first volume. The processor may be configured to obtain a second volume corresponding to the visual object by adjusting the first volume based on a positional relationship between the external object and the visual object based on recognizing that the acoustic signal is output from the external object. The processor may be configured to output an audio signal corresponding to the visual object through a speaker based on the obtained second volume.

[0004] According to an example embodiment, a method of operating a wearable device may include displaying a visual object associated with an external object visible through a display of the wearable device. The method may include receiving an acoustic signal through a microphone of the wearable device while displaying the visual object. The method may include, in response to receiving an acoustic signal having a first volume, identifying whether the acoustic signal is output from the external object based at least on an image obtained through a camera of the wearable device. The method may include obtaining a second volume corresponding to the visual object by adjusting the first volume based on a positional relationship between the external object and the visual object based on identifying that the acoustic signal is output from the external object. The method may include outputting an audio signal corresponding to the visual object through a speaker of the wearable device based on the obtained second volume.

[0005] According to an example embodiment, a wearable device may include: a communication circuit, a speaker, a microphone, a display, a camera, and a processor. The display may be configured to allow at least a portion of light incident on a first surface to pass through the display to a second surface opposite to the first surface. The camera may be configured to capture an image. The processor may be configured to control the display to display a visual object associated with an external object visible through the display. The processor may be configured to receive an acoustic signal having a first volume through a microphone when displaying the visual object. The processor may be configured to identify whether the acoustic signal is output from an external object for reproducing an audio signal in response to receiving the acoustic signal. The processor may be configured to obtain a second volume corresponding to the visual object by adjusting the first volume based on a positional relationship between the external object and the visual object based on identifying that the acoustic signal is output from the external object. The processor may be configured to output at least a portion of the audio signal received from the external object through the communication circuit based on the obtained second volume.

[0006] According to an example embodiment, a method of operating a wearable device may include: displaying a visual object associated with an external object visible through a display of the wearable device. The method may include: receiving an acoustic signal having a first volume through a microphone of the wearable device while displaying the visual object. The method may include: in response to receiving the acoustic signal, identifying whether the acoustic signal is output from the external object for reproducing an audio signal. The method may include, based on identifying that the acoustic signal is output from the external object, obtaining a second volume corresponding to the visual object by adjusting the first volume based on a positional relationship between the external object and the visual object. The method may include outputting at least a portion of an audio signal received from the external object through a communication circuit of the wearable device through a speaker of the wearable device based on the obtained second volume. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent through the following detailed description in conjunction with the accompanying drawings, in which: Figure 1 is a block diagram illustrating example electronic devices in a network environment according to various embodiments.

[0008] Figure 2 is a diagram of an example of a user interface (UI) provided by a wearable device according to an embodiment.

[0009] Figure 3a is a stereoscopic diagram showing an example of a wearable device according to an embodiment.

[0010] Figure 3b is a perspective view showing an example of one or more hardware provided in a wearable device according to an embodiment.

[0011] Figure 4a and Figure 4b is a stereoscopic diagram showing an example of the appearance of a wearable device according to an embodiment.

[0012] Figure 5a and Figure 5b is a diagram showing an example of an operation of a wearable device recognizing a position of an external object and a position of a visual object based on a virtual space according to an embodiment.

[0013] Figure 6a and Figure 6b is a diagram showing an example of a UI provided by a wearable device for adjusting the volume of an audio signal output through a speaker according to an embodiment.

[0014] Figure 7a , Figure 7b and Figure 7c is a diagram illustrating an example of an operation for adjusting the volume of an audio signal through a wearable device according to an embodiment.

[0015] Figure 8 is a diagram showing an example of an operation of a wearable device displaying a visual object associated with an external electronic device according to an embodiment.

[0016] Fig. 9 is a flowchart illustrating an example operation of a wearable device according to an embodiment.

[0017] Fig.10 is a flowchart illustrating example operations performed by a wearable device based on reception of an acoustic signal according to an embodiment.

[0018] Fig.11 is a flowchart illustrating an example operation of a wearable device according to an embodiment.

[0019] Fig.12 is a signal flow diagram illustrating example operations of a wearable device and an external electronic device according to an embodiment. DETAILED DESCRIPTION

[0020] Hereinafter, various exemplary embodiments of the present disclosure will be described in more detail with reference to the accompanying drawings.

[0021] The various example embodiments and terms used herein are not intended to limit the technical features described herein to a specific embodiment, but should be understood to include various modifications, equivalents or substitutions of the embodiment. With respect to the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more items unless there is a clear different indication in the relevant context. In the present disclosure, each phrase such as "A or B", "at least one of A and B", "at least one of A, B and C", "at least one of A, B or C" and "at least one of A, B or C" may include any of these phrases, or all possible combinations thereof. Terms such as "the first", "the second" or "first" or "second" may be simply used to distinguish a corresponding component from another corresponding component, and are not limited to other aspects (e.g., importance or order). When some (e.g., first) components are referred to as having or not having the term "functionally" or "communicatively", "coupled" or "connected" in another (e.g., second) component, this means that some components may be connected directly (e.g., wired), wirelessly, or through a third component.

[0022] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit system. A module may be a minimum unit or a part of a component of an overall configuration, or a component that performs one or more functions. For example, according to an embodiment, a module may be implemented in the form of an application specific integrated circuit (ASIC).

[0023] Figure 1 1 is a block diagram illustrating an example electronic device 101 in a network environment 100 according to various embodiments. Figure 1, the electronic device 101 in the network environment 100 may communicate with the electronic device 102 via the first network 198 (e.g., a short-range wireless communication network), or communicate with at least one of the electronic device 104 or the server 108 via the second network 199 (e.g., a long-range wireless communication network). According to an embodiment, the electronic device 101 may communicate with the electronic device 104 via the server 108. According to an embodiment, the electronic device 101 may include a processor 120, a memory 130, an input module 150, a sound output module 155, a display module 160, an audio module 170, a sensor module 176, an interface 177, a connection terminal 178, a haptic module 179, a camera module 180, a power management module 188, a battery 189, a communication module 190, a user identification module (SIM) 196, or an antenna module 197. In an embodiment, at least one of the above components (e.g., the connection terminal 178) may be omitted from the electronic device 101, or one or more other components may be added to the electronic device 101. In embodiments, some of the above-described components (eg, sensor module 176, camera module 180, or antenna module 197) may be implemented as a single integrated component (eg, display module 160).

[0024] The processor 120 may run, for example, software (e.g., program 140) to control at least one other component (e.g., hardware component or software component) of the electronic device 101 connected to the processor 120, and may perform various data processing or calculations. According to an embodiment, as at least part of the data processing or calculation, the processor 120 may store a command or data received from another component (e.g., sensor module 176 or communication module 190) in the volatile memory 132, process the command or data stored in the volatile memory 132, and store the resultant data in the non-volatile memory 134. According to an embodiment, the processor 120 may include a main processor 121 (e.g., a central processing unit (CPU) or an application processor (AP)) or an auxiliary processor 123 (e.g., a graphics processing unit (GPU), a neural processing unit (NPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that is independent of or combined with the main processor 121 in operation. For example, when the electronic device 101 includes a main processor 121 and an auxiliary processor 123, the auxiliary processor 123 may be adapted to consume less power than the main processor 121, or adapted to be dedicated to a specific function. The auxiliary processor 123 may be implemented separately from the main processor 121, or as part of the main processor 121.

[0025] When the main processor 121 is in an inactive (e.g., sleep) state, the auxiliary processor 123 (rather than the main processor 121) may control at least some of the functions or states related to at least one component among the components of the electronic device 101 (e.g., the display module 160, the sensor module 176, or the communication module 190), or when the main processor 121 is in an active state (e.g., running an application), the auxiliary processor 123 may control at least some of the functions or states related to at least one component among the components of the electronic device 101 (e.g., the display module 160, the sensor module 176, or the communication module 190) together with the main processor 121. According to an embodiment, the auxiliary processor 123 (e.g., an image signal processor or a communication processor) may be implemented as part of another component (e.g., a camera module 180 or a communication module 190) that is functionally related to the auxiliary processor 123. According to an embodiment, the auxiliary processor 123 (e.g., a neural processing unit) may include a hardware structure dedicated to artificial intelligence model processing. The artificial intelligence model may be generated through machine learning. For example, such learning may be performed by the electronic device 101 where the artificial intelligence is executed or via a separate server (e.g., server 108). The learning algorithm may include, but is not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, for example. The artificial intelligence model may include multiple artificial neural network layers. The artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or a deep Q network or a combination of two or more thereof, but is not limited thereto. Additionally or optionally, the artificial intelligence model may include a software structure in addition to a hardware structure.

[0026] The memory 130 may store various data used by at least one component of the electronic device 101 (e.g., the processor 120 or the sensor module 176). The various data may include, for example, software (e.g., the program 140) and input data or output data for commands related thereto. The memory 130 may include a volatile memory 132 or a nonvolatile memory 134.

[0027] The program 140 may be stored as software in the memory 130 , and may include, for example, an operating system (OS) 142 , middleware 144 , or applications 146 .

[0028] The input module 150 may receive commands or data to be used by other components (e.g., the processor 120) of the electronic device 101 from outside (e.g., a user) of the electronic device 101. The input module 150 may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus).

[0029] The sound output module 155 can output sound signals to the outside of the electronic device 101. The sound output module 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as playing multimedia or playing records. The receiver can be used to receive incoming calls. Depending on the embodiment, the receiver can be implemented as a separate part from the speaker, or as part of the speaker.

[0030] The display module 160 may visually provide information to the outside of the electronic device 101 (e.g., a user). The display device 160 may include, for example, a display, a holographic device, or a projector, and a control circuit for controlling a corresponding one of the display, the holographic device, and the projector. According to an embodiment, the display module 160 may include a touch sensor adapted to detect a touch or a pressure sensor adapted to measure the strength of a force caused by a touch.

[0031] The audio module 170 may convert sound into an electrical signal, or vice versa. According to an embodiment, the audio module 170 may obtain sound via the input module 150, or output sound via the sound output module 155 or an earphone of an external electronic device (e.g., electronic device 102) directly (e.g., wired) or wirelessly connected to the electronic device 101.

[0032] The sensor module 176 may detect an operating state (e.g., power or temperature) of the electronic device 101 or an environmental state (e.g., a state of a user) outside the electronic device 101, and then generate an electrical signal or a data value corresponding to the detected state. According to an embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illumination sensor.

[0033] The interface 177 may support one or more specific protocols to be used to connect the electronic device 101 directly (e.g., wired) or wirelessly with an external electronic device (e.g., the electronic device 102). According to an embodiment, the interface 177 may include, for example, a high-definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

[0034] The connection end 178 may include a connector, wherein the electronic device 101 can be physically connected to an external electronic device (e.g., the electronic device 102) via the connector. According to an embodiment, the connection end 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0035] The haptic module 179 may convert the electric signal into mechanical stimulation (eg, vibration or motion) or electric stimulation that can be recognized by the user via his sense of touch or kinesthetic sense. According to an embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electric stimulator.

[0036] The camera module 180 may capture still images or moving images. According to an embodiment, the camera module 180 may include one or more lenses, an image sensor, an image signal processor, or a flash.

[0037] The power management module 188 may manage power supply to the electronic device 101. According to an embodiment, the power management module 188 may be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0038] The battery 189 may power at least one component of the electronic device 101. According to an embodiment, the battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0039] The communication module 190 may support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (e.g., electronic device 102, electronic device 104, or server 108), and perform communication via the established communication channel. The communication module 190 may include one or more communication processors capable of operating independently from the processor 120 (e.g., an application processor (AP)) and supporting direct (e.g., wired) communication or wireless communication. According to an embodiment, the communication module 190 may include a wireless communication module 192 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules can communicate with an external electronic device via a first network 198 (e.g., a short-range communication network such as Bluetooth, Wireless Fidelity (Wi-Fi) Direct, or Infrared Data Association (IrDA)) or a second network 199 (e.g., a long-range communication network such as a traditional cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or a wide area network (WAN))). These various types of communication modules may be implemented as a single component (e.g., a single chip), or may be implemented as multiple components (e.g., multiple chips) separated from each other. The wireless communication module 192 may identify and authenticate the electronic device 101 in a communication network (such as the first network 198 or the second network 199) using user information (e.g., an International Mobile Subscriber Identity (IMSI)) stored in the user identification module 196.

[0040] The wireless communication module 192 can support 5G networks after 4G networks and next-generation communication technologies (e.g., new radio (NR) access technologies). NR access technologies can support enhanced mobile broadband (eMBB), massive machine type communications (mMTC), or ultra-reliable low-latency communications (URLLC). The wireless communication module 192 can support high-frequency bands (e.g., millimeter wave bands) to achieve, for example, high data transmission rates. The wireless communication module 192 can support various technologies for ensuring performance on high-frequency bands, such as, for example, beamforming, massive multiple-input multiple-output (massive MIMO), full-dimensional MIMO (FD-MIMO), array antennas, analog beamforming, or massive antennas. The wireless communication module 192 can support various requirements specified in the electronic device 101, an external electronic device (e.g., electronic device 104), or a network system (e.g., a second network 199). According to an embodiment, the wireless communication module 192 may support a peak data rate for implementing eMBB (e.g., 20 Gbps or greater), loss coverage for implementing mMTC (e.g., 164 dB or less), or U-plane delay for implementing URLLC (e.g., 0.5 ms or less for each of the downlink (DL) and uplink (UL), or 1 ms or less for a round trip).

[0041] The antenna module 197 may transmit or receive a signal or power to or from the outside of the electronic device 101 (e.g., an external electronic device). According to an embodiment, the antenna module 197 may include an antenna including a radiation element formed of a conductive material or a conductive pattern formed in or on a substrate (e.g., a printed circuit board (PCB)). According to an embodiment, the antenna module 197 may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication scheme used in a communication network (such as the first network 198 or the second network 199) may be selected from the plurality of antennas by, for example, the communication module 190 (e.g., the wireless communication module 192). A signal or power may then be transmitted or received between the communication module 190 and the external electronic device via the selected at least one antenna. According to an embodiment, another component (e.g., a radio frequency integrated circuit (RFIC)) other than the radiation element may be additionally formed as a part of the antenna module 197.

[0042] According to various embodiments, the antenna module 197 may form a mmWave antenna module. According to an embodiment, the mmWave antenna module may include a printed circuit board, a radio frequency integrated circuit (RFIC), and a plurality of antennas (e.g., array antennas), wherein the RFIC is disposed on a first surface (e.g., bottom surface) of the printed circuit board, or adjacent to the first surface and capable of supporting a designated high frequency band (e.g., mmWave band), and the plurality of antennas are disposed on a second surface (e.g., top surface or side surface) of the printed circuit board, or adjacent to the second surface and capable of transmitting or receiving signals of the designated high frequency band.

[0043] At least some of the above components may be connected to each other via an inter-peripheral communication scheme (e.g., a bus, a general purpose input output (GPIO), a serial peripheral interface (SPI), or a mobile industry processor interface (MIPI)) and communicatively transmit signals (e.g., commands or data) therebetween.

[0044] According to an embodiment, a command or data may be sent or received between the electronic device 101 and the external electronic device 104 via the server 108 connected to the second network 199. Each of the electronic device 102 or the electronic device 104 may be a device of the same type as the electronic device 101, or a device of a different type from the electronic device 101. According to an embodiment, all or some operations to be executed in the electronic device 101 may be executed in one or more of the external electronic device 102, the external electronic device 104, or the server 108. For example, if the electronic device 101 should automatically execute a function or service or should execute a function or service in response to a request from a user or another device, the electronic device 101 may request the one or more external electronic devices to execute at least part of the function or service instead of executing the function or service, or the electronic device 101 may request the one or more external electronic devices to execute at least part of the function or service in addition to executing the function or service. The one or more external electronic devices receiving the request may execute at least part of the requested function or service, or execute another function or another service related to the request, and transmit the result of the execution to the electronic device 101. The electronic device 101 may provide the result as at least a partial reply to the request with or without further processing the result. To this end, cloud computing technology, distributed computing technology, mobile edge computing (MEC) technology, or client-server computing technology, for example, may be used. The electronic device 101 may use, for example, distributed computing or mobile edge computing to provide ultra-low latency services. In an embodiment, the external electronic device 104 may include an Internet of Things (IoT) device. The server 108 may be an intelligent server using machine learning and / or neural networks. According to an embodiment, the external electronic device 104 or the server 108 may be included in the second network 199. The electronic device 101 may be applied to intelligent services (e.g., smart homes, smart cities, smart cars, or healthcare) based on 5G communication technology or IoT-related technologies.

[0045] The electronic device according to various embodiments may be one of various types of electronic devices. The electronic device may include, for example, a portable communication device (e.g., a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, a home appliance, etc. According to an embodiment of the present disclosure, the electronic device is not limited to those electronic devices described above.

[0046] It should be understood that the various embodiments of the present disclosure and the terms used therein are not intended to limit the technical features set forth herein to specific embodiments, but include various changes, equivalent forms or alternative forms for the corresponding embodiments. For the description of the accompanying drawings, similar reference numerals may be used to refer to similar or related elements. It will be understood that the nouns in the singular form corresponding to the term may include one or more things unless the relevant context clearly indicates otherwise. As used herein, each of the phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C" and "at least one of A, B or C" may include any one or all possible combinations of the items listed together with the corresponding one of the multiple phrases. As used herein, terms such as "1st" and "2nd" or "first" and "second" may be used to simply distinguish the corresponding component from another component, and do not limit the component in other aspects (e.g., importance or order). It will be understood that if an element (e.g., a first element) is referred to as being “coupled with another element (e.g., a second element)”, “coupled to another element (e.g., the second element)”, “connected with another element (e.g., the second element)” or “connected to another element (e.g., the second element)” with or without the terms “operably” or “communicatively” being used, the element may be directly (e.g., wired) connected to the other element, wirelessly connected to the other element, or connected to the other element via a third element.

[0047] As used in connection with various embodiments of the present disclosure, the term "module" may include a unit implemented in hardware, software, or firmware, or any combination thereof, and may be used interchangeably with other terms (e.g., "logic," "logic block," "portion," or "circuit"). A module may be a single integrated component adapted to perform one or more functions or a minimum unit or portion of the single integrated component. For example, according to an embodiment, a module may be implemented in the form of an application specific integrated circuit (ASIC).

[0048] The various embodiments described herein may be implemented as software (e.g., program 140) including one or more instructions stored in a storage medium (e.g., internal memory 136 or external memory 138) that can be read by a machine (e.g., electronic device 101). For example, under the control of a processor, a processor (e.g., processor 120) of the machine (e.g., electronic device 101) may call at least one of the one or more instructions stored in the storage medium and execute the at least one instruction with or without the use of one or more other components. This enables the machine to operate to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. A machine-readable storage medium may be provided in the form of a non-transitory storage medium. Among them, the term "non-transitory" storage medium is a tangible device and may not include a signal (e.g., an electromagnetic wave), but the term does not distinguish between data being semi-permanently stored in the storage medium and data being temporarily stored in the storage medium.

[0049] According to an embodiment, the method according to various embodiments of the present disclosure may be included and provided in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be published in the form of a machine-readable storage medium (e.g., a compact disk read-only memory (CD-ROM)), or may be published online (e.g., downloaded or uploaded) via an application store (e.g., Play StoreTM), or may be distributed (e.g., downloaded or uploaded) directly between two user devices (e.g., smart phones). If published online, at least part of the computer program product may be temporarily generated, or at least part of the computer program product may be at least temporarily stored in a machine-readable storage medium (such as a manufacturer's server, an application store's server, or a memory of a forwarding server).

[0050] According to various embodiments, each of the above-mentioned components (e.g., a module or a program) may include a single entity or multiple entities, and some of the multiple entities may be separately arranged in different components. According to various embodiments, one or more of the above-mentioned components may be omitted, or one or more other components may be added. Alternatively or additionally, multiple components (e.g., a module or a program) may be integrated into a single component. In this case, according to various embodiments, the integrated component may still perform the one or more functions of each of the multiple components in the same or similar manner as a corresponding one of the multiple components performing one or more functions before integration. According to various embodiments, the operations performed by a module, a program or another component may be performed sequentially, in parallel, repeatedly or in a heuristic manner, or one or more of the operations may be run or omitted in a different order, or one or more other operations may be added.

[0051] Figure 2 is a diagram showing an example of a user interface (UI) provided by the wearable device 101 according to an embodiment. According to an embodiment, the wearable device 101 may have the form of glasses that can be worn on a body part of a user (e.g., a head). The wearable device 101 may include a head-mounted display (HMD). For example, the housing of the wearable device 101 may include a flexible material such as rubber and / or silicon, which is shaped to closely adhere to a portion of the user's head (e.g., a portion of the face covering both eyes). For example, the housing of the wearable device 101 may include one or more straps that can be wrapped around the user's head and / or one or more temples that can be attached to the ears of the head. According to an embodiment, reference may be made to Figure 3a , Figure 3b , Figure 4a and / or Figure 4b The form of the wearable device 101 is described.

[0052] Figure 2 A block diagram illustrating an example configuration of a wearable device 101 according to an embodiment is included. Figure 2 The wearable device 101 can be connected to the Figure 1 The electronic device 101. Figure 2According to an embodiment, the wearable device 101 may include at least one of a processor (e.g., including a processing circuit) 120, a memory 130, a speaker 210, a microphone 220, a display 230, a camera 240, and / or a communication circuit 250. The processor 120, the memory 130, the speaker 210, the microphone 220, the display 230, the camera 240, and the communication circuit 250 may be electronically and / or operably coupled to each other through electronic components such as a communication bus 205. Operably coupled hardware may, for example, refer to a direct connection or an indirect connection between hardware established by wire or wirelessly to ensure that specific hardware in the hardware is controlled by another hardware and / or other hardware is controlled by specific hardware. The type and / or number of hardware included in the wearable device 101 is not limited to Figure 2 .

[0053] According to an embodiment, the processor 120 of the wearable device 101 may include a circuit for processing data based on one or more instructions. For example, a hardware component for processing data may include an arithmetic and logic unit (ALU), a floating point unit (FPU), a field programmable gate array (FPGA), and / or a central processing unit (CPU). The number of processors 120 may be one or more. For example, the processor 120 may have a structure of a multi-core processor such as a dual-core, quad-core, or hexa-core processor. Figure 2 The processor 120 may include Figure 1 The processor 120.

[0054] According to an embodiment, the memory 130 of the wearable device 101 may include hardware for storing data and / or instructions input to and / or output from the processor 120. For example, the memory 130 may include a volatile memory such as a random access memory (RAM), and / or a non-volatile memory such as a read-only memory (ROM). For example, the volatile memory may include at least one of a dynamic RAM (DRAM), a static RAM (SRAM), a cache RAM, or a pseudo SRAM (PSRAM). For example, the non-volatile memory may include at least one of a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), a flash memory, a hard disk, an optical disk, or an embedded multimedia card (eMMC). Figure 2 The memory 130 may include Figure 1 Memory 130.

[0055] One or more instructions indicating calculations and / or operations to be performed on data by the processor 120 may be stored in the memory 130. A set of one or more instructions may be referred to as firmware, an operating system, a process, a routine, a subroutine, and / or an application. For example, when executing a set of multiple instructions deployed in the form of an operating system, firmware, a driver, and / or an application, the wearable device 101 and / or the processor 120 may perform Figures 9 to 12 At least one operation in. Hereinafter, the fact that an application is installed in the wearable device 101 may indicate that one or more instructions provided in the form of an application are stored in the memory 130 of the wearable device 101, and may, for example, refer to one or more applications stored in an executable format (e.g., a file with an extension specified by the operating system of the wearable device 101) by the processor 120 of the wearable device.

[0056] According to an embodiment, the speaker 210 of the wearable device 101 may output an audio signal. The number of speakers 210 included in the wearable device 101 may be one or more. The speaker 210 may be controlled by the processor 120 of the wearable device 101. In a state controlled by the processor 120, the speaker 210 may output an audio signal stored in the wearable device 101 or transmitted to the wearable device 101. For example, the processor 120 may identify compressed data from an audio signal based on an audio codec such as MPEG-1 Audio Layer-3 (mp3). The processor 120 may generate an electrical signal based on the data for outputting sound through the speaker 210.

[0057] According to an embodiment, the microphone 220 of the wearable device 101 may output an electrical signal indicating atmospheric vibration. The number of microphones 220 included in the wearable device 101 may be one or more. The electrical signal output from the microphone 220 may be sent to the processor 120. The processor 120 may obtain an audio signal for reconfiguring the vibration from the electrical signal using the speaker 210.

[0058] According to an embodiment, the display 230 of the wearable device 101 can output visual information (e.g., Figure 2 , Figure 5a , Figure 6a to Figure 6b and / or Figure 8At least one of the screens of the wearable device 101). The number of displays 230 included in the wearable device 101 may be one or more. For example, the display 230 may be controlled by the processor 120 and / or a graphics processing unit (GPU) to output visual information to the user. The display 230 may include a flat panel display (FPD) and / or an electronic paper. The FPD may include a liquid crystal display (LCD), a plasma display panel (PDP), a digital mirror device (DMD), one or more light emitting diodes (LEDs) and / or micro-LEDs. The LED may include an organic LED (OLED). In an embodiment where the display 230 includes an LCD, the display 230 may further include a light source (e.g., a backlight) for emitting light toward the LCD. The light source may be omitted in an embodiment where the display 230 includes an OLED.

[0059] In an embodiment, at least a portion of the display 230 may allow light to pass through. The wearable device 101 may provide a user with a user experience associated with augmented reality by providing a combination of light output through the display 230 and light passing through the display 230. Figure 3a and Figure 3b An example of a structure of a wearable device 101 for providing a user experience related to augmented reality is described in more detail. In an embodiment, the wearable device 101 may have a structure in which the display 230 overlaps the entire field of view (FoV) of the user when worn on a body part of the user (such as the head). The display 230 may block the light around the wearable device 101 from being emitted to the user's eyes in this state. For example, the wearable device 101 may use the display 230 to provide the user with a user experience related to virtual reality. Figure 4a and Figure 4b An example of a structure of a wearable device 101 for providing a user experience related to virtual reality is described in more detail.

[0060] According to an embodiment, the camera 240 of the wearable device 101 may include one or more optical sensors (e.g., a charge coupled device (CCD) sensor, a complementary metal oxide semiconductor (CMOS) sensor) that generate an electrical signal indicating the color and / or brightness of light. The plurality of optical sensors included in the camera 240 may be arranged in the form of a two-dimensional array. The camera 240 may generate an image including a plurality of pixels arranged in a two-dimensional form by substantially simultaneously obtaining an electrical signal of each of the plurality of optical sensors, the pixels corresponding to the light reaching the optical sensor of the two-dimensional array. For example, photo data captured using the camera 240 may indicate an image obtained from the camera 240. For example, video data captured using the camera 240 may refer to a sequence of a plurality of images obtained from the camera 240, for example, according to a specified frame rate. The wearable device 101 according to an embodiment may further include a flashlight that is disposed in a direction in which the camera 240 receives light and is used to output light in that direction. The number of cameras 240 included in the wearable device 101 may be one or more.

[0061] In an embodiment, the FoV of the camera 240 may include an area formed based on a viewing angle at which a lens of the camera 240 can receive light, and may correspond to an area corresponding to an image generated by the camera 240. Hereinafter, a subject and / or an external object refers to an object included in the FoV of the camera 240 and distinguished from the wearable device 101. In an embodiment, as Figure 2 As shown by FoV 270 , the FoV of camera 240 may at least partially match the visual environment of the user via display 230 .

[0062] According to an embodiment, the communication circuit 250 of the wearable device 101 may include hardware to support sending and / or receiving electrical signals between the wearable device 101 and an external electronic device (eg, an external object 284). Figure 2 As an example of an external electronic device, an external object 284 as a speaker is shown. For example, the communication circuit 250 may include at least one of a modem, an antenna, and an optical / electrical (O / E) converter. The communication circuit 250 may support transmission and / or reception of electrical signals based on various types of protocols such as Ethernet, local area network (LAN), wide area network (WAN), wireless fidelity (WiFi), Bluetooth, Bluetooth low energy (BLE), ZigBee, long term evolution (LTE), and 5G new radio (NR). According to an embodiment, the following will refer to Figure 8 The operations performed by the wearable device 101 based on the communication between the external electronic device and the wearable device 101 are described in more detail.

[0063] Although not shown, the wearable device 101 may include other output devices for outputting information in a form other than a visual form or an audible form. For example, the wearable device 101 may include a motor for providing tactile feedback based on vibration. Meanwhile, although shown based on different blocks, the embodiments are not limited thereto, and Figure 2 Part of the hardware components shown in FIG. 1 (eg, processor 120 , memory 130 , and at least a portion of communication circuit 250 ) may be included in a single integrated circuit such as a system on a chip (SoC).

[0064] As described above, according to an embodiment, the wearable device 101 may include one or more hardware for providing a user experience based on augmented reality (AR) (or mixed reality (MR)) and / or virtual reality (VR). Figure 2 , shows an example of a field of view (FoV) 270 visible to a user through a display 230 of a wearable device 101. In an embodiment of the wearable device 101 including a display 230 (the display 230 is configured to allow light incident on a first surface to pass through a second surface opposite the first surface), the FoV 270 can be formed by light transmitted through the display 230 to both eyes of the user. The wearable device 101 can provide a user experience based on augmented reality by adding virtual objects superimposed on the FoV 270 and tangible objects within the FoV 270 using the display 230.

[0065] Reference Figure 2 According to an embodiment, the wearable device 101 may allow a user wearing the wearable device 101 to recognize an external object 280 included in the FoV 270 displayed through the display 230. Figure 2 , external objects 282 and 284 are shown as examples of external objects 280 included in the FoV 270. External objects 280 such as external objects 282 and 284 may indicate objects recognized by the wearable device 101.

[0066] According to an embodiment, the wearable device 101 may use the camera 240 to identify the FoV 270 and / or an external object 280 adjacent to the wearable device 101. The external object 280 may be referred to as a subject and / or a tangible object. For example, the camera 240 may obtain or capture an image arranged based on the direction of the first surface of the display 230 and including light incident on the first surface. For example, the optical axis of the camera included in the wearable device 101 may match the direction of the first surface.

[0067] In an embodiment, the screen displayed by the wearable device 101 in the FoV 270 using the display 230 may include enhanced or annotated information based on the environment displayed to the user. For example, the wearable device 101 may use the display 230 to display one or more visual objects (e.g., visual object 290) by overlapping within the FoV 270. The visual object 290 may be referred to as a virtual (or imaginary) object. The visual object 290 may be displayed in the display 230 based on an application executed by the processor 120 of the wearable device 101 and / or input received from a user wearing the wearable device 101. The visual object 290 may be displayed by the wearable device 101 to support interaction between the wearable device 101 and the user wearing the wearable device 101.

[0068] According to an embodiment, the wearable device 101 may support interaction between an external electronic device (e.g., external object 284 as a speaker) different from the wearable device 101 and a user using a visual object (e.g., visual object 290) displayed in the FoV 270 by the display 230. For example, the visual object 290 may represent an external electronic device of the user connected to the wearable device 101 and / or the external electronic device through a network. For example, the connection between the wearable device 101 and the external electronic device may be established based on a network service such as a video conference. Figure 2 In order to represent the external electronic device connected through the network service and / or the user of the external electronic device, the wearable device 101 may display a visual object 290 in the display 230, such as an avatar representing the external electronic device or the user of the external electronic device. For example, the visual object 290 may be controlled by the user's input to the external electronic device, which is indicated by a signal received from the external electronic device. For example, in response to receiving a signal indicating a gesture of the user to the external electronic device, the wearable device 101 may change the display of the visual object 290.

[0069] According to an embodiment, the wearable device 101 may add information in the sound heard by the user using the speaker 210, similar to adding information to the FoV 270 using the display 230. The speaker 210 may be disposed adjacent to at least one of the two ears of the user. In an embodiment, the speaker 210 may have a shape for keeping the user's external auditory canal open. Based on the speaker 210 opening the external auditory canal, the sound generated by the external object 280 may be transmitted to the user. For example, when the wearable device 101 outputs an audio signal using the speaker 210, the user can hear all the audio signals and the sound generated from the external object 280. The embodiment is not limited thereto, and the speaker 210 may have a form for enhancing sound insulation (e.g., a form covering the user's ears, such as headphones).

[0070] According to an embodiment, the wearable device 101 may output an audio signal using the speaker 210 to provide a combination of a sound and an audio signal output from the external object 280 to the user's ears. In order to balance the sound and the audio signal, the wearable device 101 may adjust the volume of the audio signal output from the speaker 210 based on the volume of the sound. The volume may include the amplitude, average intensity, maximum intensity, and / or minimum intensity of the sound and / or audio signal. For example, the wearable device 101 may adjust the volume of the audio signal based on the volume of the sound output from the external object 280. The wearable device 101 may obtain the sound output from the external object 280 using the microphone 220. The wearable device 101 may adjust the second volume of the audio signal sent to the user through the speaker 210 based on the first volume of the sound.

[0071] According to an embodiment, the wearable device 101 may adjust the volume of the audio signal output through the speaker 210 based on the external object 280 adjacent to the wearable device 101 and the visual object 290 displayed through the display 270. In an embodiment in which the wearable device 101 represents an external electronic device connected to the wearable device 101 and / or a user of the external electronic device based on the visual object 290, the wearable device 101 may output the audio signal received from the external electronic device based at least on the volume output from the external object 280.

[0072] In an embodiment, the wearable device 101 may identify the source of the sound received through the microphone 220 based on the image obtained from the camera 240, so as to adjust the volume of the audio signal using the sound received through the microphone 220. For example, the wearable device 101 may identify whether the sound received through the microphone 220 is output from the external object 280. In the case where the sound is output from the external object 280, the wearable device 101 may adjust the volume of the audio signal based on the volume of the sound. For example, the wearable device 101 may adjust the volume of the audio signal by comparing the position of the external object 280 and the position of the visual object 290 displayed through the FoV 270. According to an embodiment, the following will refer to Figure 5a and Figure 5b and / or Figure 7a , Figure 7b and Figure 7c The operation of the wearable device 101 comparing the position of the external object 280 and the position of the visual object 290 is described in more detail. According to an embodiment, in a state where the wearable device 101 adjusts the volume of the audio signal to be output through the speaker 210 based on the sound received through the microphone 220, the following will refer to Figure 6a , Figure 6b and / or Figure 8 An example of the UI displayed through the display 230 is described in more detail.

[0073] As described above, according to an embodiment, in a state where the user views all real external objects 280 and virtual visual objects 290 through the display 230 of the wearable device 101, the wearable device 101 can optimize the volume of the audio signal output through the speaker 210 based on the volume of the sound output from the external object 280. Since the wearable device 101 optimizes the volume of the audio signal, the user can be prevented / prohibited from redundantly adjusting the volume of the speaker 210. Since the wearable device 101 optimizes the volume of the audio signal, the user can focus on the augmented reality provided by the wearable device 101 without adjusting the volume of the speaker 210. For example, the wearable device 101 can provide the user with immersive augmented reality and / or virtual reality.

[0074] In the following, reference is made to Figure 3a and Figure 3b and / or Figure 4a and Figure 4b , according to an embodiment, examples of form factors of the wearable device 101 will be described.

[0075] Figure 3a is a stereoscopic diagram showing an example of a wearable device 101 according to an embodiment. Figure 3b is a stereoscopic diagram showing an example of one or more hardware provided in the wearable device 101 according to an embodiment. Figure 3a and Figure 3b The wearable device 101 may be Figure 1 and Figure 2 An example of a wearable device 101. Figure 3a According to an embodiment, the wearable device 101 may include at least one display 230 and a frame 300 supporting the at least one display 230. The at least one display 230 may be Figure 2 An example of display 230 is shown.

[0076] According to an embodiment, the wearable device 101 may be worn on a part of a user's body. The wearable device 101 may provide augmented reality (AR), virtual reality (VR), or mixed reality (MR) that is a mixture of augmented reality and virtual reality to the user wearing the wearable device 101. For example, the wearable device 101 may respond to a user's Figure 3b The designated gesture of the user obtained by the motion recognition camera 240-2 is displayed on at least one display 230. Figure 3b A virtual reality image is provided by at least one of optical devices 382 and 384.

[0077] According to an embodiment, at least one display 230 may provide visual information to the user. For example, at least one display 230 may include a transparent or translucent lens. At least one display 230 may include a first display 230-1 and / or a second display 230-2 spaced apart from the first display 230-1. For example, the first display 230-1 and the second display 230-2 may be respectively disposed in positions corresponding to the left eye and the right eye of the user.

[0078] Reference Figure 3b , at least one display 230 can provide the user with visual information transmitted from external light and other visual information distinguished from the visual information through a lens included in at least one display 230. The lens can be formed based on at least one of a Fresnel lens, a pancake lens, or a multi-channel lens. For example, at least one display 230 may include a first surface 331 and a second surface 332 opposite to the first surface 331. The display area may be formed on the second surface 332 of the at least one display 230. When the user wears the wearable device 101, the external light can be transmitted to the user by being incident on the first surface 331 and passing through the second surface 332. For another example, at least one display 230 can display a virtual reality image that will be coupled to a real screen transmitted by external light. The virtual reality image output from the at least one display 230 can be transmitted to the user's eyes through one or more hardware (e.g., at least one optical device 382 and 384, and / or at least one waveguide 333 and 334) included in the electronic device 101.

[0079] According to an embodiment, the electronic device 101 may include at least one waveguide 333 and 334 that diffracts light transmitted from at least one display 230 and relayed by at least one optical device 382 and 384, and transmits the light to the user. At least one waveguide 333 and 334 may be formed based on at least one of glass, plastic, or polymer. A nano pattern may be formed at least a portion of the outside or inside of at least one waveguide 333 and 334. The nano pattern may be formed based on a polygon and / or grating structure of a curved surface. Light incident to one end of at least one waveguide 333 and 334 may propagate to the other end of at least one waveguide 333 and 344 through the nano pattern. At least one waveguide 333 and 334 may include at least one of the following: at least one diffraction element (e.g., a diffraction optical element (DOE), a hologram optical element (HOE)) and a reflective element (e.g., a reflector). For example, at least one waveguide 333 and 334 may be provided in the wearable device 101 to guide the picture displayed by at least one display 230 to the user's eyes. For example, the picture may be transmitted to the user's eyes based on total internal reflection (TIR) ​​generated in at least one waveguide 333 and 334 .

[0080] The wearable device 101 can analyze the Figure 2 The object included in the real image collected by the camera 240) and a virtual object corresponding to the object to be provided with augmented reality is combined in the analyzed object (for example, Figure 2 The wearable device 101 may include at least one of text and images for various information related to the object included in the real image. The wearable device 101 may analyze the object based on multiple cameras such as a stereo camera. The wearable device 101 may perform time of flight (ToF) and / or simultaneous localization and mapping (SLAM) supported by multiple cameras to analyze the object. A user wearing the wearable device 101 may view the image displayed on the at least one display 230.

[0081] According to an embodiment, the frame 300 may be configured in a physical structure in which the wearable device 101 may be worn on a user. According to an embodiment, the frame 300 may be configured so that when the user wears the wearable device 101, the first display 230-1 and the second display 230-2 may be located at positions corresponding to the left eye and the right eye of the user. The frame 300 may support at least one display 230. For example, the frame 300 may support the first display 230-1 and the second display 230-2 so that they are located at positions corresponding to the left eye and the right eye of the user.

[0082] Reference Figure 3a , when the user wears the wearable device 101, the frame 300 may include an area 320 at least a portion of which contacts a portion of the user's body. For example, the area 320 of the frame 300 that contacts a portion of the user's body may include an area that contacts a portion of the user's nose, a portion of the user's ear, and a portion of the side of the user's face. According to an embodiment, the frame 300 may include a nose pad 310 that contacts a portion of the user's body. When the wearable device 101 is worn by the user, the nose pad 310 may contact a portion of the user's nose. The frame 300 may include a first temple 304 and a second temple 305 that contact another portion of the user's body different from the portion of the user's body.

[0083] For example, the frame 300 may include a first edge 301 surrounding at least a portion of the first display 230-1, a second edge 302 surrounding at least a portion of the second display 230-2, a bridge 303 disposed between the first edge 301 and the second edge 302, a first pad 311 disposed along a portion of an edge of the first edge 301 from one end of the bridge 303, a second pad 312 disposed along a portion of an edge of the second edge 302 from the other end of the bridge 303, a first temple 304 extending from the first edge 301 and fixed to a portion of a wearer's ear, and a second temple 305 extending from the second edge 303 and fixed to a portion of an ear opposite to the above-mentioned ear. The first pad 311 and the second pad 312 may contact a portion of the user's nose, and the first temple 304 and the second temple 305 may contact a portion of the user's face and a portion of the ear. The temples 304 and 305 may be provided by Figure 3b The hinge units 306 and 307 of the frame 300 are rotatably connected to the edge. The first temple 304 can be rotatably connected to the first edge 301 by the first hinge unit 306 provided between the first edge 301 and the first temple 304. The second temple 305 can be rotatably connected to the second edge 305 by the second hinge unit 307 provided between the second edge 302 and the second temple 305. According to an embodiment, the wearable device 101 can use a touch sensor, a grip sensor, and / or a proximity sensor formed on at least a portion of the surface of the frame 300 to recognize an external object (e.g., a user's fingertip) that touches the frame 300 and / or a gesture performed by the external object.

[0084] According to an embodiment, the wearable device 101 may include hardware for performing various functions (for example, the above-mentioned Figure 2 For example, the hardware may include a battery module 370, an antenna module 375, at least one optical device 382 and 384, a speaker 210, a microphone 220, a light module (not shown), and / or a printed circuit board 390. Various hardware may be provided within the frame 300.

[0085] According to an embodiment, the microphone 220 of the wearable device 101 may be disposed on at least a portion of the frame 300 to obtain an acoustic signal. Figure 3b 2 shows a first microphone 220-1 disposed on the nose pad 310, a second microphone 220-2 disposed on the second edge 302, and a third microphone 220-3 disposed on the first edge 301, but the number and arrangement of the microphones 220 are not limited to Figure 3b When the number of microphones 220 included in the wearable device 101 is two or more, the wearable device 101 may identify the direction of the acoustic signal using the plurality of microphones disposed on different portions of the frame 300 .

[0086] According to an embodiment, at least one optical device 382 and 384 may transmit a virtual object transmitted from at least one display 230 to at least one waveguide 333 and 334. For example, at least one optical device 382 and 384 may be a projector. At least one optical device 382 and 384 may be disposed adjacent to at least one display 230, or may be included in at least one display 230 as a part of at least one display 230. According to an embodiment, the wearable device 101 may include a first optical device 382 corresponding to the first display 230-1 and a second optical device 384 corresponding to the second display 230-2. For example, at least one optical device 382 and 384 may include a first optical device 382 disposed on an edge of the first display 230-1 and a second optical device 384 disposed on an edge of the second display 230-2. The first optical device 382 may transmit light output from the first display 230-1 to the first waveguide 333, and the second optical device 384 may transmit light output from the second display 230-2 to the second waveguide 334.

[0087] In an embodiment, the camera 240 (e.g., Figure 2 The camera 240 may include a photographic camera, an eye tracking camera (ETCAM) 240-1, and / or a motion recognition camera 240-2. The photographic camera, the eye tracking camera 240-1, and the motion recognition camera 240-2 may be set in different positions on the frame 300 and may perform different functions. The eye tracking camera 240-1 may output data indicating the line of sight of the user wearing the wearable device 101. For example, the wearable device 101 may detect the line of sight from an image including the user's pupil and obtained by the eye tracking camera 240-1. Although Figure 3b 2 shows an example in which the eye tracking camera 240-1 is set toward the right eye of the user, but the embodiment is not limited thereto, and the eye tracking camera 240-1 may be set toward the left eye of the user alone or toward both eyes.

[0088] In an embodiment, the camera may capture an actual image or background that matches the virtual image to implement augmented reality content or mixed reality content. The camera may capture the location the user sees (e.g., Figure 2The at least one display 230 may display an image of a specific object existing in the FoV 270 of the camera and provide the image to the at least one display 230. The at least one display 230 may display one image overlay information about an actual image or a background including an image of a specific object obtained using a photographic camera and a virtual image provided by at least one optical device 382 and 384. In an embodiment, the photographic camera may be provided on a nose bridge 303 provided between the first edge 301 and the second edge 302.

[0089] The eye tracking camera 240-1 can achieve a more realistic augmented reality by tracking the line of sight of the user wearing the wearable device 101 and matching the line of sight of the user with the visual information provided on the at least one display 230. For example, when the user looks straight ahead, the wearable device 101 can naturally display environmental information related to the front surface of the user on the at least one display 230 at the user's location. The eye tracking camera 240-1 can be configured to capture an image of the user's pupil to determine the user's line of sight. For example, the eye tracking camera 240-1 can receive line of sight detection light reflected from the user's pupil and track the user's line of sight based on the position and movement of the received line of sight detection light. In an embodiment, the eye tracking camera 240-1 can be set in a position corresponding to the left eye and the right eye of the user. For example, the eye tracking camera 240-1 can be set in the first edge 301 and / or the second edge 302 in the direction of the user wearing the wearable device 101.

[0090] The motion recognition camera 240-2 can provide a specific event to a screen set on at least one display 230 by recognizing the movement of all or part of the user's body (such as the user's torso, hand, or face). The motion recognition camera 240-2 can obtain a signal corresponding to the motion by recognizing the user's gesture and provide an indication corresponding to the signal to at least one display 230. The processor can recognize the signal corresponding to the operation and perform a specified function based on the recognition. In an embodiment, the motion recognition camera 240-2 can be set on the first edge 301 and / or the second edge 302.

[0091] The camera 240 included in the wearable device 101 is not limited to the above-mentioned eye tracking camera 240-1 and motion recognition camera 240-2. For example, the wearable device 101 may use a FoV (e.g., Figure 2 The camera 240 is configured to recognize an external object (e.g., Figure 2The wearable device 101 may include a camera 240 (e.g., a face tracking (FT) camera) disposed toward the face to obtain an image including the face of the user wearing the wearable device 101.

[0092] Although not shown, according to an embodiment, the wearable device 101 may further include a light source (e.g., an LED) that emits light to a subject (e.g., the user's eyes, face, and / or an external object within the FoV) captured using the camera 240. The light source may include an infrared wavelength LED. The light source may be provided on at least one of the frame 300 and the hinge units 306 and 307.

[0093] According to an embodiment, the battery module 370 may supply power to the electronic components of the wearable device 101. In an embodiment, the battery module 370 may be disposed in the first temple 304 and / or the second temple 305. For example, the battery module 370 may be a plurality of battery modules 370. A plurality of battery modules 370 may be disposed on each of the first temple 304 and the second temple 305. In an embodiment, the battery module 370 may be disposed at the end of the first temple 304 and / or the second temple 305.

[0094] The antenna module 375 can transmit signals or power to the outside of the wearable device 101, or receive signals or power from the outside. The antenna module 375 can be electrically connected and / or operably connected to Figure 2 In an embodiment, the antenna module 375 may be disposed in the first temple 304 and / or the second temple 305. For example, the antenna module 375 may be disposed adjacent to one surface of the first temple 304 and / or the second temple 305.

[0095] The speaker 210 may output an acoustic signal to the outside of the wearable device 101. The sound output module may be referred to as a speaker. In an embodiment, the speaker 210 may be disposed in the first temple 304 and / or the second temple 305 to be disposed adjacent to the ear of the user wearing the wearable device 101. For example, the speaker 210 may include a second speaker 210-2 disposed adjacent to the right ear of the user by being disposed in the first temple 304, and a first speaker 210-1 disposed adjacent to the left ear of the user by being disposed in the second temple 305.

[0096] The light emitting module (not shown) may include at least one light emitting element. In order to visually provide the user with information about a specific state of the wearable device 101, the light emitting module may emit light of a color corresponding to the specific state, or may emit light in an operation corresponding to the specific state. For example, when charging is required, the wearable device 101 may emit red light during a specific period of time. In an embodiment, the light emitting module may be disposed on the first edge 301 and / or the second edge 302.

[0097] Reference Figure 3b According to an embodiment, the wearable device 101 may include a printed circuit board (PCB) 390. The PCB 390 may be included in at least one of the first temple 304 and the second temple 305. The PCB 390 may include an interposer disposed between at least two sub-PCBs. On the PCB 390, one or more hardware (e.g., Figure 2 The wearable device 101 may include a flexible PCB (FPCB) for interconnecting hardware.

[0098] According to an embodiment, the wearable device 101 may include at least one of a gyro sensor, a gravity sensor, and / or an acceleration sensor for detecting the posture of the wearable device 101 and / or the posture of a body part (e.g., the head) of the user wearing the wearable device 101. Each of the gravity sensor and the acceleration sensor can measure gravity acceleration and / or acceleration based on specified three-dimensional axes (e.g., x-axis, y-axis, and z-axis) that are perpendicular to each other. The gyro sensor can measure the angular velocity of each specified three-dimensional axis (e.g., x-axis, y-axis, and z-axis). At least one of the gravity sensor, the acceleration sensor, and the gyro sensor may be referred to as an inertial measurement unit (IMU). According to an embodiment, the wearable device 101 may recognize motions and / or gestures performed by the user based on the IMU to run or stop a specific function of the wearable device 101.

[0099] Figure 4a and Figure 4b is a stereoscopic diagram showing an example of the appearance of the wearable device 100 according to the embodiment. Figure 4a and / or Figure 4b The wearable device 101 may be Figure 2 According to an embodiment, an example of the appearance of the first surface 410 of the housing of the wearable device 101 may be Figure 4a , and an example of the appearance of the second surface 420 opposite to the first surface 410 can be seen in Figure 4b Shown in.

[0100] Reference Figure 4aAccording to an embodiment, the first surface 410 of the wearable device 101 may have a shape that can be attached to a body part of a user (e.g., a face of the user). Although not shown, the wearable device 101 may further include a strap for fixing to a body part of the user, and / or one or more temples (e.g., Figure 3a to Figure 3b The first temple 304 and / or the second temple 305 in the first surface 410). A first display 230-1 for outputting an image to the left eye of the user's eyes and a second display 230-2 for outputting an image to the right eye of the eyes may be provided on the first surface 410. The wearable device 101 may further include a rubber or silicon filler formed on the first surface 410 and used to prevent and / or reduce interference due to light (e.g., ambient light) different from light emitted from the first display 230-1 and the second display 230-2.

[0101] According to an embodiment, the wearable device 101 may include cameras 240-3 and 240-4 for capturing and / or tracking the eyes of the user adjacent to each of the first display 230-1 and the second display 230-2. Cameras 240-3 and 240-4 may be referred to as ET cameras. According to an embodiment, the wearable device 101 may include cameras 240-5 and 240-6 for capturing and / or recognizing the user's face. Cameras 240-5 and 240-6 may be referred to as FT cameras.

[0102] Reference Figure 4b , a camera (e.g., cameras 240-7, 240-8, 240-9, 240-10, 240-11, 240-12) and / or a sensor (e.g., a depth sensor 430) for obtaining information related to the external environment of the wearable device 101 may be disposed at a position corresponding to Figure 4a For example, cameras 240-7, 240-8, 240-9, and 240-10 may be disposed on the second surface 420 to identify external objects different from the wearable device 101 (e.g., Figure 2 280 in the wearable device 101). For example, the wearable device 101 can use cameras 240-11 and 240-12 to obtain images and / or videos to be transmitted to each eye of the user. The camera 240-11 can be set on the second surface 420 of the wearable device 101 to obtain images to be displayed through the second display 230-2 corresponding to the right eye of the two eyes. The camera 240-12 can be set on the second surface 420 of the wearable device 101 to obtain images to be displayed through the first display 230-1 corresponding to the left eye of the two eyes.

[0103] According to an embodiment, the wearable device 101 may include a depth sensor 430 disposed on the second surface 420 to identify the distance between the wearable device 101 and an external object. The wearable device 101 may use the depth sensor 430 to obtain a FoV (e.g., Figure 2 The invention also provides spatial information (eg, a depth map) of at least a portion of the FoV 270 in the image processing unit.

[0104] Although not shown, a microphone (eg, Figure 2 The microphone 220 in the wearable device 101 may be disposed on the second surface 420 of the wearable device 101. According to an embodiment, the number of the microphones may be one or more.

[0105] As described above, according to an embodiment, the wearable device 101 may have a form factor that is worn on the user's head. The wearable device 101 may provide a user experience based on augmented reality, virtual reality, and / or mixed reality while being worn on the head. Figure 2 In a state where a speaker 210 of the wearable device 101 outputs an audio signal, when the volume of the audio signal is different from the volume of an external sound (e.g., a sound generated from an external object adjacent to the wearable device 101), the user wearing the wearable device 101 may feel heterogeneity. In order to compensate for the heterogeneity, the wearable device 101 may adjust the volume of the audio signal based on the external sound. The wearable device 101 may realistically reproduce the audio signal by adjusting the volume of the audio signal based on the external sound.

[0106] In the following, refer to Figure 5a and Figure 5b According to an embodiment, an operation of the wearable device 101 to adjust the volume of an audio signal based on an external sound will be described.

[0107] Figure 5a and Figure 5b is a diagram showing an operation example of the wearable device 101 according to an embodiment to recognize the position of the external object 282 and the position of the visual object 290 based on the virtual space. Figure 5a and Figure 5b The wearable device 101 may be Figure 2 An example of a wearable device 101. For example, Figure 2 , Figure 3a to Figure 3b and / or Figure 4a to Figure 4b The wearable device 101 may include Figure 5a and Figure 5b Wearable device 101.

[0108] Reference Figure 5a, showing an example state of a user wearing the wearable device 101. Figure 5a In the state of , it is assumed that an external object 282 such as a person exists in a space adjacent to the wearable device 101. According to an embodiment, the wearable device 101 can stimulate the user's vision and / or hearing based on the space including the external object 282.

[0109] According to an embodiment, the wearable device 101 may have a structure for transmitting the sound generated by the external object 282 to the user wearing the wearable device 101. For example, Figure 3a and Figure 3b As shown in the first temple 302 and the second temple 304, the wearable device 101 may have a structure for transmitting sound directly to the external auditory canal of the user wearing the wearable device 101. For example, the wearable device 101 may use a speaker covering the external auditory canal (e.g., Figure 2 The wearable device 101 may include a microphone (e.g., Figure 2 The microphone 220 in the wearable device 101 is used to receive sounds generated in a space adjacent to the wearable device 101, such as sounds generated by an external object 282. The number of microphones and / or speakers included in the wearable device 101 may be two or more.

[0110] According to an embodiment, the wearable device 101 may use a display (e.g., Figure 2 The display 230) displays the space adjacent to the wearable device 101 to the user. The display can be configured to cover at least a portion of the user's eyes when the user is wearing the wearable device 101. In an embodiment, based on a transparent or translucent material, the display can directly transmit light incident toward a second surface opposite to the first surface to the first surface of the display that is set toward the user. In an embodiment, the wearable device 101 can indirectly transmit light incident toward the second surface to the user through the display by displaying an image obtained using a camera in the display set on the first surface, and the camera is set on a first surface of the wearable device 101 that is set toward the user (for example, Figure 4a The first surface 410) is opposite to the second surface (eg, Figure 4b For example, the wearable device 101 may transmit light indirectly through the display based on a transparent mode (“transparent mode” or “see-through mode”).

[0111] Reference Figure 5a, according to an embodiment, an example state of the wearable device 101 using the display to display the visual object 290 in the FoV 270 is shown. According to an embodiment, the wearable device 101 can display the visual object 290 associated with the display and / or the external object 282 to the user through the FoV 270. The visual object 290 may have a form based on account information related to an external electronic device, which is connected to the wearable device 101 through a network service (e.g., video conferencing) supported by the wearable device 101. For example, the account information may be uniquely assigned to a user logged in to the external electronic device.

[0112] According to an embodiment, the wearable device 101 may set the position of the visual object 290 in the FoV 270 based on a two-dimensional and / or three-dimensional virtual space. For example, the wearable device 101 may use a camera (e.g., Figure 2 The wearable device 101 may obtain a virtual space using a camera 240 of the wearable device 101, and the external space of the wearable device including the FoV 270 is mapped to the virtual space. The virtual space may be formed based on a point (e.g., an origin) corresponding to the position of the wearable device 101. The wearable device 101 may identify or select a portion of a display in which to display the visual object 290 based on the point of the visual object 290 in the virtual space.

[0113] Reference Figure 5a , the wearable device 101 may use a communication circuit (e.g., Figure 2 The wearable device 101 may receive information related to the visual object 290 by using a communication circuit 250 of the wearable device 101. The information may indicate a shape and / or size of the visual object 290. The information may include data for adjusting the position of the visual object 290 in the virtual space based on an input received from an external electronic device corresponding to the visual object 290. For example, based on identifying the data for adjusting the position of the visual object 290 from the information, the wearable device 101 may adjust the position of the visual object 290 in the FoV 270 by changing the display of the visual object 290. For example, based on the data, the wearable device 101 may adjust the point corresponding to the visual object 290 in the virtual space. The operation of the wearable device 101 to change the display of the visual object 290 is not limited to the above operation. For example, the wearable device 101 may change the display of the visual object 290 based on the movement of the user wearing the wearable device 101. For example, when the direction 510 of the FoV 270 is adjusted by the movement of the user, the wearable device 101 may change the position and / or size of the visual object 290 based on the adjusted direction 510.

[0114] According to an embodiment, the information related to the visual object 290 received by the wearable device 101 may include an audio signal transmitted from an external electronic device corresponding to the visual object 290. For example, in the case where the visual object 290 corresponds to an external electronic device connected through a network service such as a video conference, the wearable device 101 may receive an audio signal from the external electronic device, the audio signal including a statement of a user of the external electronic device and received through a microphone of the external electronic device. Based on receiving the audio signal, the wearable device 101 may use a speaker (e.g., Figure 2 The speaker 210 in the apparatus outputs the received audio signal.

[0115] According to an embodiment, the wearable device 101 may output an audio signal related to the visual object 290 based on the positional relationship between the external object 282 displayed by the FoV 270 and the visual object 290. For example, the wearable device 101 may adjust the volume of the audio signal based on the positional relationship. In a state of identifying the volume of the sound generated from the external object 282, the wearable device 101 may adjust the volume of the audio signal based on the volume of the sound and the positional relationship. For example, the wearable device 101 may obtain the sound of the external space of the wearable device 101 using a microphone to identify the volume of the sound generated from the external object 282.

[0116] According to an embodiment, the wearable device 101 may identify the direction of the sound received by the microphone. The direction of the sound may be the azimuth of the source (e.g., the external object 282) outputting the sound to the wearable device 101. In an embodiment where at least two microphones are provided on different parts of the wearable device 101, the wearable device 101 may identify the phase difference of the sound received by each microphone. Based on the phase difference, the wearable device 101 may obtain the azimuth of the sound received by the microphone.

[0117] According to an embodiment, the wearable device 101 may identify whether the sound received through the microphone is output by the external object 282, so as to adjust the volume of the audio signal corresponding to the visual object 290 using the sound output from the external object 282. For example, the wearable device 101 may identify whether the sound is output by the external object 282 by comparing the azimuth of the sound received through the microphone with the position of the external object 282 within the FoV 270. The wearable device 101 may obtain the position of the external object 282 within the FoV 270 based on an image and / or video obtained using a camera. Based on recognizing that the difference between the azimuth and the obtained position is less than a specified range, the wearable device 101 may identify whether the sound received through the microphone is output from the external object 282.

[0118] According to an embodiment, the wearable device 101 may adjust the volume and / or direction of the audio signal based on the volume and / or azimuth of the sound output from the external object 282. The wearable device 101 may adjust the volume and / or direction of the audio signal based on at least one of the position of the visual object 290 corresponding to the audio signal in the virtual space or the position of the external object 282 in the virtual space. Figure 5b ,exist Figure 5a In the state, an example state 520 of the virtual space obtained by the wearable device 101 from the external space is shown.

[0119] Reference Figure 5b According to an embodiment, the wearable device 101 may identify the point 532 as the origin of the virtual space as a position in the virtual space of the wearable device 101. The direction 510 of the y-axis of the virtual space may be Figure 5a The wearable device 101 may determine the position of the external object 282 identified from the FoV 270 using the camera based on the point 532 corresponding to the position of the wearable device 101 in the virtual space. For example, based on identifying the external object 282 in the image obtained from the camera, the wearable device 101 may determine a point 534 in the virtual space as the position of the external object 282 in the virtual space based on the position of the external object 282. For example, in the virtual space, the angle between the line connecting the points 532 and 534 and the y-axis may be the angle between the direction 510 of the FoV 270 identified by the wearable device 101 based on the image and the external object 282. The distance 542 between the points 532 and 534 may be related to the distance between the wearable device 101 and the external object 282 identified by the wearable device 101 based on the image.

[0120] Reference Figure 5b , the wearable device 101 may determine the point 536 in the virtual space as Figure 5a The angle between the line connecting points 532 and 536 and the y-axis may match the angle between the visual object 290 displayed in the display by the wearable device 101 and the direction 510 of the FoV 270. Based on the distance 546 between points 532 and 536, the wearable device 101 may adjust the size of the visual object 290 displayed in the display. According to an embodiment, the wearable device 101 may adjust the position of the point 536 in the coordinate space based on the movement of the user wearing the wearable device 101. The wearable device 101 may receive information for adjusting the position of the point 536 in the virtual space from an external electronic device corresponding to the visual object 290. As the position of the point 536 is adjusted, the wearable device 101 may change the position and / or size of the visual object 290 in the display.

[0121] According to an embodiment, the wearable device 101 may adjust the volume and / or direction of the audio signal corresponding to the visual object 290 based on at least one of the position of the wearable device 101 mapped to the virtual space (e.g., point 532), the position of the external object 282 mapped to the virtual space (e.g., point 534), or the position of the visual object 290 mapped to the virtual space (e.g., point 536). For example, based on identifying the first volume of the sound generated from the external object 282, the wearable device 101 may identify the positional relationship of points 534 and 536, which respectively indicate the position of the external object 282 and the position of the visual object 290 in the virtual space. The wearable device 101 may obtain the second volume of the audio signal corresponding to the visual object 290 by adjusting the first volume based on the positional relationship. For example, the wearable device 101 may obtain the second volume by adjusting the first volume based on the distance 544 between the points 534 and 536.

[0122] exist Figure 5b In state 520 of , distance 546 between point 536 (position of visual object 290 mapped to virtual space) and point 532 (position of wearable device 101 mapped to virtual space) may be greater than distance 542 between point 534 (position of external object 282 mapped to virtual space) and point 532. When it is identified that distance 546 exceeds distance 542, wearable device 101 may obtain a second volume of the audio signal corresponding to visual object 290 by proportionally reducing the first volume of the sound generated by external object 282 based on a ratio between distances 542 and 546 and / or distance 544.

[0123] According to an embodiment, the wearable device 101 may adjust the azimuth of the audio signal corresponding to the visual object 290 mapped to the point 536 based on the azimuth of the sound of the external object 282 mapped to the point 534. For example, based on at least one of the distance 544 between the points 534 and 536 or the azimuth of the sound of the external object 282, the wearable device 101 may obtain the azimuth of the audio signal. For example, the wearable device 101 may obtain the azimuth of the audio signal based on the angle between the line connecting the point 536 corresponding to the visual object 290 in the virtual space and the point 532 corresponding to the wearable device 101 in the virtual space and the y-axis (or x-axis). Based on the obtained azimuth, the wearable device 101 may adjust the phase difference of the audio signal output from each speaker corresponding to each of the two ears. Based on the phase difference, the wearable device 101 may provide a three-dimensional effect associated with the audio signal to the user wearing the speaker.

[0124] As described above, according to an embodiment, the wearable device 101 may adjust the volume and / or azimuth of the audio signal corresponding to the visual object 290 based on the position of the external object 282 and the position of the visual object 290 within the FoV 270 obtained using the camera. In a state where each of the positions of the external object 282 and the visual object 290 is mapped to the points 534 and 536 in the virtual space, the wearable device 101 may obtain the sound of the external space in which the wearable device 101 is set. The device 101 may identify whether the obtained sound is generated from the external object 282 based on the image obtained using the wearable camera. When the obtained sound is generated from the external object 282, the wearable device 101 may obtain the second volume of the audio signal corresponding to the visual object 290 by adjusting the first volume of the obtained sound based on the points 534, 536 in the virtual space. The wearable device 101 may express the positional relationship between the external object 282 and the visual object 290 visualized through the FoV 270 with sound by outputting the audio signal based on the second volume.

[0125] In the following, reference is made to Figure 6a and Figure 6b , according to the embodiment, will be described in more detail in Figure 5a An example of a UI displayed in a state where the wearable device 101 adjusts the volume of an audio signal based on the volume of a sound output from an external object 282.

[0126] Figure 6a and Figure 6b is a diagram showing an example of a UI for adjusting the volume of an audio signal output through a speaker, provided by the wearable device 101 according to an embodiment. Figure 6a and Figure 6b The wearable device 101 may be Figure 2 An example of a wearable device 101. For example, Figure 2 , 3a and Figure 3b and / or Figure 4a The wearable device 101 may include Figure 6a and Figure 6b Wearable device 101. Figure 6a The status can be Figure 5a status related.

[0127] According to an embodiment, the wearable device 101 may respond to a signal received through a microphone (e.g., Figure 2 The wearable device 101 can obtain the direction (or azimuth) of the sound by detecting the sound with a microphone 220. Figure 2The wearable device 101 may capture an image of at least a portion of the FoV 270 that is recognized by the camera 240 of the wearable device 101. The wearable device 101 may recognize a portion corresponding to the recognized direction in the obtained image. Based on recognizing the external object 282 from the portion, the wearable device 101 may determine that the sound is output from the external object 282.

[0128] In an embodiment, the wearable device 101 may recognize the external object 282 from the image based on a neural network trained for visual recognition and / or at least one feature point in the image. For example, the wearable device 101 may recognize whether the sound detected by the microphone is output from the external object 282 based on the category of the external object 282 recognized from the image. Figure 6a In the example case, the wearable device 101 can determine that the sound is output from the external object 282 based on identifying the external object 282 included in the specified category representing a person, and identifying that the first direction of the external object 282 within the FoV 270 matches the second direction of the sound detected by the microphone.

[0129] In an embodiment, based on receiving a sound output from the external object 282 and having a first volume, the wearable device 101 may adjust a second volume of an audio signal corresponding to the visual object 290 based on the first volume. Figure 6a , the wearable device 101 may use a display (e.g., Figure 2 The display 230 of the wearable device 101 displays a visual object 610 such as a pop-up window in the FoV 270 for guiding the adjustment of the volume of the audio signal based on the sound output from the external object 282. Within the visual object 610, the wearable device 101 may display a specified text (e.g., "Set to volume standard?") for guiding the use of the first volume of the sound output from the external object 282 as a reference volume for adjusting the second volume of the audio signal. The wearable device 101 may display a visual object 612, which is a button (e.g., a button including a specified text such as "Yes") for confirming that the first volume is used as the reference volume.

[0130] According to an embodiment, the wearable device 101 may display a visual object 614, which is a button (e.g., a button including a specified text such as "No") within the visual object 610 for stopping the display of the visual object 610. The button may be selected based on a gesture detected by the wearable device 101 and performed by a user wearing the wearable device 101. For example, the gesture may include at least one of a gesture of tapping the wearable device 101, a user's line of sight detected by the wearable device 101, or a gesture of a body part (e.g., a hand) of the user recognized by a camera of the wearable device 101. For example, in response to an input indicating the selection of the visual object 614, the wearable device 101 may at least temporarily stop displaying the visual object 610. In response to an input indicating the selection of the visual object 612, the wearable device 101 may obtain a second volume of an audio signal corresponding to the visual object 290 from the first volume.

[0131] In an embodiment, the wearable device 101 may display a visual object 620, which is a pop-up window associated with the visual object 290 for adjusting the volume of the audio signal corresponding to the visual object 290. For example, the wearable device 101 may display the visual object 620 adjacent to the visual object 290 displayed through the FoV 270. The embodiment is not limited thereto, and the wearable device 101 may display the visual object 620 as overlapping at least a portion of the visual object 290. Within the visual object 620, the wearable device 101 may display a line 624 formed along a specified direction (e.g., a width direction in a width direction and a height direction of the FoV 270) and a visual object 626 superimposed on the line 624. Although a circular visual object 626 is shown, the embodiment is not limited thereto, and the visual object 626 may include an image and / or an icon.

[0132] According to an embodiment, wearable device 101 can visualize the volume of the audio signal corresponding to visual object 290 based on the position of visual object 626 in line 624. The ratio of line 624 divided by visual object 626 can be a ratio of the speaker of wearable device 101 (e.g., Figure 2 The wearable device 101 may receive an input indicating that the position of the visual object 626 is adjusted within the line 624 based on a gesture of the user wearing the wearable device 101. Based on the input, the wearable device 101 may change the position of the visual object 626. The wearable device 101 may adjust the volume of the audio signal based on the changed position of the visual object 626 in the line 624. Although not shown, the wearable device 101 may visualize the volume based on the numerical value displayed in the visual object 620.

[0133] According to an embodiment, the wearable device 101 may adjust the second volume of the audio signal corresponding to the visual object 290 based on the first volume of the sound output from the external object 282. For example, the adjustment of the second volume by the wearable device 101 based on the first volume may be initiated based on an input indicating the selection of the visual object 612. According to an embodiment, the wearable device 101 may display a visual object related to the first volume in the visual object 620 to adjust the second volume of the audio signal. For example, the wearable device 101 may represent the first volume recognized from the external object 282 based on the line 622 overlapping the line 624. The line 622 and the line 624 may be perpendicular to each other.

[0134] According to an embodiment, the wearable device 101 can visualize the first volume of the sound of the external object 282 recognized by the microphone and the second volume of the audio signal based on the position of the visual object 626 and the line 622 overlapping the line 624. For example, the wearable device 101 can display the second volume obtained by adjusting the first volume by adjusting the interval between the line 622 and the visual object 626 within the line 624. Figure 5b In the example case where the wearable device 101 identifies the positions of the visual object 290 and the external object 282 in the virtual space as shown in the state 520, the second volume corresponding to the visual object 290 can be lower than the first volume corresponding to the external object 282. Since the second volume is lower than the first volume, the wearable device 101 can indicate that the second volume is lower than the first volume by displaying the visual object 626 closer to the left side (or the origin) than the line 622.

[0135] According to an embodiment, the wearable device 101 may display a visual object 628 such as a button in the visual object 620 for adjusting a reference volume for adjusting the volume of the audio signal (e.g., a button with a designated text such as “reference adjustment”), thereby adjusting the volume of the audio signal corresponding to the visual object 290. Figure 6a In the example case of , in which the wearable device 101 adjusts the volume of the audio signal based on the volume of the sound output from the external object 282, the visual object 628 may be associated with a function that changes the external object used to adjust the volume of the audio signal to another external object different from the external object 282. Based on an input indicating a selection of the visual object 628, the wearable device 101 may request the user to select an external object to be used to adjust the volume of the audio signal from one or more external objects displayed in the FoV 270.

[0136] Reference Figure 6b , according to an embodiment, an example state is shown in which the wearable device 101 receives an audio signal output from the external object 286 through the external electronic device 630. Figure 6bIn an embodiment, where the external object 282 is another user different from the user wearing the wearable device 101, the external electronic device 630 may include a wearable device worn by the other user. The present embodiment is not limited thereto, and the external electronic device 630 may include an electronic device (e.g., a terminal) different from the wearable device. The wearable device 101 may establish a communication link between the wearable device 101 and the external electronic device 630 based on the recognition of the external electronic device 630.

[0137] The external electronic device 630 may include one or more microphones for recording sound. The external electronic device 630 may transmit an audio signal received through the one or more microphones to the wearable device 101. The external electronic device 630 may transmit information related to the audio signal. The information may be transmitted to the wearable device 101 together with the audio signal based on the external electronic device 630. The information may include information indicating the source of the audio signal. The information may include the volume of the audio signal measured by the external electronic device 630. Figure 6b , in a state where an audio signal including a sound output from an external object 282 is transmitted to the wearable device 101, the external electronic device 630 may transmit information including a volume of the audio signal to the wearable device 101.

[0138] According to an embodiment, the wearable device 101 may output an audio signal received from the external electronic device 630 through a speaker included in the wearable device 101. The wearable device 101 may receive information related to the audio signal from the external electronic device 630. Based on the volume of the audio signal included in the information and measured by the external electronic device 630, the wearable device 101 may adjust the volume of the audio signal output through the speaker of the wearable device 101. For example, the wearable device 101 may output the sound output from the external object 286 to the user wearing the wearable device 101 based on the audio signal of the external object 286 obtained by the external electronic device 630 adjacent to the external object 286. For example, in an environment where the sound output from the external object 286 cannot be directly transmitted to the wearable device 101, the wearable device 101 may transmit the sound output from the external object 286 to the user wearing the wearable device 101 based on the audio signal received from the external electronic device 630.

[0139] According to an embodiment, the wearable device 101 may adjust a second volume related to the external object 286 based on a first volume of an audio signal recognized by the external electronic device 630 (e.g., the volume of a sound output from the external object 282). The first volume may be included in information sent from the external electronic device 630 to the wearable device 101. For example, the first volume may be used as a reference volume for the second volume. The second volume may include a volume of an audio signal related to the external object 286 obtained based on the external electronic device 630 and / or the wearable device 101. Reference Figure 6b , the wearable device 101 can notify, based on the visual object 610 adjacent to the external electronic device 630, that the first volume of the audio signal received through the external electronic device 630 is used as a reference volume. Based on the visual objects 612 and 614 included in the visual object 610, the wearable device 101 can receive an input from the user indicating whether to set the first volume as the reference volume. In a state where the first volume is set as the reference volume, the wearable device 101 can display a visual object 620 in the FoV 270 for adjusting the second volume corresponding to the external object 286. The line 622 in the visual object 620 may correspond to the first volume set as the reference volume. With reference to Figure 6a Similar to the above operation, based on the position of line 622 and visual object 626 on line 624, the wearable device 101 can change the second volume based on the first volume as a reference volume.

[0140] As described above, according to an embodiment, the wearable device 101 can adjust the second volume of the audio signal corresponding to the visual object 290 based on the first volume of the sound of the external object 282 that actually exists. The wearable device 101 can use the display to display a UI (e.g., visual objects 610 and 620) for adjusting the second volume. Based on the UI, the wearable device 101 can visualize the adjustment of the second volume based on the first volume.

[0141] In the following, reference is made to Figure 7a , Figure 7b and Figure 7c According to an embodiment, an operation example of the wearable device 101 adjusting the volume of the audio signal corresponding to the visual object 290 based on the movement of at least one of the external object 282 or the visual object 290 recognized according to the virtual space will be described in more detail.

[0142] Figure 7a , Figure 7b and Figure 7c is a diagram illustrating an example of an operation for adjusting the volume of an audio signal through a wearable device according to an embodiment. Figures 7a to 7c Wearable devices may include Figure 2 To the wearable device 101 of FIG. 6 . Figures 7a to 7c According to an embodiment, a wearable device is shown based on a display (e.g., Figure 2 One or more external objects (e.g., Figure 2 external objects 280) and visual objects (e.g., Figure 2 The different states 710, 720 and 730 of the virtual space are identified based on the position of the visual object 290 in the virtual space. The point 532 in the virtual space can indicate the position of the wearable device.

[0143] According to an embodiment, the wearable device may use a camera (e.g., Figure 2 camera 240) and / or sensors (e.g., Figure 4b The depth sensor 430) is used to identify one or more points of one or more external objects in the virtual space. Figures 7a to 7c In an embodiment in which the wearable device identifies two external objects, points 534 and 705 may represent the location of each external object identified by the wearable device based on the virtual space. A user wearing the wearable device may see the external objects through the display of the wearable device. For example, the external objects may be included in the user's FoV (e.g., Figure 2 The distance 542 may indicate the distance between points 532 and 534 (e.g., the distance between the wearable device and a first external object corresponding to point 534). The distance 712 may indicate the distance between points 705 and 532 (e.g., the distance between the wearable device and a second external object corresponding to point 705).

[0144] In an embodiment, the wearable device may identify at least one visual object (e.g., Figure 2 The visual object 290) is at least one point in the virtual space. Figures 7a to 7c In an embodiment in which the wearable device displays a visual object within a display, point 536 may represent a location of the visual object recognized by the wearable device based on the virtual space.

[0145] As mentioned above Figure 2 As described in FIG. 6 , according to an embodiment, the wearable device may obtain a second volume of an audio signal corresponding to a visual object based on a first volume of a sound of an external object that actually exists. The wearable device may obtain the second volume based on the first volume, the difference between the positions of the external object and the visual object in the virtual space. Figure 7a In state 710, the wearable device may obtain the volume of the audio signal corresponding to the visual object based on the sound output from the external object closest to the visual object among the multiple external objects. Figure 7aIn state 710 of , the wearable device may identify the external object closest to the visual object based on the distances 544 and 714 between the points 534 and 705 indicating the location of each external object and the point 536 indicating the location of the visual object. For example, the wearable device may obtain a second volume of an audio signal corresponding to the visual object based on a first volume of a sound output from a first external object corresponding to point 534. The second volume may be obtained based on the distance 544 between the points 534 and 536 in the virtual space and the first volume. For example, in Figure 7a In state 710, the wearable device may determine a first volume of a first external object as a reference volume and / or a dominant volume for adjusting a second volume of a visual object.

[0146] According to an embodiment, the wearable device may recognize the movement of an external object and / or a visual object. The wearable device may recognize the movement of an external object using a camera. The wearable device may recognize the movement of a visual object based on at least one of the movement of a user recognized by the wearable device or a signal transmitted from an external electronic device indicated by the visual object. The movement of an external object and / or a visual object recognized by the wearable device may trigger the movement of at least one point corresponding to the external object or the visual object in the virtual space. Figure 7a In state 710, it is assumed that the wearable device has recognized an input for moving the visual object to point 716 in the virtual space. Figure 7b , showing the state 720 of the virtual space after the position of the point 536 corresponding to the visual object is adjusted based on the input.

[0147] Reference Figure 7b Based on the recognition that the position of the point 736 corresponding to the visual object is adjusted, the wearable device can again recognize the external object closest to the visual object. Figure 7b In state 720, the wearable device may identify point 705 that is closest to point 536 indicating the location of the visual object from points 534 and 705 indicating the location of each of the multiple external objects. The wearable device may obtain a second volume of the audio signal based on a third volume of a second external object corresponding to point 705 among the multiple external objects. The second volume may be obtained based on a distance 714 between points 536 and 705 and the third volume. When point 536 corresponding to the visual object moves toward point 532 corresponding to the wearable device, the second volume of the audio signal in state 720 may be greater than the second volume of the audio signal in state 710. Figure 7b In the state 720, Figure 7a Unlike state 710 of , the wearable device may recognize the volume of the acoustic signal output from the second external object as a reference volume for adjusting the second volume of the audio signal. Figure 7bIn state 720, the wearable device can adjust the position of point 536 corresponding to the visual object based on input indicating that the visual object is moved to point 722 in the virtual space.

[0148] Figure 7c State 730 may represent a state of the virtual space after the wearable device adjusts the position of point 536 based on the input. In state 730, based on the first external object corresponding to point 534 closest to point 536 (point 536 corresponds to the visual object) among points 534 and 705 representing the positions of multiple external objects, the wearable device may adjust the volume of the audio signal corresponding to the visual object. For example, the wearable device may obtain a second volume of the audio signal based on the first volume of the sound output from the first external object and the distance 544 between points 534 and 536 in the virtual space.

[0149] Although the operation of the wearable device adjusting the volume of the audio signal corresponding to the visual object based on the movement of the visual object in the virtual space is described, the embodiments are not limited thereto. For example, based on recognizing that at least one of the external objects indicated by points 534 and 536 is moving, the wearable device may again recognize a reference volume for adjusting the volume of the audio signal corresponding to the visual object, similar to the above reference to Figures 7a to 7c Although an embodiment in which a wearable device adjusts the volume of an audio signal based on the volume of a sound output from a single external object is described, the embodiment is not limited thereto. Figure 7c In state 730, according to an embodiment, the wearable device can obtain the volume of the audio signal by mixing the volume of the sound of the external object corresponding to points 534 and 705 based on the distances 714 and 544 formed by point 536 and each of points 534 and 705.

[0150] As described above, according to an embodiment, in a state where a first external object and a second external object are recognized, the wearable device may obtain a first distance (e.g., distance 544) between the first external object and the visual object, and a second distance (e.g., distance 714) between the second external object and the visual object. Based on obtaining the first distance that is less than the second distance, the wearable device may obtain a second volume of the audio signal according to the first volume of the sound output from the first external object. In a state where the second volume is obtained based on the first volume, based on recognizing that the second distance is reduced to less than the first distance according to the movement of at least one of the first external object, the second external object, or the visual object, the wearable device may obtain a second volume according to a third volume of another sound output from the second external object. Since the wearable device obtains the volume of the audio signal based on the volume of the sound output from the external object and the position of the external object in the virtual space, the wearable device may reduce the sense of heterogeneity generated when reproducing the audio signal together with the sound.

[0151] According to an embodiment, the wearable device may display a visual object for controlling an external electronic device. Based on the positions of the external electronic device and the visual object displayed through FoV, the wearable device may control the external electronic device or adjust the volume of an audio signal corresponding to the visual object. Figure 8 According to an embodiment, an example of an operation performed by a wearable device based on a visual object for controlling an external electronic device will be described.

[0152] Figure 8 is a diagram showing an operation example of displaying a visual object 810 in association with an external electronic device according to an embodiment. Figure 8 The wearable device 101 may include Figure 2 To Figure 6 and / or Figures 7a to 7c wearable devices. For example, Figure 2 The wearable device 101 and FoV 270 may correspond to Figure 8 Wearable device 101 and FoV 270 .

[0153] According to an embodiment, the wearable device 101 may use an external object 284 as an external electronic device to communicate with a communication circuit (e.g., Figure 2 The wearable device 101 may establish a communication link with the external object 284 through the communication circuit 250 of the wearable device 101 to receive the audio signal from the external object 284. The communication link between the wearable device 101 and the external object 284 may be established based on a process (e.g., pairing) through a communication protocol. The wearable device 101 may recognize that the external object 284 is a speaker for reproducing the audio signal through the communication link.

[0154] Reference Figure 8 According to an embodiment, the wearable device 101 may recognize an input indicating generation of a visual object 810 related to the external object 284 while the external object 284 is displayed in the FoV 270. The input may be recognized based on a gesture of a user wearing the wearable device 101. For example, the user may tap the wearable device 101 according to a specified number of times while looking at the external object 284. In the example, based on a gesture of tapping the wearable device 101 and the user's gaze detected when performing the gesture, the wearable device 101 may recognize the input. The embodiment is not limited thereto. For example, the wearable device 101 may receive an input based on a gesture of dragging the external object 284.

[0155] Reference Figure 8 , shows that the wearable device 101 displays a display (eg, Figure 2According to an embodiment, in a state where the visual object 810 related to the external object 284 for reproducing the audio signal is displayed, the wearable device 101 may, based on the position of the visual object 810 and the position of the external object 284, transmit the audio signal to the wearable device 101 through a speaker (e.g., Figure 2 The speaker 210 outputs at least a portion of the audio signal. The position of the visual object 810 and the position of the external object 284 can be identified by the wearable device 101 in a similar manner to the above reference Figure 5a to Figure 5b and / or Figures 7a to 7c The execution described.

[0156] According to an embodiment, the wearable device 101 may perform at least one of reproducing an audio signal received from the external object 284 based on the visual object 810 or controlling the audio signal reproduced by the external object 284. For example, the wearable device 101 may request the external object 284 to send an audio signal corresponding to the visual object 810. The wearable device 101 may output the audio signal received from the external object 284 based on the position of the external object 284 and the position of the visual object 810. For example, the volume of the audio signal output by the wearable device 101 may be adjusted based on at least one of the volume of the sound output from the external object 284, the position of the external object 284, or the position of the visual object 810. For example, adjusting the volume of the audio signal output by the wearable device 101 may be similar to the above reference to Figure 5a to Figure 5b and / or Figures 7a to 7c The execution described.

[0157] According to an embodiment, the wearable device 101 may control the reproduction of the audio signal by the wearable device 101 and / or the external object 284 based on multiple channels such as stereo. For example, the wearable device 101 may request the external object 284 to send an audio signal of a specific channel. The specific channel may be related to the positional relationship between the external object 284 and the visual object 810 in the FoV 270 of the wearable device 101. In the FoV 270, Figure 8 In the example case where the visual object 810 is shown to be located to the right of the external object 284, the wearable device 101 can request the external object 284 to send the audio signal of the right channel. Based on the request, the external object 284 can send the audio signal of the right channel to the wearable device 101. Based on the request, the external object 284 can reproduce the audio signal of the left channel. Figure 8 In the example case, the external object 284 can reproduce the audio signal of the left channel, and the wearable device 101 can reproduce the audio signal of the right channel, thereby providing a stereo-based user experience.

[0158] Reference Figure 8According to an embodiment, the wearable device 101 may display a visual object 820 for controlling the reproduction of an audio signal related to the visual object 810. The visual object 820 may include a pop-up window for controlling the reproduction of the audio signal. In the visual object 820, the wearable device 101 may display a visual object 822 including text indicating an external object 284 related to the visual object 810. Figure 8 , the wearable device 101 may display a visual object 822 including a name (e.g., “sample speaker”) assigned to the external object 284. In the visual object 820, the wearable device 101 may display a visual object 824 including text indicating a channel of an audio signal output from the wearable device 101 based on the visual object 810. Figure 8 In the example case of , the wearable device 101 may display a visual object 824 including a designated text indicating that the audio signal of the right channel is reproduced (e.g., "Right (R)"). Based on the input associated with the visual object 824, the wearable device 101 may receive an input indicating the selection and / or adjustment of the channel of the audio signal output from the wearable device 101. In the visual object 820, the wearable device 101 may display a visual object 826 such as a button for stopping displaying the visual object 810 (e.g., a button including designated text such as "Delete"). In response to the input indicating the selection of the visual object 826, the wearable device 101 may at least temporarily stop displaying the visual object 810. Based on the input associated with the visual object 826, the wearable device 101 may release the communication link between the external object 284 and the wearable device 101. In response to the input to the visual object 826, the wearable device 101 may at least temporarily stop reproducing the audio signal associated with the visual object 810.

[0159] As described above, according to an embodiment, the wearable device 101 may request an audio signal related to a sound output from the external object 284 through the communication circuit based on the positional relationship between the external object 284 displayed through the display and the visual object 810 displayed within the display. The wearable device 101 may output an audio signal through the speaker based on the audio signal received as a response to the request through the communication circuit. The wearable device 101 may request the external object 284 through the communication circuit for a first channel (in the plurality of channels of sound output from the external object) identified based on the positional relationship. Figure 8 The request to the external object 284 may indicate to at least temporarily stop playing the first channel of the plurality of channels. In this case, the wearable device 101 may support the reproduction of a stereo audio signal based on the speakers of the wearable device 101 and the external object 284.

[0160] In the following, reference is made to Fig. 9 , Fig.10 , Fig.11 and Fig.12 , the above reference will be described in more detail Figures 2 to 8 An example of operation of wearable device 101 is described.

[0161] Fig. 9 is a flowchart illustrating an example operation of a wearable device according to an embodiment. Fig. 9 Wearable devices may include Figures 2 to 8 The wearable device 101. For example, Fig. 9 The operation can be done by Figure 2 The wearable device 101 and / or the processor 120 executes.

[0162] Reference Fig. 9 In operation 910, the wearable device according to the embodiment may display a visual object. The visual object may include Figure 2 Visual Object 290 and / or Figure 8 The wearable device can be used to Figure 2 The display 230 displays the visual object. The wearable device may perform operation 910 by using a camera (e.g., Figure 2 The camera 240 in the wearable device recognizes one or more external objects (e.g., Figure 2 The execution is performed under the state of the external object 280).

[0163] Reference Fig. 9 According to an embodiment, in operation 920, the wearable device may receive an acoustic signal having a first volume. For example, the wearable device may use Figure 2The microphone 220 of the wearable device receives an acoustic signal. According to an embodiment, based on receiving the acoustic signal, in operation 930, the wearable device may determine whether the received acoustic signal is output from an external object. The wearable device may determine whether the acoustic signal is output from an external object by comparing the direction of the acoustic signal received based on operation 920 and the position of the external object recognized using the camera. For example, the wearable device may identify a first direction of the acoustic signal for the wearable device in response to receiving the acoustic signal of operation 920. The wearable device may identify a second direction of the external object based on an image obtained from the camera. The wearable device may identify whether the acoustic signal is output from an external object based on the first direction and the second direction. In an embodiment, the wearable device may identify whether the acoustic signal is output from an external object based on the category into which the external object recognized according to the image is classified. For example, based on whether the external object is suitable for outputting the acoustic signal of operation 920, the wearable device may identify whether the acoustic signal is output from an external object. When the acoustic signal is not output from the external object (No of 930), the wearable device may receive the acoustic signal based on operation 920.

[0164] According to an embodiment, when the acoustic signal of operation 920 is output from an external object (yes of 930), in operation 940, the wearable device may obtain a second volume from the first volume of operation 920 based on at least one of the positions of the external object and the visual object. For example, the wearable device may obtain the second volume based on at least one of the first direction of the acoustic signal, the second direction of the external object, or the positional relationship. The positions of the external object and the visual object may include positions within a virtual space identified by the wearable device. The wearable device may identify the difference between the position of the external object and the position of the visual object within the virtual space. Based on the difference, the wearable device may obtain the second volume by adjusting the first volume of the acoustic signal.

[0165] Reference Fig. 9 In operation 950, according to an embodiment, the wearable device may output an audio signal corresponding to the visual object based on the second volume. The wearable device may use a speaker (e.g., Figure 2 The wearable device may output an audio signal to a speaker 210 of the user. As shown in visual objects 610 and 620 of FIG. 6 , the wearable device may display a UI for adjusting the second volume of operation 950 to the user. Since the audio signal is output based on the second volume adjusted from the first volume of the acoustic signal, the wearable device may reduce the sense of heterogeneity based on the significant difference between the first volume and the second volume. In an embodiment, when the wearable device includes two speakers corresponding to each of the two ears of the user, the wearable device may adjust the azimuth of the audio signal based on the phase difference of the audio signal output from each speaker.

[0166] Fig.10 is a flowchart illustrating example operations performed by a wearable device based on reception of an acoustic signal according to an embodiment. Fig.10 Wearable devices may include Figures 2 to 8 Wearable device 101. Fig.10 The operation can be done by Figure 2 The wearable device 101 and / or the processor 120 executes. Fig.10 The operation may be related to operations 920, 930, and 940. However, the present embodiment is not limited thereto.

[0167] Reference Fig.10 According to an embodiment, in operation 1010, the wearable device may use a microphone (e.g., Figure 2 The camera 240 receives the acoustic signal. Fig.10 Operation 1010 may be similar to Fig. 9 For example, in a state where the visual object of operation 910 is displayed, the wearable device may receive an acoustic signal based on operation 1010.

[0168] Reference Fig.10 In operation 1020, according to an embodiment, the wearable device may identify a first direction of an acoustic signal for the wearable device. The first direction may include an azimuth of the sound signal. In an embodiment where the wearable device includes at least two microphones spaced apart from each other, the wearable device may identify the first direction based on a difference between a volume and / or a phase of an acoustic signal received by each microphone.

[0169] Reference Fig.10 In operation 1030, according to an embodiment, the wearable device may use a camera (e.g., Figure 2 The wearable device may use a camera 240 to identify a second direction of an external object. For example, the wearable device may use a camera to obtain a second direction of an external object including a display of the wearable device (eg, Figure 2 The display 230) displays the FoV (e.g., Figure 2 According to an embodiment, the wearable device may process the image based on object recognition to identify an external object from the image. The wearable device may identify the second direction based on the position and / or size of the external object in the image.

[0170] Reference Fig.10, in operation 1040, according to an embodiment, the wearable device may identify whether the acoustic signal is output from an external object based on the first direction and the second direction. For example, in at least a portion of the image that overlaps with the first direction in which the acoustic signal is output, the wearable device may determine that the acoustic signal is output from the external object in response to identifying the external object. For example, when the difference between the first direction and the second direction is less than a specified threshold, the wearable device may determine that the acoustic signal is output from the external object. In a state in which an acoustic signal is output from an external object based on operation 1040, the wearable device may adjust the volume of an audio signal corresponding to a visual object displayed in the display based on the volume of the acoustic signal. For example, in this state, the wearable device may perform Fig. 9 At least one of operations 940 and 950.

[0171] Fig.11 is a flowchart illustrating an example operation of a wearable device according to an embodiment. Fig.11 Wearable devices may include Figures 2 to 10 wearable device. Fig.11 The operation can be done by Figure 2 The wearable device 101 and / or the processor 120 executes. Fig.11 The operation can be Figures 9 and 10 is related to at least one of the operations.

[0172] Reference Fig.11 In operation 1110, according to an embodiment, the wearable device may display a visual object. The wearable device may display the visual object through a display of the wearable device (e.g., Figure 2 The visual object (eg, external object 280) associated with the external object (eg, external object 280) displayed on the display 230) Figure 2 Visual Object 290 and / or Figure 8 In an embodiment, operation 1110 may be performed based on input related to an external object displayed by the display, similar to the above reference to Figure 8 described.

[0173] Reference Fig.11 In operation 1120, according to an embodiment, the wearable device may receive an acoustic signal having a first volume. While displaying the visual object based on operation 1110, the wearable device may use a microphone (e.g., Figure 2 The microphone 220 of the wearable device receives an acoustic signal having a first volume. The wearable device may perform operation 1120 similarly to Fig. 9 The method may be performed by performing operation 920 and / or operations 1010 and 1020.

[0174] Reference Fig.11In operation 1130, according to an embodiment, the wearable device may determine whether an acoustic signal is output from an external object to reproduce an audio signal. In response to receiving the acoustic signal of operation 1120, the wearable device may identify whether the acoustic signal is output from an external object (e.g., Figure 2 and / or Figure 8 The wearable device may perform operation 1130 similar to Fig.10 Operation and / or Fig. 9 The wearable device may determine whether the acoustic signal is output from the external object based on at least one of the difference between the direction of the acoustic signal and the direction of the external object or the type of the external object. When the acoustic signal is not output from the external object (1130-NO), the wearable device may keep receiving the acoustic signal based on operation 1120.

[0175] In the case where the acoustic signal is output from the external object (1130-Yes), in operation 1140, according to an embodiment, the wearable device may obtain a second volume from the first volume based on at least one of the position of the external object and the position of the visual object. Based on recognizing that the acoustic signal is output from the external object, the wearable device may obtain a second volume corresponding to the visual object by adjusting the first volume based on the positional relationship between the external object and the visual object. The wearable device performing operation 1140 may be similar to Fig. 9 The operation 940 is performed.

[0176] Reference Fig.11 In operation 1150, according to an embodiment, the wearable device may output at least a portion of the audio signal received from the external object based on the second volume. Based on the second volume of operation 1140, the wearable device may output at least a portion of the audio signal received from the external object through a speaker (e.g., Figure 2 The speaker 210) outputs the signal through the communication circuit (e.g., Figure 2 The wearable device may transmit a signal to the external object using the communication circuit 250. When the external object reproduces the audio signal based on multiple channels, based on operation 1150, in a state where the audio signal is output based on a first channel among the multiple channels, the wearable device may use the communication circuit to transmit a signal to the external object, the signal indicating that the reproduction of the audio signal based on the first channel is at least temporarily stopped and the reproduction of the audio signal based on other channels different from the first channel is maintained. Figure 8 As shown in the visual object 820, the wearable device can display a UI for adjusting the second volume of operation 1150 to the user.

[0177] Fig.12 is a signal flow diagram illustrating example operations of the wearable device 101 and the external electronic device 1210 according to an embodiment. Fig.12The wearable device 101 may include Figures 2 to 11 wearable device. Fig.12 The operation of the wearable device 101 may be related to Fig.11 is related to at least one of the operations. Fig.12 The operation of the wearable device 101 may be performed by Figure 2 The wearable device 101 and / or the processor 120 executes. Fig.12 The external electronic device 1210 may include an external electronic device capable of establishing a communication link with the wearable device 101, such as Figure 2 and / or Figure 8 External object 284.

[0178] Reference Fig.12 In operation 1220, according to an embodiment, the external electronic device 1210 may reproduce an audio signal based on a plurality of channels. The external electronic device 1210 may be disposed in an external space of the wearable device 101 that can be recognized by the wearable device 101.

[0179] Reference Fig.12 In operation 1230, according to an embodiment, the wearable device 101 may display a visual object related to the external electronic device 1210 (eg, Figure 8 As described above with reference to Figure 8 As described above, the wearable device 101 may display a visual object in response to an input indicating the insertion of a visual object associated with the external electronic device 1210. In an embodiment, the wearable device 101 may display a visual object by executing Figures 9 to 11 At least one operation in which the volume of the audio signal corresponding to the visual object is obtained based on the volume of the external electronic device 1210.

[0180] Reference Fig.12 In operation 1240, according to an embodiment, the wearable device 101 may identify at least one first channel among the channels of the audio signal based on the positions of the external electronic device 1210 and the visual object. The at least one first channel may be adjusted based on the positional relationship between the external electronic device 1210 and the visual object displayed through the FoV. At least one first channel may be identified from the channels of the audio signal in the virtual space of the wearable device 101 based on at least one of the positions of the wearable device 101, the external electronic device 1210, and the visual object. The number of channels of the audio signal may be related to the type of stereo (e.g., stereo and / or multi-channel stereo of more than two) supported by the audio signal.

[0181] Reference Fig.12In operation 1250, according to an embodiment, the wearable device 101 may request an audio signal of at least one first channel. The wearable device 101 may send a signal 1252 to the external electronic device 1210, the signal 1252 indicating a request for an audio signal for at least one first channel. Based on receiving the signal 1252, the external electronic device 1210 may send an audio signal 1254 related to at least one first channel to the wearable device 101. In operation 1260, according to an embodiment, the external electronic device 1210 may reproduce another channel different from at least one first channel among a plurality of channels when sending the audio signal 1254 to the wearable device 101. Based on operation 1260, the external electronic device 1210 may at least temporarily stop reproducing at least one first channel.

[0182] Based on the received audio signal 1254, in operation 1270, according to an embodiment, the wearable device 101 may reproduce at least one first channel audio signal. The wearable device 101 may use a speaker (e.g., Figure 2 The speaker 210 of the wearable device 101 reproduces the audio signal 1254 received from the external electronic device 1210. The reproduction of the audio signal 1254 by the wearable device 101 may be performed based on at least one of a visual object recognized by the wearable device 101 according to the virtual space, and a position between the wearable device 101 and the external electronic device 1210. The reproduction of the audio signal 1254 by the wearable device 101 may be performed based on at least one of a visual object recognized by the wearable device 101 according to the virtual space, and a position between the wearable device 101 and the external electronic device 1210. For example, in operation 1270, when the wearable device 101 adjusts the volume of the external electronic device 1210, the volume of the audio signal 1254 may be obtained based on the difference between the position of the external electronic device 1210 and the visual object.

[0183] As described above, according to an embodiment, the wearable device 101 can adjust the volume of an audio signal output through a speaker of the wearable device 101 based on a sound generated from an actual external object. The volume can be adjusted based on the volume of the sound. The wearable device 101 can adjust the volume of the audio signal by comparing the position of the visual object related to the audio signal with the position of the external object.

[0184] When reproducing an audio signal using a wearable device, a method of adjusting the volume of the audio signal based on the volume of sounds generated around the wearable device may be needed.

[0185] As described above, according to example embodiments, a wearable device (e.g., Figure 2 The wearable device 101 may include: a speaker (e.g., Figure 2 Speaker 210 in), microphone (e.g., Figure 2 microphone 220 in), a display (e.g., Figure 2 230 in the display), a camera (e.g., Figure 2 240), and a processor (e.g., Figure 2 The display may be configured to allow at least a portion of light incident on a first surface of the display to pass through a second surface opposite the first surface. The camera may be configured to capture an image. The processor may be configured to display an image related to an external object visible through the display (e.g., Figure 2 The visual object associated with the external object 280 (e.g., Figure 2 Visual Object 290 and / or Figure 8 The processor may be configured to receive an acoustic signal through a microphone when displaying the visual object. The processor may be configured to identify whether the acoustic signal is output from an external object based on at least an image obtained by the camera in response to receiving the acoustic signal having a first volume. The processor may be configured to obtain a second volume corresponding to the visual object by adjusting the first volume based on a positional relationship between the external object and the visual object based on the identification that the acoustic signal is output from the external object. The processor may be configured to output an audio signal corresponding to the visual object through a speaker based on the obtained second volume.

[0186] For example, the processor may be configured to, in response to receiving the acoustic signal, identify a first direction of the acoustic signal with respect to the wearable device. The processor may be configured to identify a second direction of the external object based on the image. The processor may be configured to identify whether the acoustic signal is output from the external object based on the first direction and the second direction.

[0187] For example, the processor may be configured to identify whether the acoustic signal is output from an external object based on a category into which the external object recognized from the image is classified.

[0188] For example, the processor may be configured to control the wearable device to output an audio signal having a second volume based on at least one of a second direction or a positional relationship of the external object.

[0189] For example, the processor may be configured to obtain a first distance between the first external object and the visual object and a second distance between the second external object and the visual object in a state where a first external object and a second external object different from the first external object are recognized as external objects using a camera. The processor may be configured to obtain a second volume according to a first volume of an acoustic signal output from the first external object based on obtaining a first distance smaller than the second distance.

[0190] For example, the processor may be configured to obtain the second volume based on a third volume of another acoustic signal output from the second external object based on recognizing that the second distance is reduced to be less than the first distance based on movement of at least one of the first external object, the second external object, or the visual object.

[0191] For example, the wearable device may also include communication circuitry (e.g., Figure 2 The communication circuit 250 in the embodiment of the present invention is configured to communicate with an external object that outputs an acoustic signal. The processor may be configured to control the display to display the visual object in the display in response to an input indicating the insertion of the visual object based on the external object.

[0192] For example, the processor may be configured to request an audio signal associated with an acoustic signal output from the external object through the communication circuit based on a positional relationship between an external object visible through the display and a visual object displayed in the display. The processor may be configured to output an audio signal through a speaker based on an audio signal received as a response to the request through the communication circuit.

[0193] For example, the processor may be configured to request, from the external object through the communication circuit, an audio signal of a first channel identified based on the positional relationship among a plurality of channels included in the acoustic signal output from the external object.

[0194] For example, the processor may be configured to request the external object to at least temporarily stop reproduction of a first channel among the plurality of channels.

[0195] As described above, according to example embodiments, a method of operating a wearable device may include: displaying (e.g., Fig.11 The method may include receiving, through a microphone of the wearable device, a signal (e.g., Fig.11 The method may include: in response to receiving the acoustic signal, identifying (for example, Fig.11 In operation 1130 of the embodiment, the method may include: based on recognizing that the acoustic signal is output from the external object, adjusting the first volume based on the positional relationship between the external object and the visual object to obtain (for example, Fig.11 The method may include: outputting (for example, Fig.11 In operation 1150), at least a portion of an audio signal received from an external object through a communication circuit of the wearable device.

[0196] For example, the identifying may include obtaining an image using a camera of the wearable device, the image including at least a portion of a field of view (FoV) visible through the display. The identifying may include performing the identifying in response to identifying an external object in at least a portion of the image that overlaps with a direction of the output acoustic signal.

[0197] For example, obtaining the second volume may include adjusting the first volume based on a difference between a first position of the external object mapped to the virtual space and a second position of the visual object mapped to the virtual space.

[0198] For example, displaying the visual object may include displaying the visual object in response to an input associated with an external object visible through the display.

[0199] For example, the output may include: in a state where an audio signal is output through a speaker based on a first channel among multiple channels, sending a signal to an external object, the signal indicating that reproduction of the audio signal based on the first channel is at least temporarily stopped and reproduction of an audio signal based on another channel different from the first channel is maintained.

[0200] As described above, according to example embodiments, a method of operating a wearable device may include: displaying (e.g., Fig. 9 The method may include: receiving, through a microphone of the wearable device, a signal (e.g., Fig. 9 The method may include: in response to receiving the acoustic signal having a first volume, identifying (for example, based on at least an image obtained by a camera of the wearable device) Fig. 9 The method may include: based on recognizing that the acoustic signal is output from the external object, adjusting the first volume based on the positional relationship between the external object and the visual object to obtain (for example, Fig. 9 The method may include: outputting (for example, Fig. 9 In operation 950 ) an audio signal corresponding to the visual object.

[0201] For example, the identifying may include: in response to receiving the acoustic signal, identifying (e.g., Fig.10 In operation 1020), the acoustic signal is about a first direction of the wearable device. The identifying may include: based on the image, identifying (for example, Fig.10 The identifying may include: based on the first direction and the second direction, identifying (for example, Fig.10 In operation 1040 ) whether the acoustic signal is output from an external object.

[0202] For example, the identifying may include identifying whether the acoustic signal is output from the external object based on a category into which the external object is classified according to the image.

[0203] For example, the outputting may include outputting an audio signal having a second volume based on at least one of a second direction or a positional relationship of the external object.

[0204] For example, the obtaining may include: obtaining a first distance between the first external object and the visual object and a second distance between the second external object and the visual object in a state where a first external object and a second external object different from the first external object are recognized as the external object using a camera. The obtaining may include: obtaining a second volume based on a first volume of an acoustic signal output from the first external object based on obtaining the first distance smaller than the second distance.

[0205] For example, the displaying may include displaying the visual object in the display in response to input indicating the insertion of the visual object based on the external object.

[0206] For example, the outputting may include: requesting an audio signal associated with an acoustic signal output from the external object through a communication circuit based on a positional relationship between an external object visible through the display and a visual object displayed in the display. The outputting may include: outputting an audio signal through a speaker based on an audio signal received as a response to the request through the communication circuit.

[0207] For example, the request may include outputting, to the external object through the communication circuit, an audio signal of a first channel identified based on the positional relationship among a plurality of channels included in the acoustic signal output from the external object.

[0208] For example, the request may include requesting an external object to at least temporarily stop reproducing a first channel among the plurality of channels.

[0209] As described above, according to example embodiments, a wearable device (e.g., Figure 2 The wearable device 101 may include: a communication circuit (eg, Figure 2 communication circuit 250), a speaker (e.g., Figure 2 Speaker 210 in), microphone (e.g., Figure 2 microphone 220 in), a display (e.g., Figure 2 230 in the display), a camera (e.g., Figure 2 240) and a processor (e.g., Figure 2The display may be configured to allow at least a portion of light incident on a first surface of the display to pass through a second surface opposite the first surface. The camera may be configured to capture an image. The processor may be configured to display an image of an external object (e.g., Figure 2 284) associated with a visual object (e.g., Figure 8 The processor may be configured to: receive an acoustic signal having a first volume through a microphone when displaying the visual object. The processor may be configured to: in response to receiving the acoustic signal, identify whether the acoustic signal is output from an external object for reproducing an audio signal. The processor may be configured to: based on identifying that the acoustic signal is output from the external object, obtain a second volume corresponding to the visual object by adjusting the first volume based on a positional relationship between the external object and the visual object. The processor may be configured to: based on the obtained second volume, output at least a portion of the audio signal received from the external object through the communication circuit.

[0210] For example, the processor may be configured to obtain, using a camera, an image including at least a portion of a field of view (FoV) visible through the display. The processor may be configured to recognize that the acoustic signal is output from the external object in response to recognizing the external object in at least a portion of the image that overlaps with a direction in which the acoustic signal is output.

[0211] For example, the processor may be configured to adjust the first volume based on a difference between a first position of the external object mapped to the virtual space and a second position of the visual object mapped to the virtual space.

[0212] For example, the processor may be configured to control the display to display a visual object in response to an input associated with an external object visible through the display.

[0213] For example, the processor may be configured to, in a state where an audio signal is output through a speaker based on a first channel among multiple channels, send a signal to an external object, the signal indicating that reproduction of the audio signal based on the first channel is at least temporarily stopped, and reproduction of an audio signal based on another channel different from the first channel is maintained.

[0214] The above-mentioned devices may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in various example embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as processors, controllers, arithmetic logic units (ALUs), digital signal processors, microcomputers, field programmable gate arrays (FPGAs), PLUs (programmable logic units), microprocessors, and any other device capable of executing and responding to instructions. The processing device may execute an operating system OS and one or more software applications executed on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, although it may be described as using a processing device, those skilled in the art may see that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, the processing device may include multiple processors or a processor and a controller. In addition, other processing configurations (such as parallel processors) are also possible.

[0215] Software may include computer programs, codes, instructions, or a combination of one or more of them, and configures a processing device to operate as required or to command a processing device independently or in combination. Software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device that interprets instructions or data by or provides instructions or data to a processing device. Software may be distributed on networked computer systems and stored or run in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

[0216] The method according to the embodiment can be implemented in the form of program instructions, which can be executed by various computer devices and recorded in non-transient computer-readable media. In this case, the medium can store computer executable programs continuously, or temporarily store the program for running or downloading. In addition, the medium can be various recording devices or storage devices, in which single or multiple hardware are combined, and is not limited to the medium directly connected to any computer system, and can be distributed on the network. Examples of media can include magnetic media such as hard disks, floppy disks and tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floppy disks, ROMs, RAMs, flash memories, etc., for storing program instructions. Examples of other media include app stores that distribute applications, websites that provide or distribute various software, and recording media or storage media managed by servers.

[0217] Although various embodiments have been shown and described according to the various exemplary embodiments and drawings described above, various modifications and alternatives are possible according to the above description. For example, even if the technology is performed in a different order from the method, and / or components such as the system, structure, device, circuit, etc. are combined or combined in a different form from the method, or replaced or replaced by other components or equivalent forms, appropriate results can be obtained.

[0218] Accordingly, other implementations of the claims, other embodiments, and equivalents are within the scope of the disclosure, including the appended claims and their equivalents.It will also be understood that any embodiment described herein can be used in combination with any other embodiment described herein.

Claims

1. A wearable device (101), comprising: Speaker (210); Microphone (220); a display (230) configured to allow at least a portion of incident light to pass through the first surface of the display through a second surface of the display opposite the first surface of the display; a camera (240) configured to capture an image; as well as processor; Wherein, the processor is configured to: controlling the display to display a visual object (290; 810) associated with an external object (280) visible through the display; receiving an acoustic signal via the microphone while displaying the visual object; identifying whether the acoustic signal is output from the external object based at least on the image obtained by the camera in response to receiving the acoustic signal having a first volume; Based on recognizing that the acoustic signal is output from the external object, obtaining a second volume corresponding to the visual object by adjusting the first volume based on a positional relationship between the external object and the visual object; and Based on the obtained second volume, an audio signal corresponding to the visual object is output through the speaker.

2. The wearable device according to claim 1, wherein: The processor is configured to: In response to receiving the acoustic signal, identifying a first direction of the acoustic signal with respect to the wearable device; Based on the image, identifying a second direction of the external object; and Based on the first direction and the second direction, it is identified whether the acoustic signal is output from the external object.

3. A wearable device according to any one of the preceding claims, wherein: The processor is configured to: Based on the category into which the external object recognized from the image is classified, it is identified whether the acoustic signal is output from the external object.

4. A wearable device according to any one of the preceding claims, wherein: The processor is configured to: Based on at least one of the second direction or the positional relationship of the external object, an audio signal having the second volume is output.

5. A wearable device according to any one of the preceding claims, wherein: The processor is configured to: obtaining a first distance between the first external object and the visual object and a second distance between the second external object and the visual object in a state where a first external object and a second external object different from the first external object are recognized as the external object using the camera; as well as Based on the first distance being obtained to be smaller than the second distance, the second volume is obtained according to the first volume of the acoustic signal output from the first external object.

6. A wearable device according to any one of the preceding claims, wherein: The processor is configured to: The second volume is obtained according to a third volume of another acoustic signal output from the second external object based on recognizing that the second distance is reduced to be smaller than the first distance according to movement of at least one of the first external object, the second external object, or the visual object.

7. The wearable device according to any one of the preceding claims, further comprising a communication circuit (250) configured to communicate with the external object outputting the acoustic signal, and in, The processor is configured to: In response to input indicating insertion of the visual object based on the external object, the display is controlled to display the visual object in the display.

8. A wearable device according to any one of the preceding claims, wherein: The processor is configured to: requesting, from the external object through the communication circuit, the audio signal associated with the acoustic signal output from the external object based on the positional relationship between the external object visible through the display and the visual object displayed in the display; as well as Based on the audio signal being received through the communication circuit as a response to the request, the audio signal is output through the speaker.

9. A wearable device according to any one of the preceding claims, wherein: The processor is configured to: The audio signal of a first channel identified based on the positional relationship among a plurality of channels included in the acoustic signal output from the external object is requested from the external object through the communication circuit.

10. A wearable device according to any one of the preceding claims, wherein: The processor is configured to: A request is made to the external object to at least temporarily stop reproduction of the first channel among the plurality of channels.

11. A method of operating a wearable device, the method comprising: displaying (1110) a visual object associated with an external object visible through a display of the wearable device; While displaying the visual object, receiving (1120) an acoustic signal having a first volume via a microphone of the wearable device; in response to receiving the acoustic signal, identifying ( 1130 ) whether the acoustic signal is output from the external object to reproduce an audio signal; Based on recognizing that the acoustic signal is output from the external object, obtaining (1140) a second volume corresponding to the visual object by adjusting the first volume based on a positional relationship between the external object and the visual object; as well as Based on the obtained second volume, at least a portion of the audio signal received from the external object through the communication circuit of the wearable device is output (1150) through a speaker of the wearable device.

12. The method according to claim 11, wherein: The identification also includes: obtaining an image using a camera of the wearable device, the image comprising at least a portion of a field of view (FoV) visible through the display; and In response to recognizing the external object in at least a portion of the image overlapping with a direction in which the acoustic signal is output, it is recognized that the acoustic signal is output from the external object.

13. A method according to any one of the preceding claims, wherein: Obtaining the second volume further includes: The first volume is adjusted based on a difference between a first position of the external object mapped to the virtual space and a second position of the visual object mapped to the virtual space.

14. A method according to any one of the preceding claims, wherein: Displaying the visual object comprises: The visual object is displayed in response to input associated with the external object visible through the display.

15. A method according to any one of the preceding claims, wherein: The output also includes: In a state where the audio signal is output through the speaker based on a first channel among a plurality of channels, a signal is transmitted to the external object, the signal indicating that reproduction of the audio signal based on the first channel is at least temporarily stopped and reproduction of the audio signal based on another channel different from the first channel is maintained.

Images