Electronic device and audio signal output control method using same

Abstract

An electronic device according to the present document may comprise: a display disposed in the vicinity of the eyes of a user when worn on the body of the user; a plurality of microphones for inputting an external signal; a camera; a speaker for signal playback; a processor; and a memory. The processor can: operate, according to a designated condition, on the basis of execution of instructions stored in the memory, either a first mode for displaying a virtual space on the display or a second mode for displaying, on the display, an actual space captured by a camera; on the basis the operation of either the first mode or the second mode, determine, to be different, the position of a microphone activated for external signal input from among the plurality of microphones disposed on the electronic device; determine, to be different, the number of microphones activated for external signal input from among the plurality of microphones disposed on the electronic device; makes different determinations about whether to activate functions related to amplification or blocking of an external sound; and determine, to be different, a beamforming generation direction for receiving voice signals of the microphone.

Description

Electronic device and method for controlling audio signal output using the same

[0001] Embodiments of the present disclosure relate to an electronic device and a method for controlling audio signal output using the same.

[0002] With recent technological advancements, electronic devices are increasingly evolving from a uniform rectangular shape to a more diverse range of shapes. For example, electronic devices may include wearable electronic devices that can be worn on certain parts of the body.

[0003] Electronic devices (e.g., wearable electronic devices) are evolving into various forms, such as augmented reality (AR) glasses or head-mounted displays (HMDs). These electronic devices may include multiple audio output modules and output audio signals through the multiple audio output modules.

[0004] The HMD may support a see-through mode providing augmented reality (AR) and / or a see-closed mode providing virtual reality (VR).

[0005] See-through mode utilizes the characteristics of semi-transparent lenses to synthesize and combine virtual objects or things on top of the real world, reinforcing additional information that is difficult to obtain from the real world alone. Cycloned mode is a form in which two displays are placed in front of the eyes, allowing you to enjoy content provided through external input (such as games, movies, streaming, or broadcasting) on ​​an independent screen, providing an excellent sense of immersion.

[0006] An electronic device according to this document may include, for example, a virtual see-through (VST) device. The VST device may operate by selecting either a virtual mode, which shows a virtual space to the user, or a see-through mode, which shows a real space.

[0007] According to a comparative example, the VST (virtual see through) device does not perform any separate operation other than turning audio on / off according to the virtual mode and see through mode, so it may be difficult to provide audio performance that suits the user's intention for each mode.

[0008] An electronic device according to the present invention may include a display disposed around the user's eyes when worn on the user's body, a plurality of microphones for inputting external signals, a camera, a speaker for playing back signals, a processor, and a memory. The processor may operate either a first mode for displaying a virtual space on the display or a second mode for displaying an actual space captured by a camera on the display based on specified conditions when instructions stored in the memory are executed, and may determine a different position of a microphone to be activated for inputting an external signal among a plurality of microphones disposed on the electronic device based on whether either the first mode or the second mode is operated, determine a different number of microphones to be activated for inputting an external signal among a plurality of microphones disposed on the electronic device, determine differently whether to activate functions related to amplifying or blocking external sounds, and determine differently a beamforming generation direction for receiving a voice signal from the microphone.

[0009] The method of operating an electronic device may include an operation of activating either a first mode for displaying a virtual space on a display or a second mode for displaying an actual space captured by a camera on the display according to a specified condition, an operation of differently determining a position of a microphone to be activated for external signal input among a plurality of microphones arranged on the electronic device based on whether either the first mode or the second mode is activated, an operation of differently determining a number of microphones to be activated for external signal input among a plurality of microphones arranged on the electronic device, an operation of differently determining whether to activate functions related to amplification or blocking of external sounds, and an operation of differently determining a beamforming generation direction for receiving a voice signal of the microphone.

[0010] An electronic device according to this document can provide a method of utilizing audio appropriate to the user's intention depending on the environment in which the user is currently located or whether the screen the user is viewing is a virtual space or a real space.

[0011] The above and other features, functions and advantages of specific embodiments of the present invention will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings.

[0012] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0013] FIG. 2A is a perspective view schematically illustrating a configuration of a wearable electronic device according to one embodiment of the present disclosure.

[0014] FIG. 2b is a perspective view schematically illustrating a front side of a wearable electronic device according to one embodiment of the present disclosure.

[0015] FIG. 2c is a perspective view schematically illustrating the rear surface of a wearable electronic device according to one embodiment of the present disclosure.

[0016] FIG. 3 is a block diagram illustrating an electronic device according to one embodiment of the present disclosure.

[0017] FIG. 4 illustrates a situation in which an electronic device (301) according to one embodiment operates in a first mode.

[0018] FIG. 5 illustrates a situation in which an electronic device (301) according to one embodiment operates in a second mode.

[0019] Fig. 6 is a flowchart illustrating a method for controlling audio signal output of an electronic device according to one embodiment.

[0020] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with the electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of the electronic device (104) or the server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0021] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0022] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0023] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0024] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0025] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0026] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0027] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0028] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0029] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0030] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0031] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0032] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0033] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0034] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0035] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0036] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0037] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0038] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0039] According to various embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0040] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0041] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0042] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0043] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0044] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0045] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0046] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0047] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0048] FIG. 2A is a perspective view schematically illustrating a configuration of a wearable electronic device according to one embodiment of the present disclosure.

[0049] The wearable electronic device (200) of FIG. 2A may include embodiments described in the electronic device (101) of FIG. 1. The wearable electronic device (200) may include augmented reality (AR) glasses or smart glasses in the form of glasses.

[0050] Referring to FIG. 2A, a wearable electronic device (200) according to various embodiments may include a bridge (bridge, 201), a first rim (rim, 210), a second rim (220), a first end piece (end piece, 230), a second end piece (240), a first temple (temple, 250), and / or a second temple (260).

[0051] In one embodiment, the bridge (201) can connect the first rim (210) and the second rim (220). The bridge (201) can be positioned over the user's nose when the user wears the wearable electronic device (200). The bridge (201) can separate the first rim (210) and the second rim (220) based on the user's nose.

[0052] According to various embodiments, the bridge (201) may include a camera module (203), a first gaze tracking camera (205), a second gaze tracking camera (207), and / or an audio module (209).

[0053] According to various embodiments, the camera module (203) (e.g., the camera module (180) of FIG. 1) can capture a front view (e.g., in the -y-axis direction) of a user (e.g., a user of a wearable electronic device (200)) and obtain image data. The camera module (203) can capture an image corresponding to the user's field of view (FoV) or measure a distance to a subject (e.g., an object). The camera module (203) can include an RGB camera, a high resolution (HR) camera, and / or a photo video (PV) camera. The camera module (203) can include a color camera having an auto focus (AF) function and an optical image stabilization (OIS) function to obtain high-quality images.

[0054] According to various embodiments, the first gaze tracking camera (205) and the second gaze tracking camera (207) can determine the gaze of the user. The first gaze tracking camera (205) and the second gaze tracking camera (207) can capture the pupil of the user's eyes in a direction opposite to the capturing direction of the camera module (203). For example, the first gaze tracking camera (205) can partially capture the left eye of the user, and the second gaze tracking camera (207) can partially capture the right eye of the user. The first gaze tracking camera (205) and the second gaze tracking camera (207) can detect the pupil of the user's eyes (e.g., the left eye and the right eye) and track the gaze direction. The tracked gaze direction can be utilized to move the center of a virtual image including a virtual object in response to the gaze direction. The first gaze tracking camera (205) and / or the second gaze tracking camera (207) can track the user's gaze using, for example, at least one of an EOG sensor (electro-oculography or electrooculogram), a coil system, a dual Purkinje system, bright pupil systems, or dark pupil systems.

[0055] According to various embodiments, an audio module (209) (e.g., audio module (170) of FIG. 1) may be positioned between the first gaze tracking camera (205) and the second gaze tracking camera (207). The audio module (209) may convert a user's voice into an electrical signal or convert an electrical signal into sound. The audio module (209) may include a microphone.

[0056] According to one embodiment, the first rim (210) and the second rim (220) may form a frame (e.g., a glasses frame) of a wearable electronic device (200) (e.g., an AR glass). The first rim (210) may be disposed in a first direction (e.g., an x-axis direction) of the bridge (201). The first rim (210) may be disposed at a position corresponding to the user's left eye. The second rim (220) may be disposed in a second direction (e.g., a -x-axis direction) of the bridge (201) opposite to the first direction (e.g., an x-axis direction). The second rim (220) may be disposed at a position corresponding to the user's right eye. The first rim (210) and the second rim (220) may be formed of a metal material and / or a non-conductive material (e.g., a polymer).

[0057] According to various embodiments, the first rim (210) may surround and support at least a portion of the first glass (215) (e.g., the first display) disposed on the inner surface. The first glass (215) may be positioned in front of the user's left eye. The second rim (220) may surround and support at least a portion of the second glass (225) (e.g., the second display) disposed on the inner surface. The second glass (225) may be positioned in front of the user's right eye. The user of the wearable electronic device (200) may view a foreground (e.g., a real image) of an external object (e.g., a subject) through the first glass (215) and the second glass (225). The wearable electronic device (200) may implement augmented reality by superimposing a virtual image on the foreground (e.g., the real image) of the external object and displaying it.

[0058] According to various embodiments, the first glass (215) and the second glass (225) may include a projection type transparent display. The first glass (215) and the second glass (225) may each form a reflective surface as a transparent plate (or transparent screen), and an image generated from the wearable electronic device (200) may be reflected (e.g., total internal reflection) through the reflective surface and incident on the left and right eyes of the user. In one embodiment, the first glass (215) may include an optical waveguide that transmits light generated from a light source of the wearable electronic device (200) to the left eye of the user. For example, the optical waveguide may be formed of a glass, plastic, or polymer material, and may include a nano-pattern (e.g., a grating structure or a mesh structure having a polygonal or curved shape) formed on the inside or the surface of the first glass (215). The optical waveguide may include at least one diffractive element (e.g., a Diffractive Optical Element (DOE), a Holographic Optical Element (HOE)) or at least one reflective element (e.g., a reflective mirror). The optical waveguide may guide display light emitted from a light source toward a user's eye using at least one diffractive element or reflective element included in the optical waveguide. In various embodiments, the diffractive element may include an input / output optical element, and the reflective element may include a total internal reflection (TIR). For example, light emitted from a light source may be guided along an optical path to the optical waveguide through an input optical element, and light traveling inside the optical waveguide may be guided toward a user's eye through an output optical element. The second glass (225) may be implemented in substantially the same manner as the first glass (215).

[0059] According to various embodiments, the first glass (215) and the second glass (225) may include, for example, a liquid crystal display (LCD), a digital mirror device (DMD), a liquid crystal on silicon (LCoS), an organic light emitting diode (OLED), or a micro light emitting diode (micro LED). Although not shown, when the first glass (215) and the second glass (225) are formed of one of a liquid crystal display, a digital mirror display, or a silicon liquid crystal display, the wearable electronic device (200) may include a light source that irradiates light to the screen output areas of the first glass (215) and the second glass (225). In another embodiment, if the first glass (215) and the second glass (225) are capable of generating light on their own, for example, if they are made of either organic light emitting diodes or micro LEDs, the wearable electronic device (200) can provide a good quality virtual image to the user even without including a separate light source.

[0060] According to various embodiments, the first rim (210) may include a first microphone (211), a first recognition camera (213), a first light-emitting device (217), and / or a first display module (219). The second rim (220) may include a second microphone (221), a second recognition camera (223), a second light-emitting device (227), and / or a second display module (229).

[0061] In various embodiments, the first light emitting device (217) and the first display module (219) may be included in the first end piece (230), and the second light emitting device (227) and the second display module (229) may be included in the second end piece (240).

[0062] According to various embodiments, the first microphone (211) and / or the second microphone (221) can receive the voice of a user of the wearable electronic device (200) and convert it into an electrical signal.

[0063] According to various embodiments, the first recognition camera (213) and / or the second recognition camera (223) can recognize the surrounding space of the wearable electronic device (200). The first recognition camera (213) and / or the second recognition camera (223) can detect a user's gesture within a certain distance (e.g., a certain space) of the wearable electronic device (200). The first recognition camera (213) and / or the second recognition camera (223) can include a global shutter (GS) camera capable of reducing the RS (rolling shutter) phenomenon in order to detect and track the user's rapid hand movements and / or subtle movements of the fingers. The wearable electronic device (200) can detect an eye corresponding to the dominant eye and / or the auxiliary eye among the user's left eye and / or right eye by using the first gaze tracking camera (205), the second gaze tracking camera (207), the first recognition camera (213), and / or the second recognition camera (223). For example, the wearable electronic device (200) can detect the eye corresponding to the dominant eye and / or the auxiliary eye based on the user's gaze direction toward an external object or a virtual object.

[0064] According to various embodiments, the first light emitting device (217) and / or the second light emitting device (227) may emit light to increase the accuracy of the camera module (203), the first gaze tracking camera (205), the second gaze tracking camera (207), the first recognition camera (213), and / or the second recognition camera (223). The first light emitting device (217) and / or the second light emitting device (227) may be used as an auxiliary means to increase the accuracy when capturing the user's pupils using the first gaze tracking camera (205) and / or the second gaze tracking camera (207). The first light emitting device (217) and / or the second light emitting device (227) may be used as an auxiliary means when capturing the user's gesture using the first recognition camera (213) and / or the second recognition camera (223), when it is not easy to detect an object to be captured (e.g., a subject) due to a dark environment or mixing and reflection of multiple light sources. The first light emitting device (217) and / or the second light emitting device (227) may include, for example, an LED, an IR LED, or a xenon lamp.

[0065] According to various embodiments, the first display module (219) and / or the second display module (229) can emit light and transmit the light to the user's left eye and / or right eye using the first glass (215) and / or the second glass (225). The first glass (215) and / or the second glass (225) can display various image information using the light emitted through the first display module (219) and / or the second display module (229). The first display module (219) and / or the second display module (229) can include the display module (160) of FIG. 1. The wearable electronic device (200) can display a foreground for an external object and an image emitted through the first display module (219) and / or the second display module (229) by overlapping them through the first glass (215) and / or the second glass (225).

[0066] In one embodiment, the first end piece (230) may be coupled to a portion (e.g., in the x-axis direction) of the first rim (210). The second end piece (240) may be coupled to a portion (e.g., in the -x-axis direction) of the second rim (220). In various embodiments, the first light-emitting device (217) and the first display module (219) may be included in the first end piece (230). The second light-emitting device (227) and the second display module (229) may be included in the second end piece (240).

[0067] According to various embodiments, the first end piece (230) can connect the first rim (210) and the first temple (250). The second end piece (240) can connect the second rim (220) and the second temple (260).

[0068] In one embodiment, the first temple (250) may be operatively connected to the first end piece (230) using a first hinge portion (255). The first hinge portion (255) may be configured to be rotatable so that the first temple (250) can be folded or unfolded with respect to the first rim (210). The first temple (250) may extend, for example, along the left side of the user's head. A distal portion (e.g., in the y-axis direction) of the first temple (250) may be configured in a curved shape so as to be supported, for example, on the user's left ear when the user wears the wearable electronic device (200). The second temple (260) may be operatively connected to the second end piece (240) using a second hinge portion (265). The second hinge portion (265) may be configured to be rotatable so that the second temple (260) can be folded or unfolded relative to the second rim (220). The second temple (260) may extend, for example, along the right side of the user's head. The distal portion (e.g., in the y-axis direction) of the second temple (260) may be configured in a curved shape so as to be supported, for example, on the user's right ear when the user wears the wearable electronic device (200).

[0069] According to various embodiments, the first temple (250) may include a first printed circuit board (251), a first audio output module (253) (e.g., audio output module (155) of FIG. 1), and / or a first battery (257) (e.g., battery (189) of FIG. 1). The second temple (260) may include a second printed circuit board (261), a second audio output module (263) (e.g., audio output module (155) of FIG. 1), and / or a second battery (267) (e.g., battery (189) of FIG. 1).

[0070] According to various embodiments, various electronic components (e.g., at least some of the components included in the electronic device (101) of FIG. 1), such as the processor (120), memory (130), interface (177), and / or wireless communication module (192) disclosed in FIG. 1, may be disposed on the first printed circuit board (251) and / or the second printed circuit board (261). The processor may include, for example, one or more of a central processing unit, an application processor, a graphics processing unit, an image signal processor, a sensor hub processor, or a communication processor. The first printed circuit board (251) and / or the second printed circuit board (261) may include, for example, a printed circuit board (PCB), a flexible PCB (FPCB), or a rigid-flexible PCB (RFPCB). In some embodiments, the first printed circuit board (251) and / or the second printed circuit board (261) may include a Main PCB, a slave PCB arranged to partially overlap the Main PCB, and / or an interposer substrate between the Main PCB and the slave PCB. The first printed circuit board (251) and / or the second printed circuit board (261) may be electrically connected to other components (e.g., a camera module (203), a first gaze tracking camera (205), a second gaze tracking camera (207), an audio module (209), a first microphone (211), a first recognition camera (213), a first light-emitting device (217), a first display module (219), a second microphone (221), a second recognition camera (223), a second light-emitting device (227), a second display module (229), a first audio output module (253), and / or a second audio output module (263)) using electrical paths such as FPCBs and / or cables.For example, the FPCB and / or cable may be disposed on at least a portion of the first rim (210), the bridge (201), and / or the second rim (220). In some embodiments, the wearable electronic device (200) may include only one of the first printed circuit board (251) and the second printed circuit board (261).

[0071] According to various embodiments, the first audio output module (253) and / or the second audio output module (263) may transmit audio signals to the user's left and / or right ears. The first audio output module (253) and / or the second audio output module (263) may include, for example, a piezo speaker (e.g., a bone conduction speaker) that transmits audio signals without a speaker hole. In some embodiments, the wearable electronic device (200) may include only one of the first audio output module (253) and the second audio output module (263).

[0072] According to various embodiments, the first battery (257) and / or the second battery (267) may supply power to the first printed circuit board (251) and / or the second printed circuit board (261) using a power management module (e.g., the power management module (188) of FIG. 1 ). The first battery (257) and / or the second battery (267) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell. In some embodiments, the wearable electronic device (200) may include only one of the first battery (257) and the second battery (267).

[0073] According to various embodiments, the wearable electronic device (200) may include a sensor module (e.g., the sensor module (176) of FIG. 1). The sensor module may generate an electrical signal or data value corresponding to an internal operating state of the wearable electronic device (200) or an external environmental state. The sensor module may further include, for example, at least one of a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a color sensor, an infrared (IR) sensor, a biometric sensor (e.g., an HRM sensor), a temperature sensor, a humidity sensor, or an illuminance sensor. In some embodiments, the sensor module may recognize biometric information of a user by using various biometric sensors (or biometric recognition sensors), such as an e-nose sensor, an electromyography sensor, an electroencephalogram sensor, an ECG sensor, or an iris sensor.

[0074] According to various embodiments, although the wearable electronic device (200) is described as a device that displays augmented reality using the first glass (215) and the second glass (225) in the above description, it is not limited thereto and may be a device that displays virtual reality (VR).

[0075] In FIG. 2A according to various embodiments, the wearable electronic device (200) is described as a device that displays augmented reality or virtual reality using the first glass (215) and the second glass (225), but is not limited thereto. For example, the wearable electronic device (200) may include a video see-through (VST) device. In this regard, various embodiments will be described in FIGS. 2B and 2C described below.

[0076] FIGS. 2B and 2C are perspective views schematically illustrating the front and back of a wearable electronic device (270) according to one embodiment of the present disclosure.

[0077] Referring to FIGS. 2B and 2C, the wearable electronic device (270) may be provided with a plurality of cameras (e.g., a first camera (273) (e.g., a first recognition camera (213) of FIG. 2A) and a second camera (274) (e.g., a second recognition camera (223) of FIG. 2A)) corresponding to the front direction (e.g., -y direction, a user's gaze direction) of the wearable electronic device (270). For example, the wearable electronic device (270) may include a first camera (273) corresponding to the user's left eye and a second camera (274) corresponding to the user's right eye. The wearable electronic device (270) may capture an external environment in the front direction (e.g., -y direction) of the wearable electronic device (270) using the first camera (273) and the second camera (274). The wearable electronic device (270) may be exposed to the external environment. It may include a first side (271) (e.g., front side) (e.g., see FIG. 2b) and a second side (272) (e.g., back side) (e.g., see FIG. 2c) that is in close contact with the user's skin when worn, but is not exposed to the external environment. For example, when the wearable electronic device (270) is worn on the user's face, the first side (271) of the wearable electronic device (270) may be exposed to the external environment, and the second side (272) of the wearable electronic device (270) may be in a state of at least partially in close contact with the user's face.

[0078] In one embodiment, at least one distance sensor (281, 282, 283, and / or 284) may be disposed on a first surface (271) of the wearable electronic device (270). For example, at least one distance sensor (281, 282, 283, and / or 284) may measure a distance to at least one object disposed around the wearable electronic device (270). The at least one distance sensor (281, 282, 283, and / or 284) may include an infrared sensor, an ultrasonic sensor, and / or a light detection and ranging (LiDAR) sensor. The at least one distance sensor (281, 282, 283, and / or 284) may be implemented based on an infrared sensor, an ultrasonic sensor, and / or a LiDAR sensor.

[0079] In FIG. 2b according to various embodiments, four distance sensors (281, 282, 283, and 284) are shown arranged on the first side (271) of the wearable electronic device (270), but are not limited thereto.

[0080] In one embodiment, the wearable electronic device (270) may have multiple displays (e.g., a first display (275) (e.g., the first glass (215) of FIG. 2A) and a second display (276) (e.g., the second glass (225) of FIG. 2A)) arranged in response to the rear direction of the wearable electronic device (270) (e.g., the +y direction, the direction opposite to the user's gaze direction). For example, the second side (272) (e.g., the rear) of the wearable electronic device (270) may have a first display (275) arranged in response to the user's left eye and a second display (276) arranged in response to the user's right eye. For example, when the wearable electronic device (270) is worn on the user's face, the first display (275) may be arranged in response to the user's left eye, and the second display (276) may be arranged in response to the user's right eye.

[0081] In one embodiment, the wearable electronic device (270) may have a plurality of gaze tracking cameras (e.g., a first gaze tracking camera (291) (e.g., the first gaze tracking camera (205) of FIG. 2A) or a second gaze tracking camera (292) (e.g., the second gaze tracking camera (207) of FIG. 2A) disposed at least partially on the second face (272). For example, the plurality of gaze tracking cameras (291, 292) may track eye movements of the user. The first gaze tracking camera (291) may track eye movements of the user, and the second gaze tracking camera (292) may track eye movements of the user. In one embodiment, the wearable electronic device (270) may determine a direction in which the user is looking based on eye movements tracked using the plurality of gaze tracking cameras (291, 292).

[0082] In one embodiment, a second face (272) of a wearable electronic device (270) may be at least partially provided with a plurality of face recognition cameras (e.g., a first face recognition camera (295) or a second face recognition camera (296)). For example, the plurality of face recognition cameras (295, 296) may recognize a user's face when the wearable electronic device (270) is worn on the user's face. In one embodiment, the wearable electronic device (270) may also use the plurality of face recognition cameras (295, 296) to determine whether the wearable electronic device (270) is worn on the user's face.

[0083] FIG. 3 is a block diagram (300) illustrating an electronic device (301) according to one embodiment of the present disclosure.

[0084] According to various embodiments, the electronic device (301) can be worn by a user and operate in a stand-alone manner to provide augmented reality services.

[0085] Not limited thereto, the electronic device (301) may be connected to an external electronic device (e.g., a smartphone) (e.g., the electronic device (102) or the electronic device (104) of FIG. 1) by wire or wirelessly, receive data (e.g., rendered data) for providing an augmented reality service from the external electronic device, and provide an augmented reality service based on the data for providing the augmented reality service through a display (e.g., a first display (331) and / or a second display module (333)).

[0086] In one embodiment, the electronic device (301) may include, but is not limited to, augmented reality (AR) glasses or smart glasses in the form of glasses.

[0087] Referring to FIG. 3, an electronic device (301) (e.g., the electronic device (101) of FIG. 1, the wearable electronic devices (200, 270) of FIGS. 2A to 2C) may include a wireless communication circuit (310) (e.g., the communication module (190) of FIG. 1), a memory (320) (e.g., the memory (130) of FIG. 1), a camera (325) (e.g., the camera module (180) of FIG. 1), a display (330) (e.g., the display module (160) of FIG. 1), an audio output circuit (340) (e.g., the sound output module (155) of FIG. 1), a sensor circuit (345) (e.g., the sensor module (176) of FIG. 1), and / or a processor (350) (e.g., the processor (120) of FIG. 1).

[0088] According to one embodiment of the present disclosure, a wireless communication circuit (310) (e.g., a communication module (190) of FIG. 1) may establish a communication channel with an external electronic device (e.g., an electronic device (102) of FIG. 1) and support transmitting and receiving various data with the external electronic device.

[0089] In one embodiment, the wireless communication circuit (310) may, under the control of the processor (350), connect communication between the electronic device (301) and an external electronic device (e.g., a smartphone).

[0090] In one embodiment, when the electronic device (301) is connected to an external electronic device (e.g., a smartphone) by wire or wirelessly and is set to receive data for providing an augmented reality service from the external electronic device, the wireless communication circuit (310) may, under the control of the processor (350), be connected to the external electronic device (e.g., a smartphone) to transmit information about a gesture detected by the electronic device (301) to the external electronic device (e.g., a smartphone). The wireless communication circuit (310) may, under the control of the processor (350), receive an object (e.g., a virtual object) from the external electronic device (e.g., a smartphone) to control the output of an audio signal rendered by the external electronic device (e.g., a smartphone) based on the information about the gesture received from the electronic device (301).

[0091] In one embodiment, when the electronic device (301) is connected to an external electronic device (e.g., a smartphone) by wire or wirelessly and is set to receive data for providing an augmented reality service from the external electronic device, the wireless communication circuit (310) may, under the control of the processor (350), transmit information about an output mode of an audio signal determined by a user gesture and / or information about a volume control value of the audio signal to the external electronic device (e.g., a smartphone). The wireless communication circuit (310) may receive, from the external electronic device (e.g., a smartphone), an audio signal corresponding to an output mode converted by the external electronic device (e.g., a smartphone) based on the information about the output mode of the audio signal. The wireless communication circuit (310) may, under the control of the processor (350), receive, from the external electronic device (e.g., a smartphone), an audio signal whose volume is adjusted by the external electronic device (e.g., a smartphone) based on the information about the volume control value of the audio signal.

[0092] According to one embodiment of the present disclosure, the memory (320) (e.g., the memory (130) of FIG. 1) performs a function of storing a program (e.g., the program (140) of FIG. 1) for processing and controlling the processor (350) of the electronic device (301), an operating system (OS) (e.g., the operating system (142) of FIG. 1), various applications, and / or input / output data, and may store a program that controls the overall operation of the electronic device (301). The memory (320) may store various instructions that may be executed by the processor (350).

[0093] In one embodiment, the memory (320) may store instructions for providing an augmented reality service that outputs at least one object (e.g., at least one virtual object) related to an audio signal based on executing a music playback-related application.

[0094] In one embodiment, the memory (320) may store information of gestures mapped to control the output of an audio signal, for example, to change an output mode (e.g., directivity mode, surround mode) of the audio signal, and / or to adjust the volume of the audio signal. The memory (320) may store instructions for changing an output mode (e.g., directivity mode, surround mode) of the audio signal, and / or to adjust the volume of the audio signal, based on gestures detected from an object for controlling the output of the audio signal. The memory (320) may store instructions for outputting a visual effect corresponding to the changed output mode of the audio signal and / or the adjusted volume of the audio signal.

[0095] In one embodiment, the memory (320) may store location information of a user wearing the electronic device (301), location information of an object output for controlling the output of an audio signal in real space, and / or distance information between a user wearing the electronic device (301) and the output object.

[0096] According to one embodiment of the present disclosure, the camera (325) (e.g., the camera module (180) of FIG. 1) may include a recognition camera (326) (e.g., the first recognition camera (213) and / or the second recognition camera (223) of FIG. 2A, or the first camera (273) and / or the second camera (274) of FIG. 2B). The recognition camera (326) (e.g., the first recognition camera (213) and / or the second recognition camera (223) of FIG. 2B, or the first camera (273) and / or the second camera (274) of FIG. 2B) may recognize a surrounding space of the electronic device (301) and detect a user gesture within a certain distance (e.g., a certain space).

[0097] According to one embodiment of the present disclosure, the display (330) (e.g., the display module (160) of FIG. 1) may include a first display (331) (e.g., the first glass (215) of FIG. 2A or the first display (275) of FIG. 2C) and / or a second display (333) (e.g., the glass (225) of FIG. 2A or the second display (276) of FIG. 2C).

[0098] In one embodiment, the first display (331) may be positioned in front of the user's left eye, and the second display (333) may be positioned in front of the user's right eye. Accordingly, the first display (331) and / or the second display (333) may display various information and transmit it to the user's left eye and / or right eye.

[0099] In one embodiment, the first display (331) and / or the second display (333) may, under the control of the processor (350), display an augmented reality space in which at least one object (e.g., at least one virtual object) associated with an audio signal is superimposed on at least a portion of an image (e.g., a preview image) linked to a real space acquired through the camera (325).

[0100] According to one embodiment of the present disclosure, an audio output circuit (340) (e.g., an audio output module (155) of FIG. 1) may include a first audio output circuit (341) (e.g., a first audio output module (253) of FIG. 2A) and a second audio output circuit (343) (e.g., a second audio output module (263) of FIG. 2A), but is not limited thereto.

[0101] In one embodiment, the first audio output circuit (341) and / or the second audio output circuit (343) may transmit audio signals and / or volume-adjusted audio signals corresponding to a mode for outputting audio signals to the user's left and / or right ears (e.g., directional mode or surround mode).

[0102] According to one embodiment of the present disclosure, the sensor circuit (345) (e.g., the sensor module (176) of FIG. 1) may include a motion sensor (not shown) (e.g., an acceleration sensor, a gyro sensor, and / or a magnetic field sensor) for detecting movement (e.g., up / down / left / right head movement) of the electronic device (301) (or the user).

[0103] In one embodiment, the sensor circuit (345) may include a position sensor (not shown) (e.g., global navigation satellite system (GNSS)) for detecting position information (e.g., coordinate information) of the electronic device (301).

[0104] In one embodiment, the sensor circuit (345) may transmit information about the movement of the electronic device (301) obtained through a motion sensor and / or information about the location of the electronic device (301) obtained through a location sensor to the processor (350).

[0105] According to one embodiment of the present disclosure, a processor (350) (e.g., processor (120) of FIG. 1) may include, for example, a microcontroller unit (MCU) and may control a plurality of hardware components connected to the processor (350) by running an operating system (OS) or an embedded software program. The processor (350) may control a plurality of hardware components according to, for example, instructions stored in a memory (320) (e.g., program (140) of FIG. 1).

[0106] FIG. 4 illustrates a situation in which an electronic device (301) according to one embodiment operates in a first mode.

[0107] According to one embodiment, the first mode may refer to a virtual mode. The virtual mode may include a mode for playing a game or conducting a call or conference with another user in a virtual space. When the electronic device (301) operates in the first mode, the processor (e.g., the processor (350) of FIG. 3) may determine that signals generated around the user, excluding the user's voice, are noise. The processor (350) may remove signals other than the signal corresponding to the user's voice from among signals input to the microphone, and activate noise cancellation. The processor (350) may reduce or remove external signals in the surroundings input to the microphone using various methods.

[0108] For example, the electronic device (301) may include multiple microphones at the top and bottom. The microphone located at the bottom of the electronic device (301) may be located relatively closer to the user's mouth. In this case, the user's voice may be input to the microphone located at the bottom relatively first. The user's voice may be input to the microphone located at the top after a certain period of time. The processor (350) may delay the signal from the bottom microphone for a certain period of time and amplify it by adding it to the signal input to the top microphone.

[0109] In the case of a signal input from a direction other than the user's mouth (e.g., forward) of the electronic device (101), the voice may be input simultaneously to the lower microphone and the upper microphone. In this case, if the signal from the lower microphone is delayed by a certain amount of time and added to the signal input to the upper microphone, the phase may be reversed to reduce the sound volume. The processor (350) may reduce the sound volume by adding the phase inversely in the case of a signal input from a direction other than the user's mouth (e.g., forward), and may amplify the sound volume in the case of a signal input from the mouth direction (412). In addition, the processor (350) may distinguish between the user's voice and non-voice signals by using a noise suppressor. The processor (350) may reduce the sound volume by adding the phase inversely to a signal other than the user's voice.

[0110] Various methods may include, for example, gain control, filter use, dynamic range compression (DRC), or noise suppression. Dynamic range compression (DRC) can refer to the process of adjusting or compressing the dynamic range of music and audio signals to suit a specific purpose. This can be used to control the audio signal level by uniformly adjusting the audio so that it is neither too loud nor too soft. Noise suppression can be used to improve speech quality and enhance the user experience in noisy environments.

[0111] According to one embodiment, the processor (350) may activate noise canceling to block external sounds and remove or reduce noise received on the electronic device (301) when operating in the first mode or virtual mode. When operating in the first mode, the processor (350) may operate a microphone (e.g., a bottom microphone) located at the closest distance to the user's mouth with respect to the electronic device (301), in one embodiment. For example, the bottom microphone with respect to the electronic device (301) may refer to a microphone that is relatively closest to the user's mouth. When operating in the first mode, the processor (350) may operate the bottom microphone with respect to the electronic device (301), thereby providing a situation in which the user's voice can be received at the closest distance and / or the reception of external noise can be reduced. The bottom microphone may refer to, for example, the first microphone (211) and / or the second microphone (221) of FIG. 2A. The area where the microphone is placed at the bottom may vary by device and is not fixed, but may refer to the area that is relatively closest to the user's mouth among multiple microphones.

[0112] The processor (350) can remove noise received by the microphone by disabling the ambient sound listening mode.

[0113] According to one embodiment, the processor (350) can operate the microphone at the bottom to control the beam forming direction of the signal output from the speaker to face the bottom (410).

[0114] Ambient sound mode can refer to a mode that amplifies external sounds so that the user can easily hear them. Active Noise Cancellation (ANC) can refer to a function that blocks external sounds, as opposed to ambient sound mode.

[0115] The microphone on the electronic device (301) may be positioned around or inside the user's ear. The processor (350) may use the microphone positioned around or inside the user's ear to generate a signal with an opposite phase to an external signal and cancel it out. The processor (350) may block the user from hearing external signals through the ANC function.

[0116] A noise suppressor can remove noise for the other party on the call, not the user. The processor (350) can distinguish between the user's voice and non-user's voice signals among the signals input through the microphone. The processor (350) can preserve the signal corresponding to the user's voice and remove the non-user's voice signals using the noise suppressor. The processor (350) can control the noise suppressor to transmit only the user's voice to the other party on the call, and prevent the transmission of noise that is not the user's voice.

[0117] When entering virtual mode, the processor (350) can disable the ambient sound listening function and activate the ANC function. When talking to the other party, the processor (350) can select a microphone positioned in the direction of the user's mouth (412), recognize the user's voice, and remove external noise using a noise suppressor. External noise may refer to sounds other than the user's voice.

[0118] FIG. 5 illustrates a situation in which an electronic device (301) according to one embodiment operates in a second mode.

[0119] In one embodiment, the second mode may include a see-through mode. See-through mode may include a mode in which a user can converse or hold a conference with another user in the same space while actually observing the surroundings captured by the camera. See-through mode may include, for example, a situation in which a translation of the voice of a user of another external electronic device is required when exchanging voice signals with the user of the other external electronic device.

[0120] In one embodiment, a processor (e.g., processor (350) of FIG. 3) may display appropriate information (e.g., a translation) to a user of the electronic device (301) while the electronic device (301) is operating in the second mode. In one example, in a situation where a full translation is required, the processor (350) may receive a voice surrounding the user of the electronic device (301) through a microphone, and display information corresponding to the voice surrounding the user, translated into text, to the user using the display (330).

[0121] According to one embodiment, the see-through mode may include a mode that displays to the user the surroundings captured using a camera as they are, and analyzes and processes and displays information about the user's surroundings or actual surrounding space. In one embodiment, the processor (350) may deactivate noise canceling in order to provide information about the actual space as it is when operating in the see-through mode. In one embodiment, the processor (350) may control to amplify surrounding sounds and prevent howling between external electronic devices (e.g., VST devices) (303) since other user devices may exist within a specified distance from the space where the electronic device (301) is located. In the following description, it is assumed that the external electronic device (303) is a VST device, but the type of external electronic device is not limited thereto. Howling may refer to a phenomenon in which a voice generated from one VST device is amplified and noise is generated when the voice is played from the speaker of another VST device and is re-input to the microphone of the other VST device.

[0122] In one embodiment, the processor (350) may disable noise cancellation to receive external sounds when operating in the second mode or see-through mode. In one embodiment, the processor (350) may operate the lower microphone to receive a user's voice signal while simultaneously operating another microphone to receive a voice signal from another user. For example, the other microphone may refer to a microphone separate from the lower microphone. For example, the other microphone may refer to a microphone that is relatively close to the user of the other VST device (303) and is intended to receive a voice signal from a user of the other VST device (303). In one embodiment, the processor (350) may operate multiple microphones based on the number and / or location of users of at least one other VST device located within a specified distance from the electronic device (301). For example, the processor (350) may select and operate some of the multiple microphones based on the location of the user. The processor (350) can operate microphones positioned on the left and right when one user of another VST device is detected on the left side and two users of another VST device are detected on the right side based on the location of the electronic device (301). This is merely an example, and the number or locations of users of other VST devices are not limited thereto.

[0123] According to one embodiment, the processor (350) can change the microphones that are operated based on the location of another VST device (303) located within a specified distance from the electronic device (301). The processor (350) can set the beamforming direction of the signal output from the speaker to a direction (510) toward the counterpart based on the direction of the electronic device (301) (e.g., the direction of a camera (e.g., the camera (325) of FIG. 3) on the electronic device (301) or the direction of a display (e.g., the display (330) of FIG. 3)).

[0124] For example, the processor (350) may operate a microphone that is relatively located (or positioned) farthest to the right on the electronic device (301) based on the fact that another VST device (303) is located (or positioned) to the right of the electronic device (301). The processor (350) may operate a microphone that is relatively farthest to the right on the electronic device (301) to form beam forming in a specific direction (512) based on the position of the other party's mouth.

[0125] For example, the processor (350) may stop the operation of a microphone located (or positioned) relatively farthest to the right on the electronic device (301) and operate a microphone located (or positioned) relatively farthest to the left on the electronic device (310) based on the position of another VST device (303) moving to the left with respect to the electronic device (301). For example, the processor (350) may operate a microphone located relatively farthest to the right and a microphone located relatively farthest to the left on the electronic device (301) based on the positions of other VST devices on the left and right, respectively, with respect to the electronic device (301). The processor (350) may determine a microphone to operate based on the positions and number of external VST devices.

[0126] In one embodiment, the processor (350) may activate a microphone positioned relatively close to the forehead of the user of the electronic device (301) while the user of the electronic device (301) is seated and the user of the other VST device (303) is standing. For example, the processor (350) may also simultaneously activate a microphone positioned relatively close to the mouth of the user of the electronic device (301). The other VST device (303) may also form a beamforming in the direction (514) toward the electronic device (301).

[0127] According to one embodiment, the processor (350) can control the number of microphones receiving signals and the number of channels through which signals are output when operating in the third mode.

[0128] According to one embodiment, the third mode may refer to a mode in which interaction is performed with an external electronic device (303) existing within a specified distance from the electronic device (301) and an external electronic device existing in a different space from the electronic device (301). For example, the third mode may refer to an operating mode of the processor (350) in a situation in which a user of another VST device (303) in a real space within a specified distance from the electronic device (301) attends a meeting and at the same time, a user of another VST device (not shown) in a virtual space attends a meeting. When operating in the third mode, the processor (350) may reflect the characteristics when operating in the first mode (virtual mode) by considering the users of the other VST devices (not shown) in the virtual space, and at the same time, reflect the characteristics when operating in the second mode (see-through mode) by considering the users of the other VST devices (303) in the real space, and manage the microphone through which an external signal is received and the channel through which the signal is output.

[0129] In one embodiment, the processor (350) can eliminate inter-device howling while the electronic device (301) operates in the third mode. For example, howling elimination in the third mode takes into account the users of other VST devices (303) in the real space. For example, the processor (350) can disable noise cancellation taking into account the users of other VST devices (303) in the real space. For example, the processor (350) can display a real space captured by a camera to display the users of other VST devices (303) in the real space, while simultaneously displaying the users of other VST devices (not shown) attending a meeting in the virtual space in the real space.

[0130] In one embodiment, the processor (350) may activate a microphone based on the location of a user of another VST device (303) in a real space, and may activate a microphone based on the location of a user's mouth of the electronic device (301). In addition, the processor (350) may activate noise cancellation upon entering a virtual space, and eliminate howling between devices.

[0131] According to one embodiment, the processor (350) may disable noise canceling and eliminate inter-device howling when a voice input from a user of another VST device (303) in a real space is detected. The processor (350) may activate noise canceling and not eliminate inter-device howling when a voice input from a user of another VST device (not shown) in a virtual space is detected.

[0132] According to one embodiment, the processor (350) may operate a microphone based on the location of the user of another VST device (303) in the real space when a voice input of a user of another VST device (303) in the real space is detected, and may simultaneously operate the microphone based on the location of the mouth of the user of the electronic device (301). The processor (350) may operate only the microphone closest to the mouth of the user of the electronic device (301) based on the location of the mouth of the user of the electronic device (301) when a voice input of a user of another VST device (not shown) in the virtual space is detected.

[0133] The processor (350) can disable noise canceling and eliminate howling between devices in a situation where a voice input of a user of another VST device (303) in a real space and a voice input of a user of another VST device (not shown) in a virtual space are detected together.

[0134] When a conversation partner exists in both a virtual space and a real space, the processor (350) can activate the third mode. When the third mode is in operation, the processor (350) can activate a microphone positioned in the direction of the user's mouth, and simultaneously activate a microphone positioned in front of the electronic device (301). The processor (350) can amplify the user's voice using the microphone positioned in the direction of the user's mouth and transmit it to the conversation partner in the virtual space. The processor (350) can receive, amplify, and / or provide other functions, including translation, the voice of another conversation partner located in the real space using the microphone positioned in front.

[0135] According to one embodiment, the processor (350) may activate the ANC (Active Noise Cancellation) function in the third mode, deactivate the ambient sound listening mode, and activate the howling cancellation function.

[0136] Ambient sound mode can refer to a mode that amplifies external sounds so that the user can easily hear them. Active Noise Cancellation (ANC) can refer to a function that blocks external sounds, as opposed to ambient sound mode.

[0137] A noise suppressor can remove noise for the other party on the call, not the user. The processor (350) can distinguish between the user's voice and non-user's voice signals among the signals input through the microphone. The processor (350) can preserve the signal corresponding to the user's voice and remove the non-user's voice signals using the noise suppressor. The processor (350) can control the noise suppressor to transmit only the user's voice to the other party on the call, and prevent the transmission of noise that is not the user's voice.

[0138] Fig. 6 is a flowchart illustrating a method for controlling audio signal output of an electronic device according to one embodiment.

[0139] The operations described through FIG. 6 can be implemented based on instructions that can be stored in a computer recording medium or memory (e.g., memory (130) of FIG. 1). The illustrated method (600) can be executed by an electronic device (e.g., electronic device (301) of FIG. 3) described above through FIGS. 1 to 5, and the technical features described above will be omitted below. The order of each operation of FIG. 6 can be changed, some operations can be omitted, and some operations can be performed simultaneously.

[0140] In operation 610 of FIG. 6, a processor (e.g., processor (350) of FIG. 3) may operate in either a first mode of displaying a virtual space on a display (e.g., display (330) of FIG. 3) or a second mode of displaying a screen captured using a camera (e.g., camera (325) of FIG. 3) on the display based on a specified condition.

[0141] According to one embodiment, the first mode may include a virtual mode. The virtual mode may include a mode for playing a game or conducting a call or conference with another user in a virtual space. In the virtual mode, the processor (350) may determine that signals generated around the user, excluding the user's voice, are noise. The processor (350) may limit signals input to a microphone (e.g., the first microphone (211) and / or the second microphone (221) of FIG. 2A) to the user's voice and activate noise cancellation.

[0142] In one embodiment, the second mode may include a see-through mode. See-through mode may refer to a mode in which a user can converse or hold a meeting with another user in the same space while actually observing the surroundings captured by the camera (325). See-through mode may include, for example, situations requiring interpretation and translation.

[0143] In operation 620 of FIG. 6, the processor (350) may determine differently the positions of microphones activated for external signal input among a plurality of microphones (e.g., the first microphone (211) and / or the second microphone (221) of FIG. 2A) disposed on the electronic device (301) based on whether either the first mode or the second mode is operated, and may determine differently the number of microphones activated for external signal input among a plurality of microphones disposed on the electronic device (301). The processor (350) may determine differently whether to activate functions related to amplification or blocking of external sounds, and may determine differently the direction of beamforming generation for receiving voice signals of the microphone.

[0144] According to one embodiment, the processor (350) may activate a microphone positioned relatively closest to the user's mouth when the electronic device (301) is worn based on operation in the first mode. The first mode may refer to a mode for performing communication with other users in a virtual space. When performing communication with other users in a virtual space, the processor (350) may activate a microphone positioned relatively closest to the user's mouth so that external sounds are blocked and only the voice of the user of the electronic device (301) can be input into the microphone. In addition, the processor (350) may deactivate microphones positioned in other remaining locations.

[0145] According to one embodiment, the processor (350) may, based on operation in the first mode, disable the function of amplifying external sounds input through the microphone and reduce the output of external sounds by simultaneously playing a waveform of opposite phase based on the waveform of the external sounds input through the microphone. The processor (350) may block external sounds when communicating with other users in the virtual space.

[0146] According to one embodiment, the processor (350) may determine a beamforming generation direction based on the direction of the user's mouth so as to receive a sound coming from the user's mouth based on the operation in the first mode. The processor (350) may determine a beamforming generation direction based on the direction of the user's mouth so as to transmit only the user's voice to the other party and block out the remaining external sounds when communicating with another user in a virtual space. The processor (350) may generate beamforming based on the direction of the user's mouth so that only the user's voice is input into the microphone and the remaining external sounds are not input.

[0147] According to one embodiment, the processor (350) may activate a microphone positioned relatively closest to the user's mouth when the electronic device (301) is worn based on its operation in the second mode, and at the same time, activate a microphone positioned most forward relative to the user wearing the electronic device (301). The second mode may refer to a mode for communicating with a user of another head mounted display (HMD) device within a specified distance relative to the position of the electronic device (101) in a real space rather than a virtual space. In this case, the processor (350) may activate not only the lower microphone positioned relatively closest relative to the user's mouth, but also the front microphone positioned closest to another user.

[0148] The electronic device (101) can receive the voice of a user of another HMD device using the front microphone and display the user's voice as text on the display (330). In addition, the electronic device (101) can translate and display the language of the other user. The processor (350) can deactivate microphones located in other locations besides the bottom and front microphones.

[0149] According to one embodiment, the processor (350) can amplify the output for the external sound by reproducing a waveform of the same phase based on the waveform of the external sound input through the microphone based on the operation in the second mode.

[0150] According to one embodiment, the processor (350) may determine a beamforming generation direction based on the direction of the user's mouth so that the sound coming from the user's mouth can be input when the electronic device is worn based on operation in the second mode. In addition, the processor (350) may generate beamforming in a forward direction based on the user so that the sound of the other party can be input based on the user when the electronic device is worn.

[0151] The processor (350) can generate beamforming based on the direction of the user's mouth so that the user's voice can be input to the microphone. The processor (350) can also generate beamforming in a forward direction based on the user so that the voice of the user of another HMD device located in the actual space can be received.

[0152] According to one embodiment, the processor (350) may operate in the second mode to activate a function for eliminating howling when an external device (e.g., an HMD device) is positioned within a specified distance from the electronic device (301). Howling may refer to a feedback phenomenon that occurs when audio output from an external device is input to the microphone of the electronic device (301).

[0153] According to one embodiment, the processor (350) detects whether another external device exists within a specified distance based on the location of the electronic device (301) based on operation in the second mode, and determines the location of the other party by comparing at least one of the magnitude, phase, or arrival time of signals received by the electronic device (301). The processor (350) can determine the location of a microphone to be activated for external signal input among a plurality of microphones arranged on the electronic device (301) based on the location of the user of the other external device. The processor (350) can activate a microphone close to the user of the external device and deactivate microphones at the remaining locations.

[0154] According to one embodiment, the processor (350) may determine the position of the microphone to be activated in a situation where the processor (350) operates in the first mode to perform communication with a user of a first external device in a virtual space and simultaneously perform communication with a user of a second external device within a specified distance in a real space. The processor (350) may activate the microphone that is positioned relatively closest to the user's mouth when the electronic device is worn, activate the microphone that is positioned most forward with respect to the user wearing the electronic device, and deactivate the microphones positioned in other positions. The processor (350) may deactivate the function of amplifying external sound input through the microphone, and may activate the function of reducing the output of the external sound by playing back a waveform of an opposite phase based on the waveform of the external sound input through the microphone, and eliminating howling. Howling may refer to a feedback phenomenon that occurs when audio output from the second external device is input to the microphone of the electronic device.

[0155] According to one embodiment, the processor (350) may activate noise canceling to block external sounds and block noise received on the electronic device (301) when operating in the first mode or virtual mode. When operating in the first mode, the processor (350) may operate a microphone (e.g., a bottom microphone) that is located closest to the user with respect to the electronic device (301). The bottom microphone with respect to the electronic device (301) may refer to a microphone that is relatively closest to the user's mouth. When operating in the first mode, the processor (350) may operate the bottom microphone with respect to the electronic device (301), thereby receiving the user's voice at the closest distance and creating an environment in which external noise is not received by the microphone. The bottom microphone may refer to, for example, the first microphone (211) and / or the second microphone (221) of FIG. 2A.

[0156] According to one embodiment, the processor (350) may receive the user's surrounding voice through a microphone and display translated information to the user when operating in the second mode. The see-through mode may include a mode that displays the surroundings captured by the camera to the user as is, and analyzes and processes information about the user's surroundings or the actual surrounding space and displays them. When operating in the see-through mode, the processor (350) may deactivate noise canceling to provide information about the actual space as is. For example, the processor (350) may amplify surrounding sounds and control to prevent howling between VST devices, as other user devices may exist within a specified distance from the space where the electronic device (301) is located.

[0157] According to one embodiment, the specified condition may include at least one of whether a camera for capturing the outside is operating, whether a specified application that can utilize the virtual space is running, whether a specified area is entered, or a user's selection. For example, the processor (350) may activate either a first mode for displaying a virtual screen on the display or a second mode for displaying a screen captured using the camera on the display based on the user's selection. For example, the processor (350) may activate the first mode for displaying a virtual screen on the display based on the execution of a specified application that can utilize the virtual space, and may activate the second mode for displaying a screen captured using the camera on the display based on the non-execution of the specified application that can utilize the virtual space. For example, the processor (350) may execute a first mode for displaying a virtual screen on the display (330) based on whether the location of the electronic device (301) is within a pre-specified area, and may execute a second mode for displaying a screen captured using the camera (325) on the display (330) based on whether the location of the electronic device (301) is not within a pre-specified area.

[0158] According to one embodiment, the processor (350) activates active noise cancellation (ANC) based on the selection of the first mode for displaying a virtual screen on the display to block out ambient noise of the electronic device, operates a microphone located closest to the mouth position of the user of the electronic device, and sets the direction in which a signal of the audio output circuit is output to the bottom based on the mouth position of the user of the electronic device.

[0159] According to one embodiment, the processor (350) controls the electronic device to not block external sounds around the electronic device by suspending the operation of active noise cancellation (ANC) based on the selection of the second mode that displays a screen captured using a camera, and can detect whether another external device exists within a specified distance based on the location of the electronic device. The processor (350) can activate the closest microphone among microphones arranged on the electronic device based on the location of the user of the other external device. The processor (350) can set the direction in which a signal of an audio output circuit is output to the front based on the mouth location of the user of the electronic device, and can remove howling between a voice signal output from an external device and a voice signal output from the electronic device.

[0160] According to one embodiment, the processor (350) may activate a microphone located relatively farthest to the right on the electronic device based on another VST device being located to the right with respect to the electronic device, and may stop the operation of a microphone located relatively farthest to the right on the electronic device and activate a microphone located relatively farthest to the left based on another VST device's position moving to the left with respect to the electronic device.

[0161] According to one embodiment, the processor (350) may deactivate the ambient noise cancellation function and activate active noise cancellation (ANC) based on the establishment of a communication connection with another video see through (VST) device located within a specified distance from the electronic device and the establishment of a communication connection with another video see through (VST) device in a virtual space. Active noise cancellation (ANC) may refer to a function of blocking out ambient sounds for a user of the electronic device (101). Ambient noise cancellation may refer to a function of blocking noise around the electronic device (101) from being mixed into audio for a user of the other VST device. The processor (350) may remove howling between a voice signal output from another VST device and a voice signal output from the electronic device. The processor (350) operates a plurality of microphones, including the microphone closest to the user's mouth position and the microphone closest to the other party's mouth position, among the microphones included in the electronic device, and the direction in which a signal of an audio output circuit is output is set downward based on the user's mouth position of the electronic device, and at the same time, two signal channels can be received by setting the direction downward based on the user's mouth position of the electronic device.

[0162] According to one embodiment, the processor (350) may disable noise canceling and eliminate howling between devices when a voice input of a user of an external electronic device adjacent to the electronic device in a real space is detected. The processor (350) may activate noise canceling for a voice of another user who accesses a virtual space through another external electronic device and does not apply howling cancellation between devices. When a voice input of a user of an external electronic device adjacent to the electronic device is detected, the processor may operate a microphone based on a location of the external electronic device adjacent to the electronic device, and simultaneously operate a microphone based on a location of a mouth of the user of the electronic device (301). When a voice input of a user of another external electronic device who accesses a virtual space and communicates is detected, the processor may operate a microphone closest to the mouth of the user of the electronic device (301) based on the location of the mouth of the user of the electronic device (301).

[0163] According to one embodiment, the processor (350) may control the electronic device to stop the operation of active noise cancellation (ANC) based on the selection of the second mode that displays the actual screen recognized by the camera so as not to block out external sounds around the electronic device, and may amplify the external sounds and / or recognize and translate the external sounds and display them on the display.

[0164] According to one embodiment, the processor (350) detects whether another external device exists within a specified distance based on the location of the electronic device (301), and determines the location of the other party by comparing at least one of the magnitude, phase, or arrival time of signals received on a plurality of microphones. The processor (350) activates the closest microphone among the microphones arranged on the electronic device based on the location of the user of the other external device, sets the direction in which a signal of an audio output circuit is output to the front based on the mouth location of the user of the electronic device, and can eliminate howling between a voice signal output from an external device and a voice signal output from the electronic device. The processor (350) activates a microphone located relatively farthest to the right on the electronic device (301) based on the other VST device being located to the right with respect to the electronic device (301), and stops the operation of a microphone located relatively farthest to the right on the electronic device (301) and activates a microphone located relatively farthest to the left based on the location of the other VST device moving to the left with respect to the electronic device (301).

[0165] According to one embodiment, the processor (350) may operate a function to eliminate howling when an external device is located within a specified distance from the electronic device based on operation in the second mode.

[0166] According to one embodiment, the processor (350) detects whether another external device exists within a specified distance based on the location of the electronic device based on operation in the second mode, determines the location of the other party by comparing at least one of the magnitude, phase, or arrival time of signals received by the electronic device, and determines the location of a microphone activated for external signal input among a plurality of microphones arranged on the electronic device based on the location of the user of the other external device.

[0167] According to one embodiment, the specified conditions for operating in either the first mode or the second mode may include at least one of a user selection, whether a camera photographing the outside is in operation, whether a specified application capable of utilizing the virtual space is running, whether entry into a specified area is made, or a user selection.

[0168] According to one embodiment, the processor (350) may execute a first mode for displaying a virtual screen on the display based on whether the location of the electronic device is within a pre-specified area, and may execute a second mode for displaying an actual screen recognized by the camera on the display based on whether the location of the electronic device is not within the pre-specified area.

[0169] According to one embodiment, the processor (350) may operate in a first mode to communicate with a user of a first external device in a virtual space and, at the same time, communicate with a user of a second external device within a specified distance in a real space, activate a microphone positioned at a location relatively closest to the user's mouth when the electronic device is worn, and activate a microphone positioned at the frontmost position with respect to the user wearing the electronic device. In addition, the processor (350) may deactivate microphones positioned at other locations in the same situation, deactivate a function for amplifying external sound input through the microphone, and activate a function for reducing output of external sound and eliminating howling by playing back a waveform of an opposite phase based on a waveform of an external sound input through the microphone.

[0170] According to one embodiment, the electronic device (101) may include a head mounted display (HMD) device that is mounted on the user's head and displays virtual reality or augmented reality.

[0171] While the present invention has been described and illustrated with reference to various embodiments, it should be understood that these embodiments are illustrative and not restrictive. Furthermore, those skilled in the art will appreciate that various changes in form and detail may be made without departing from the spirit and scope of the present invention. Such changes are encompassed by the appended claims and their equivalents. Furthermore, it should be understood that any embodiment described herein can be used in conjunction with any other embodiment described herein.

Claims

1. In electronic devices, A display that is placed around the user's eyes when worn on the user's body; Multiple microphones for external signal input; camera; Speaker for signal reproduction; at least one processor; and Includes a memory (130) for storing instructions, The above instructions, when executed by the at least one processor, cause the electronic device to: Based on the execution of instructions stored in the above memory Activating either a first mode for displaying a virtual space on the display or a second mode for displaying an actual space captured by the camera on the display according to specified conditions; Based on which of the above first or second modes is operated Among the multiple microphones arranged on the above electronic device, the position of the microphone activated for external signal input is determined differently, The number of microphones activated for external signal input among the plurality of microphones arranged on the above electronic device is determined differently, Differently determines whether functions related to amplification or blocking of external sounds are activated, An electronic device that controls the direction of beamforming generation for receiving voice signals from the above microphone.

2. In paragraph 1, The above instructions, when executed by the at least one processor, cause the electronic device to: Based on the operation in the above first mode When the above electronic device is worn, the microphone located relatively closest to the user's mouth is activated, The remaining microphones placed in other locations are disabled. The function of amplifying external sound input through the above microphone is disabled, By playing back a waveform of the opposite phase based on the waveform of the external sound input through the above microphone, the output of the external sound is reduced. An electronic device that controls the direction of beamforming generation based on the direction of the user's mouth so that sound coming from the user's mouth can be input when the electronic device is worn.

3. In paragraph 1, The above instructions, when executed by the at least one processor, cause the electronic device to: Based on the operation in the above second mode When the above electronic device is worn, the microphone located relatively closest to the user's mouth is activated, Activates the microphone located at the front of the user wearing the above electronic device, The remaining microphones placed in other locations are disabled. Based on the waveform of the external sound input through the above microphone, the output for the external sound is amplified by playing back the waveform of the same phase. When the above electronic device is worn, the beamforming generation direction is determined based on the direction of the user's mouth so that the sound coming from the user's mouth can be input, and An electronic device that controls beamforming to be generated in a forward direction with respect to the user so that the user can receive sound input from the other party when wearing the above electronic device.

4. In paragraph 3, The above instructions, when executed by the at least one processor, cause the electronic device to: Based on the operation in the above second mode Controls the function of eliminating howling when an external device is located within a specified distance from the electronic device; The above howling An electronic device that refers to a feedback phenomenon that occurs when audio output from the external device is input to the microphone of the electronic device.

5. In paragraph 1, The above instructions, when executed by the at least one processor, cause the electronic device to: Based on the operation in the above second mode Detects whether another external device exists within a specified distance based on the location of the electronic device; By comparing at least one of the magnitude, phase or arrival time of the signals received by the electronic device, the location of the other party is determined, An electronic device that controls the position of a microphone to be activated for external signal input among a plurality of microphones arranged on the electronic device based on the position of a user of another external device.

6. In paragraph 1, Specified conditions for operating in either the first mode or the second mode are An electronic device that includes at least one of the following: a user's choice, whether a camera is operating to capture the outside, whether a designated application that can utilize the virtual space is running, whether to enter a designated area, or a user's choice.

7. In paragraph 1, The above instructions, when executed by the at least one processor, cause the electronic device to: A first mode is executed to display a virtual screen on the display based on the execution of a designated application that can utilize the virtual space, An electronic device that controls execution of a second mode for displaying an actual screen recognized by the camera on the display based on the fact that a designated application capable of utilizing the virtual space is not running.

8. In paragraph 1, The above instructions, when executed by the at least one processor, cause the electronic device to: A first mode is executed for displaying a virtual screen on the display based on the location of the electronic device being within a pre-designated area, An electronic device that controls to execute a second mode for displaying an actual screen recognized by the camera on the display based on the location of the electronic device not being within a pre-designated area.

9. In paragraph 1, The above instructions, when executed by the at least one processor, cause the electronic device to: Operating in the first mode, it communicates with a user of a first external device in virtual space, In a situation where communication is performed with a user of a second external device within a specified distance in real space at the same time, When the above electronic device is worn, the microphone located relatively closest to the user's mouth is activated, Activates the microphone located at the front of the user wearing the above electronic device, The remaining microphones placed in other locations are disabled. The function of amplifying external sound input through the above microphone is disabled, By playing back a waveform of the opposite phase based on the waveform of the external sound input through the above microphone, the output of the external sound is reduced. Controls the operation of the function to remove howling, The above howling An electronic device that refers to a feedback phenomenon that occurs when audio output from the second external device is input to the microphone of the electronic device.

10. In paragraph 1, The above electronic device An electronic device including a head mounted display (HMD) device that is mounted on the user's head and displays virtual reality or augmented reality.

11. In the method of operating an electronic device, An operation of operating either a first mode for displaying a virtual space on a display according to specified conditions or a second mode for displaying an actual space captured by a camera on the display; Based on which of the above first or second modes is operated Among the multiple microphones arranged on the above electronic device, the position of the microphone activated for external signal input is determined differently, The number of microphones activated for external signal input among the plurality of microphones arranged on the above electronic device is determined differently, Differently determines whether functions related to amplification or blocking of external sounds are activated, A method including an operation of differently determining a beamforming generation direction for receiving a voice signal of the above microphone.

12. In paragraph 11, Based on the operation in the above first mode An action of activating a microphone positioned relatively closest to the user's mouth when wearing the above electronic device; Action to disable the remaining microphones placed in other locations; The function of amplifying external sound input through the above microphone is disabled, An operation of reducing the output of an external sound by playing back a waveform of the opposite phase based on the waveform of an external sound input through the above microphone; A method further comprising an action of determining a beamforming generation direction based on the direction of the user's mouth so as to receive sound coming from the user's mouth when the electronic device is worn.

13. In paragraph 11, Based on the operation in the above second mode An action of activating a microphone positioned relatively closest to the user's mouth when wearing the above electronic device; An action of activating the microphone located at the frontmost position based on the user wearing the electronic device and deactivating the microphones located at other positions; An operation of amplifying the output of an external sound by playing back a waveform of the same phase based on the waveform of an external sound input through the above microphone; An action for determining the direction of beamforming generation based on the direction of the user's mouth so that the sound coming from the user's mouth can be input when the above electronic device is worn; and A method further comprising an action of generating beamforming in a forward direction with respect to the user so that the other party's sound can be input with respect to the user when the electronic device is worn.

14. In paragraph 13, Based on the operation in the above second mode Further comprising an action of activating a function for eliminating howling when an external device is located within a specified distance from the electronic device; The above howling A method referring to a feedback phenomenon that occurs when audio output from the external device is input to the microphone of the electronic device.

15. In paragraph 11, Based on the operation in the above second mode An action to detect whether another external device exists within a specified distance based on the location of the electronic device; An operation of determining the location of the other party by comparing at least one of the magnitude, phase or arrival time of signals received by the electronic device; and A method further comprising the action of determining a position of a microphone to be activated for external signal input among a plurality of microphones arranged on the electronic device based on a position of a user of another external device.

Images