Head-mounted device for outputting sound and operation method thereof

Abstract

A head-mounted device according to an embodiment may comprise: a display; an embedded speaker configured to output sound; a communication circuit; a memory for storing executable instructions; and at least one processor for accessing the memory and executing the instructions. The at least one processor may be configured to: establish a communication connection with at least one external device including a speaker module through the communication circuit; display an object at a specific position in a three-dimensional space through the display, wherein the object includes corresponding first sound data; identify at least a part of the at least one external device on the basis of the specific location at which the object is displayed; and transmit the first sound data to the at least a part of the at least one external device through the communication circuit.

Description

Head-mounted device for outputting sound and method of operating the same

[0001] The present disclosure relates to a head-mounted device for outputting sound according to one embodiment and a method of operating the same.

[0002] With the advancement of electronic technology, various types of electronic devices, such as mobile terminals, personal digital assistants (PDAs), electronic notebooks, smartphones, tablet PCs (personal computers), and wearable devices, are being widely used. For example, electronic devices can provide virtual reality (VR), which allows users to have a realistic experience in a computer-generated virtual world; augmented reality (AR), which adds virtual information (or objects) to the real world; and mixed reality (MR), which combines virtual reality and augmented reality.

[0003] Wearable type electronic devices that are worn by the user, such as head-mounted devices, require compact space utilization, and to utilize the space, a technology may be required to provide light corresponding to the screen display of the display to the user's eyes through a lens.

[0004] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

[0005] A head-mounted device according to one embodiment of the present disclosure may include a display, a built-in speaker configured to output sound, a communication circuit, a memory storing executable instructions, and at least one processor configured to access the memory and execute the instructions. The at least one processor may be configured to establish a communication connection with at least one external device including a speaker module through the communication circuit. The at least one processor may be configured to display an object at a specific location in a three-dimensional space through the display. The object may include corresponding first sound data. The at least one processor may be configured to identify at least a portion of the at least one external device based on the specific location at which the object is displayed. The at least one processor may be configured to transmit the first sound data to at least a portion of the at least one external device through the communication circuit.

[0006] According to one embodiment of the present disclosure, a method of operating an electronic device may include an operation of establishing a communication connection with at least one external device including a speaker module through a communication circuit. According to one embodiment, a method of operating an electronic device may include an operation of displaying an object at a specific location in a three-dimensional space through a display. The object may include corresponding first sound data. According to one embodiment, a method of operating an electronic device may include an operation of identifying at least some of the at least one external device based on the specific location at which the object is displayed. According to one embodiment, a method of operating an electronic device may include an operation of transmitting the first sound data to at least some of the at least one external device through the communication circuit.

[0007] A non-transitory computer-readable storage medium storing one or more programs according to one embodiment of the present disclosure may include an operation of establishing a communication connection with at least one external device including a speaker module through a communication circuit based on execution of an application. The storage medium according to one embodiment may include an operation of displaying an object at a specific location in a three-dimensional space through a display. The object may include corresponding first sound data. The storage medium according to one embodiment may include an operation of identifying at least some of the at least one external device based on the specific location at which the object is displayed. The storage medium according to one embodiment may include an operation of transmitting the first sound data to at least some of the at least one external device through the communication circuit.

[0008] A head-mounted device according to one embodiment of the present disclosure may include a display, a built-in speaker configured to output sound, a communication circuit, a memory storing executable instructions, and at least one processor accessing the memory to execute the instructions. The at least one processor may be configured to execute an application or task generating sound data based on a first input. The at least one processor may be configured to display an object at a specific location in a three-dimensional space through the display based on the execution of the application or the task. The object may correspond to a sound source of at least a portion of the sound data. The at least one processor may be configured to output at least a portion of the sound data through the built-in speaker based on the specific location at which the object is displayed or the display form of the object. The at least one processor may be configured to transmit a control signal, which outputs at least a portion of the sound data, to the at least one external device through the communication circuit based on the specific location at which the object is displayed or the display form of the object.

[0009] According to one embodiment of the present disclosure, a method of operating an electronic device may include an operation of executing an application or task that generates sound data based on a first input. According to one embodiment, a method of operating an electronic device may include an operation of displaying an object at a specific location in a three-dimensional space through a display based on the execution of the application or the task. The object may correspond to a sound source of at least a portion of the sound data. According to one embodiment, a method of operating an electronic device may include an operation of outputting at least a portion of the sound data through a built-in speaker based on the specific location at which the object is displayed or the display form of the object. According to one embodiment, a method of operating an electronic device may include an operation of transmitting a control signal for outputting at least a portion of the sound data to at least one external device through a communication circuit based on the specific location at which the object is displayed or the display form of the object.

[0010] A non-transitory computer-readable storage medium storing one or more programs according to one embodiment of the present disclosure may include an operation of executing an application or task that generates sound data based on a first input. According to one embodiment, the storage medium may include an operation of displaying an object at a specific location in a three-dimensional space through a display based on the execution of the application or the task. The object may correspond to a sound source of at least a portion of the sound data. According to one embodiment, the storage medium may include an operation of outputting at least a portion of the sound data through a built-in speaker based on the specific location at which the object is displayed or the display form of the object. According to one embodiment, the storage medium may include an operation of transmitting a control signal for outputting at least a portion of the sound data to at least one external device through a communication circuit based on the specific location at which the object is displayed or the display form of the object.

[0011] FIG. 1 is a block diagram of an electronic device within a network environment according to one embodiment of the present disclosure.

[0012] FIG. 2 is a front perspective view of a head mount device according to one embodiment of the present disclosure.

[0013] FIG. 3 is a rear perspective view of a head mount device according to one embodiment of the present disclosure.

[0014] FIG. 4A is a software block diagram of a head mounted device according to one embodiment of the present disclosure.

[0015] FIG. 4b illustrates a software architecture of a head mounted device according to one embodiment of the present disclosure.

[0016] FIG. 5 is a block diagram of a head mounted device according to one embodiment of the present disclosure.

[0017] FIG. 6 is a state diagram illustrating a head mounted device in communication connection with an external device according to one embodiment of the present disclosure.

[0018] FIGS. 7A, 7B and 7C illustrate sound rendering types of built-in speakers according to various embodiments of the present disclosure.

[0019] FIG. 8 is a state diagram of a state in which a head-mounted device outputs sound according to one embodiment of the present disclosure.

[0020] FIG. 9a, FIG. 9b, and FIG. 9c are state diagrams illustrating a state in which a head mounted device displays an object on a first space according to one embodiment of the present disclosure.

[0021] FIG. 10A, FIG. 10B, and FIG. 10C illustrate user input methods according to various embodiments of the present disclosure.

[0022] FIGS. 11A, 11B, and 11C are state diagrams illustrating an operation of a head mounted device moving an object from a first space to a second space according to one embodiment of the present disclosure.

[0023] FIG. 12 and FIG. 13a illustrate a state in which the display form of an object is changed according to one embodiment of the present disclosure.

[0024] FIG. 13b illustrates a state in which sound is output using an external device of a head-mounted device according to one embodiment of the present disclosure.

[0025] FIG. 14 is a state diagram illustrating a state in which an event that generates sound data having a priority occurs according to one embodiment of the present disclosure.

[0026] FIG. 15 is a state diagram illustrating a state in which sound data corresponding to each of a plurality of objects is generated according to one embodiment of the present disclosure.

[0027] FIG. 16 is a state diagram illustrating a sound output state according to an object display position according to one embodiment of the present disclosure.

[0028] FIG. 17 is a state diagram illustrating a state of separating the sound of an object according to one embodiment of the present disclosure.

[0029] FIG. 18a and FIG. 18b are flowcharts of a method of operating a head mounted device according to one embodiment of the present disclosure.

[0030] FIG. 19 is a flowchart of an operating method of a head mounted device according to one embodiment of the present disclosure.

[0031] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to one embodiment of the present disclosure.

[0032] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0033] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0034] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0035] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0036] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0037] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0038] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0039] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0040] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0041] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0042] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0043] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0044] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0045] The camera module (180) can capture still images and moving images. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0046] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0047] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0048] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0049] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0050] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0051] In one embodiment, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0052] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0053] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0054] Fig. 2 is a front perspective view of a head mounted device (200) according to one embodiment of the present disclosure. Fig. 3 is a rear perspective view of a head mounted device (200) according to one embodiment of the present disclosure.

[0055] Referring to FIGS. 2 and 3, a head mounted device (200, e.g., the electronic device (101) of FIG. 1) according to one embodiment may include a HeT camera (210), a face tracking camera (215), a VST camera (220), a display member (225), a gaze tracking camera (227), a depth sensor (230), a speaker (240), and / or a microphone (245). The head mounted device (200) according to one embodiment may further include a face cover (not shown) to increase a wearing comfort, foam cushioning (not shown) to protect the head, and / or glasses temples (not shown) to secure the head mounted device (200) to the head.

[0056] According to one embodiment, the HeT camera (210) may be positioned to face the front of the head mounted device (200) and may be positioned at a plurality of locations (e.g., four locations). In one embodiment, the HeT camera (210) may be used for hand detection and tracking, and user gesture (e.g., hand movement) recognition. According to one embodiment, the HeT camera (210) may be used for 3 degrees of freedom (3DoF), 6DoF head tracking, location (spatial, environmental) recognition, and / or movement recognition to provide a mixed reality experience.

[0057] A face tracking camera (215) according to one embodiment may be used to recognize a user's face, or may recognize and / or track both eyes of the user. In one embodiment, the face tracking camera (215) may detect movement of the user's face.

[0058] A face tracking camera (215) according to one embodiment may be equipped with an infrared LED (217) for the face tracking camera that irradiates infrared rays toward the user's face.

[0059] According to one embodiment, the VST camera (220) can capture external images while the user is wearing the head-mounted device (200). In one embodiment, the VST camera (220) can acquire images related to the surrounding environment of the head-mounted device (200). In one embodiment, images of the surrounding environment captured through the VST camera (220) can be provided to implement mixed reality.

[0060] According to one embodiment, the VST camera (220) may be positioned at a position corresponding to the display member (225) (e.g., lens) below, facing in the opposite direction of the display member (225).

[0061] A display member (225) according to one embodiment may be configured to display an image and may include a lens that guides an image generated by a light output module. In one embodiment, the lens may be a Fresnel lens or a pancake lens.

[0062] A display member (225) according to one embodiment can display an image corresponding to mixed reality in front of a user's eyes. In one embodiment, the display member (225) can include at least one of a liquid crystal display (LCD), a digital mirror device (DMD), a liquid crystal on silicon (LCoS), a light emitting diode (LED) on silicon (LEDoS), an organic light emitting diode (OLED), or a micro light emitting diode (micro LED).

[0063] According to one embodiment, the display member (225) may further include a vision correction lens for correcting the vision of the user's eyes, and may include a vision adjustment wheel (223) that can adjust the vision correction lens to correspond to the user's vision.

[0064] According to one embodiment, a gaze tracking camera (227) detects a user's eyes, and accordingly, the head-mounted device (200) can adjust a screen displayed through a display member (225) based on the detected user's eyes or gaze. In one embodiment, the gaze tracking camera (227) can track the user's gaze or recognize an iris included in the eye. In one embodiment, the gaze tracking camera (227) may further include an infrared (IR) light source.

[0065] A depth sensor (230) according to one embodiment may be configured to transmit a signal and receive a signal reflected from a subject, and may be used for the purpose of determining the distance to an object, such as a time of flight (TOF) sensor. Instead of or in addition to the depth sensor (230), camera modules may determine the distance to an object.

[0066] A proximity sensor (235) according to one embodiment may be configured to detect whether the user is wearing the device. In one embodiment, the proximity sensor (235) may be positioned so as to face the user and be close to a display member (225) positioned to correspond to the user's eyeball, thereby detecting whether the user's face is positioned in close proximity.

[0067] A speaker (240) according to one embodiment can output a sound that can be heard by a user.

[0068] According to one embodiment, a microphone (245) can receive a user's voice or surrounding sounds. In one embodiment, the head-mounted device (200) can record sounds received through the microphone (245).

[0069] A head mounted device (200) according to one embodiment may include a power button (250) and a volume button (255) capable of receiving input from a user. In one embodiment, the power button (250) may be configured to turn the power of the head mounted device (200) on or off by a long-press input. In one embodiment, the power button (250) may be configured to turn the screen of the head mounted device (200) on or off by a short-press input. In one embodiment, the power button (250) may be configured to recognize a fingerprint corresponding to a user's finger in contact. In one embodiment, the volume button (255) may be configured to adjust the volume of the speaker (240) by a short-press input.

[0070] In one embodiment, the head mounted device (200) can execute an operation specified by a combination of inputs corresponding to the power button (250) and the volume button (255). For example, based on an input of simultaneously pressing the power button (250) and the volume button (255) (e.g., the down button) for more than 7 seconds, the head mounted device (200) can execute a reboot.

[0071] A head mounted device (200) according to one embodiment may further include a circuit board to which various components are connected, a battery bracket for fixing a battery, and a battery for charging and / or discharging power.

[0072] A head mounted device (200) according to one embodiment may further include a ventilation hole (253) for discharging heat generated inside, a FAN for circulating air to dissipate heat through the ventilation hole, and / or a status indicator light (257) for displaying the current status of the head mounted device (200) through light output. A head mounted device (200) according to one embodiment may further include a connection jack (260) for connecting to an external connector, and / or a Pogo mounting groove (270) formed to be in contact with a Pogo pin provided in a case device when mounted on the case device.

[0073] As described above, according to one embodiment, the head-mounted device (200) may have a form factor for being worn on a user's head. For example, the head-mounted device (200) may further include a strap and / or a wearing member for being secured to a body part of the user. In one embodiment, the head-mounted device (200) may provide a user experience based on augmented reality, virtual reality, and / or mixed reality while being worn on the user's head.

[0074] FIG. 4A is a software block diagram of a head mounted device (200) according to one embodiment of the present disclosure. FIG. 4B illustrates the software architecture of a head mounted device (200) according to one embodiment of the present disclosure.

[0075] Referring to FIGS. 4A and 4B , a head-mounted device (200) according to one embodiment may include a software configuration for controlling a hardware device based on a platform providing a virtual space. In one embodiment, the head-mounted device (200) may include a speaker (400) as a hardware device, and may generate data (e.g., sound data) to control the hardware device using an interface, an application, and / or a user interface (UI).

[0076] The software architecture of a head mounted device (200) according to one embodiment may be implemented by a processor (e.g., processor (120) of FIG. 1 or processor (510) of FIG. 5).

[0077] An operating system (OS) (410) for XR (extended reality) according to one embodiment can receive data or signals from an input device (420, e.g., camera, controller, keyboard / mouse, sensor) and output data or signals to an output device (440, 450, e.g., speaker, display).

[0078] In one embodiment, the OS (410) transmits an audio signal (e.g., Spatial Audio) to an audio controller (440) that controls a speaker (400), and the audio controller (440) can output sound through the speaker (400) based on the received audio signal.

[0079] A configuration based on an embodiment of the present invention may be configured as follows. The illustrated structure is only an example, and at least some of the illustrated configuration may be changed depending on the platform included in the electronic device.

[0080] In one embodiment, Spatial Audio can be interpreted as 360-degree sound and can reproduce the sound effect of the overall head mounted device (200). In one embodiment, when Spatial Audio receives an audio format composed of a multi-channel audio signal (e.g., 5.1 channels or more) or an object signal, it detects the user's head movement detected by an inertial sensor (e.g., an IMU sensor, a gyro sensor, or an acceleration sensor) included in the head mounted device (200) and realizes the illusion that virtual sounds corresponding to the image are provided in a virtual space provided by the head mounted device (200), thereby enhancing the immersive feeling of viewing the image.

[0081] In one embodiment, Spatial Audio can give users the impression that sound fills the entire room. Spatial Audio can operate in a way that mimics "surround sound," as if it were coming from multiple directions simultaneously, much like in real life.

[0082] For example, in the virtual space provided by the head-mounted device (200), a person speaking on the left or right side of the screen can be more clearly distinguished. Furthermore, the head-mounted device (200) can realistically express the sound of an approaching person's footsteps, and enhance the perception of clarity in music, allowing for better distinction between the sounds of instruments and singers' voices. For example, in the virtual space provided by the head-mounted device (200), a user can hear sounds based on their position in the up-down or left-right directions, thereby providing a completely immersive experience.

[0083] However, listening to music through Spatial Audio through a head-mounted device (200) can require a lot of work. For example, every part of the sound data must be mapped to audio, sound effects, and instruments, and the distance, intensity, and balance of the sound can be adjusted after the complex mapping process. For example, working with a home theater with multiple speakers can be easier to reproduce because each channel receives a specific sound. For example, in the virtual space provided by the head-mounted device (200), these multiple channels must be simulated so that the user can experience the same sensation as in a room with a complete sound system. For this purpose, HRTF (Head-Related Transfer Function Filter) can be used.

[0084] For example, a neural speech enhancement network can support functions such as speaker segmentation, sound localization, and reverb cancellation. For example, a neural speech enhancement network can support low-power edge computing (e.g., earbuds).

[0085] In one embodiment, the OS (410) may generate graphics to be displayed on the screen and transmit display signals (e.g., graphics) to the display controller (450).

[0086] According to one embodiment, the OS (410) can transmit and receive data or signals with a mixed reality system (e.g., VST manager (460)) or an account system (e.g., global membership system (GMS) (470)).

[0087] An OS (410) according to one embodiment can transmit and receive data or signals with an application library (430). The application library (430) according to one embodiment includes at least one library related to applications, and may include applications based on 2D screens (Android 2D / 2.5D) and / or applications based on 3D screens (Impress or Immersive).

[0088] The above software block diagram and / or software architecture is merely an example, and at least some of the configurations shown may be changed depending on the platform included in the head mounted device (200).

[0089] FIG. 5 is a block diagram of a head mounted device (200) according to one embodiment of the present disclosure.

[0090] Referring to FIG. 5, a head mounted device (200) according to one embodiment may include a processor (510), a memory (520), a display (530), a built-in speaker (540), a communication circuit (550), and / or an inertial sensor (560).

[0091] A processor (510, e.g., processor (120) of FIG. 1) according to one embodiment may control at least one other component (e.g., hardware or software component) of the head mounted device (200). In one embodiment, the processor (510) may perform various data processing or operations, and as at least part of the data processing or operations, may store commands or data received from other components in a memory (520), process the commands or data stored in the memory (520), and store result data in the memory (520).

[0092] A memory (520, e.g., memory (130) of FIG. 1) according to one embodiment can store instructions executable by the processor (510).

[0093] In one embodiment, the display (530, e.g., the display module (160) of FIG. 1 or the display member (225) of FIG. 2) may be configured to display a screen in front of the user's eyes. According to one embodiment, the display (530) may display a screen corresponding to virtual reality or mixed reality. In one embodiment, the display (530) may display a 3D screen rendered in three dimensions.

[0094] A built-in speaker (540, e.g., the audio output module (155) of FIG. 1 or the speaker (240) of FIG. 2) according to one embodiment may output a sound corresponding to sound data. In one embodiment, the built-in speaker (540) may output a sound corresponding to an application or content, such as multimedia playback or recording playback, or may output a sound corresponding to a phone ring or communication.

[0095] The built-in speaker (540) according to one embodiment is positioned adjacent to the ear of a user wearing the head-mounted device (200), so as to output sound directly to the ear of the user.

[0096] The built-in speaker (540) according to one embodiment may be capable of supporting output of at least one of channel-based sound (e.g., stereo sound), scene-based sound (e.g., ambisonics), and / or object-based sound (e.g., dolby atmos).

[0097] A communication circuit (550, e.g., communication module (190) of FIG. 1) according to one embodiment may be configured to transmit and receive data (or signals) with an external device (590). In one embodiment, the communication circuit (550) may be connected to the external device (590) via a designated communication method. For example, the designated communication method may be a wireless communication method such as Bluetooth communication or Wi-Fi communication (e.g., Wi-Fi casting), or may be a wired communication method. In one embodiment, the communication circuit (550) may transmit sound data to the external device (590) to which it is connected.

[0098] In one embodiment, the communication circuit (550) may transmit, together with the sound data, a control signal for outputting the sound data to an external device (590) to which the communication circuit is connected. In one embodiment, the communication circuit (550) may transmit, together with the control signal, data or a signal regarding a volume corresponding to the sound data.

[0099] In one embodiment, an inertial sensor (560, e.g., sensor module (176) of FIG. 1) may be configured to detect movement or rotation of the head mounted device (200). In one embodiment, the inertial sensor (560) corresponds to at least one of an IMU sensor, a gyro sensor, or an acceleration sensor, and may detect movement in three-axis directions and rotation in three-axis directions (e.g., 6 DoF). In one embodiment, the head mounted device (200) may detect movement or rotation of the user, and detect the posture and direction of the user. In one embodiment, the head mounted device (200) may detect a position and / or angle with respect to an external device (590) that changes according to the movement or rotation of the user, using the inertial sensor (560).

[0100] An external device (590) according to one embodiment may include a speaker module. In one embodiment, the external device (590) may output sound according to sound data when receiving a control signal from a head-mounted device (200) connected to the communication. For example, the external device (590) may be a speaker device or an output device including a speaker module.

[0101] In addition to the configurations shown and described, a head mounted device (200) according to one embodiment may further include or exclude some of the configurations included in the electronic device (101) of FIG. 1 or the head mounted device (200) of FIGS. 2 to 3.

[0102] FIG. 6 is a state diagram illustrating a head mounted device (200) in communication connection with an external device (590) according to one embodiment of the present disclosure.

[0103] Referring to FIG. 6, a head mounted device (200) according to one embodiment can display a screen in at least one designated space (S1, S2, S3). In one embodiment, the at least one designated space (S1, S2, S3) may include a first space (S1) corresponding to a near view, a second space (S2) corresponding to a middle view, and / or a third space (S3) corresponding to a far view. For example, the first space (S1) is positioned relatively close to the user (e.g., 1.7 m), and a task bar and / or desk content may be displayed. For example, the second space (S2) is positioned relatively far from the first space (S1) (e.g., 2.0 m), and a dashboard (Dash) and space content may be displayed. For example, the third space (S3) corresponds to an infinite distance, and content located at infinity, such as a background, can be displayed.

[0104] A head mounted device (200) according to one embodiment can display an object (O) corresponding to an application being executed or an object (O) corresponding to content provided by an application in at least one designated space. In one embodiment, the head mounted device (200) can selectively display an object (O) in a first space (S1) located relatively close to a user, a second space (S2) located relatively far away from a user, and / or a third space (S3) located at infinity, and output sound data of the object (O) so as to correspond to the space in which the object (O) is displayed.

[0105] A head-mounted device (200) according to one embodiment can output sound data through a built-in speaker (540) or through an external device (e.g., an external device (590) of FIG. 5) based on the display position (e.g., space and / or direction) of an object (O).

[0106] A head mounted device (200) according to one embodiment may be communicatively connected to at least one external device (590). In one embodiment, the head mounted device (200) may scan the location of at least one external device (590) that is communicatively connectable. In one embodiment, when the head mounted device (200) is communicatively connected to at least one external device (590), the head mounted device (200) may obtain data related to the location of the external device (590).

[0107] In one embodiment, the head mounted device (200) can transmit sound data to at least some of the at least one external device (590) with which it is communicatively connected. In one embodiment, the head mounted device (200) can transmit a control signal to at least some of the at least one external device (590) with which it is communicatively connected to output a sound corresponding to the sound data.

[0108] An external device (590) according to one embodiment may output sound corresponding to received sound data based on a control signal. For example, the external device (590) may be a speaker device (593) that outputs sound from a specific location. For example, the external device (590) may be a home theater device (595) that outputs sound corresponding to each channel from multiple locations.

[0109] For example, a home theater device (595) provides 5.1 channels and can output sound by being positioned at the positions of 11 o'clock (-30˚), 12 o'clock (0˚), 13 o'clock (30˚), 7 o'clock (-110˚), and 5 o'clock (110˚). In one embodiment, when the head mounted device (200) is connected to the home theater device (595), it can be connected to the Center, and in addition, can receive position information of Front-L, Front-R, surround-L, and Surround-R.

[0110] According to one embodiment, a head mounted device (200) may output sound data using an external device (590) based on the activation of a 3D audio mode. In one embodiment, the head mounted device (200) may separate the sound data, output a portion of the sound data to the built-in speaker (540), and output the remaining portion of the sound data using the external device (590) connected to the communication. For example, the head mounted device (200) may output sound data corresponding to a three-dimensional sound that can be heard from a distance using the external device (590) connected to the communication. Accordingly, the effect of improving the user's audio experience may be achieved.

[0111] FIG. 7a, FIG. 7b and FIG. 7c illustrate sound rendering types of the built-in speaker (540) according to various embodiments of the present disclosure.

[0112] Referring to FIGS. 7A, 7B, and 7C, an internal speaker (540) included in a head mounted device (200) according to one embodiment may be capable of supporting at least one of channel-based sound (e.g., stereo sound), scene-based sound (e.g., ambisonics), and / or object-based sound (e.g., dolby atmos). In one embodiment, a speaker module included in an external device (590) may also be capable of supporting at least one of channel-based sound, scene-based sound, and / or object-based sound.

[0113] An internal speaker (540) supporting channel-based sound according to one embodiment may output sounds corresponding to each channel at a fixed location (e.g., a virtual location). In one embodiment, the fixed location may correspond to multiple locations, and sounds generated at each location may be fused together and provided.

[0114] The built-in speaker (540) supporting scene-based sound according to one embodiment can provide sound in an omnidirectional surround manner from a single point in space. In one embodiment, the built-in speaker (540) can output sound not only in a planar direction but also in an up-and-down direction. In one embodiment, the scene-based sound can be dynamically rendered to respond to the rotation or movement of the head-mounted device (200), thereby generating a 360-degree rotation effect. For example, the built-in speaker (540) can be decoded with a conventional speaker array or can provide a three-dimensional sound by utilizing a head-related transfer function (HRTF).

[0115] The built-in speaker (540) supporting object (O)-based sound according to one embodiment can provide surround sound up to 64 channels by combining it with an audio processing algorithm that adds spatial information to the front, side, rear, back, and top speakers.

[0116] A head mounted device (200) according to one embodiment can render sound data to be provided to the built-in speaker (540) based on the sound support type of the built-in speaker (540).

[0117] FIG. 8 is a state diagram regarding a state of outputting sound of a head mounted device (200) according to one embodiment of the present disclosure.

[0118] Referring to FIG. 8, a head mounted device (200) according to one embodiment can display an object (O) at a specific location in a three-dimensional space through a display (530). In one embodiment, the head mounted device (200) can display the object (O) so that it is located on a designated space (S). For example, the designated space (S) may be a space corresponding to a relative distance (e.g., the third space (S3) of FIG. 6).

[0119] In one embodiment, the object (O) may be a screen corresponding to the content of the application and may include corresponding sound data. For example, the object (O) may be a window of a music playback application. For example, the object (O) may be a screen corresponding to music content provided by the music playback application.

[0120] In one embodiment, the object (O) may be one of at least one sound source displayed on the display (530) by the running application. For example, if the running application provides sound data including multiple sound sources, the object (O) may be a content screen (e.g., a person, an animal, a character, or a musical instrument) corresponding to at least one of the multiple sound sources.

[0121] A head mounted device (200) according to one embodiment can establish a communication connection with at least one external device (590) including a speaker module. In one embodiment, the head mounted device (200) can manually establish a communication connection with the external device (590) by a user's input. In one embodiment, the head mounted device (200) can automatically scan an external device (590) capable of establishing a communication connection, and establish a communication connection with an external device (590) identified by the scan. In one embodiment, the head mounted device (200) can establish a communication connection with the external device (590) when an application or content providing sound data is executed.

[0122] According to one embodiment, a head mounted device (200) can scan at least one external device (590) using a directional communication method (e.g., ultra-wide band (UWB) communication), and thereby identify a location of at least one external device (590). According to one embodiment, a head mounted device (200) can generate indoor map data for a location of an external device (590) based on a distance and / or an angle. In one embodiment, the head mounted device (200) can detect movement or rotation of the head mounted device (200) using an inertial sensor (e.g., the inertial sensor (560) of FIG. 5), and can detect a location and / or angle with respect to the external device (590) that changes according to the rotation of the head mounted device (200).

[0123] A head mounted device (200) according to one embodiment can output sound data corresponding to an object (O) through at least one external device (590) or through a built-in speaker (540) based on a specific location where the object (O) is displayed or the display form of the object (O).

[0124] In one embodiment, the head mounted device (200) can identify at least some of at least one external device (590) based on a specific location where the object (O) is displayed. In one embodiment, the head mounted device (200) can transmit sound data corresponding to the object (O) and a control signal for outputting the sound data to at least some of the identified at least one external device (590).

[0125] For example, the head mounted device (200) can identify an external device (590) positioned adjacent to a specific location (e.g., front, rear, left, right) where an object (O) is displayed, and transmit sound data corresponding to the object (O) to the identified external device (590).

[0126] FIG. 9a, FIG. 9b, and FIG. 9c are state diagrams illustrating a state in which a head mounted device (200) displays an object (O) on a first space (S1) according to one embodiment of the present disclosure.

[0127] Referring to FIGS. 9A, 9B, and 9C, a head mounted device (200) according to one embodiment can display an object (O) at a specific location on a first space (S1). In one embodiment, the head mounted device (200) can display an object (O) on a second space (S2) based on a user's input via a controller (C).

[0128] In one embodiment, the head mounted device (200) may display an object (O) on a dashboard (Dash) on a second space (S2). In one embodiment, the object (O) may be a 2D screen corresponding to content provided by an application.

[0129] In one embodiment, the second space (S2) may be formed on a plane or curved surface relatively close to the user. In one embodiment, the second space (S2) and / or the dashboard (Dash) may be a plane that is parallel to the x-axis and y-axis of a three-dimensional space (viewing frustum) provided from the head mounted device (200) and is spaced apart by a specified distance along the z-axis so as to intersect the z-axis. In one embodiment, the first space (S1) and / or the dashboard (Dash) may be a two-dimensional plane.

[0130] In one embodiment, the three-dimensional space provided by the head mounted device (200) may include an x-axis corresponding to the two-way direction of the user or the head mounted device (200) (e.g., the display (530)), a y-axis corresponding to the up-down direction of the head mounted device (200) (e.g., the display (530)), and a z-axis corresponding to the intersecting front-back direction of the head mounted device (200) (e.g., the display (530)).

[0131] A head mounted device (200) according to one embodiment can identify a display position and / or a display form of an object (O) displayed in a three-dimensional space. In one embodiment, the head mounted device (200) can identify the display position of the object (O) as a dashboard (Dash) on a second space (S2). Based on identifying that the display position of the object (O) is a dashboard (Dash) on the second space (S2), the head mounted device (200) according to one embodiment can output sound data corresponding to the object (O) through a built-in speaker (540).

[0132] A head-mounted device (200) according to one embodiment can output sound in the form of Mono, Stereo, 5.1 channels, 3D audio Ambisonics, or Dolby based on the display form of an object (O). In one embodiment, the head-mounted device (200) can identify the display form of an object (O) as a 2D window displayed on a 2D plane.

[0133] FIG. 10A, FIG. 10B, and FIG. 10C illustrate user input methods according to various embodiments of the present disclosure.

[0134] Referring to FIGS. 10A, 10B, and 10C, a head mounted device (200) according to one embodiment can receive a gesture input using a user's hand (H) or an input of a controller (C) using a controller (C) configured to be held by the user's hand (H).

[0135] In one embodiment, the head mounted device (200) can receive input from a controller (C) through a controller (C) held in each of the user's two hands (H), as illustrated in FIG. 10a.

[0136] In one embodiment, the head-mounted device (200) can receive a gesture input using the user's two hands (H), as illustrated in FIG. 10b. For example, the head-mounted device (200) can identify the gesture input by photographing and tracking the user's hands (H) using a camera.

[0137] In one embodiment, the head mounted device (200) can receive a gesture input using one hand (H) of the user and an input of a controller (C) through a controller (C) held by the other hand (H) of the user, as illustrated in FIG. 10c.

[0138] A head-mounted device (200) according to one embodiment can receive a user's interaction input based on various combinations of inputs and / or gesture inputs of a controller (C). For example, the interaction input may be a tap input, a drag&drop input, or a zoom in / out input of the controller (C) for an object (O). For example, the interaction input may be a gesture input of a grip, a pinch, and / or a click for an object (O).

[0139] In one embodiment, the head mounted device (200) may receive interaction input using a separate input device (e.g., a keyboard or mouse), an external device (590) (e.g., a smartphone or smartwatch), or voice recognition (e.g., Bixby).

[0140] FIG. 11a, FIG. 11b and FIG. 11c are state diagrams illustrating an operation of a head mounted device (200) moving an object (O) from a first space (S1) to a second space (S2) according to one embodiment of the present disclosure.

[0141] Referring to FIGS. 11a, 11b, and 11c, a head mounted device (200) according to one embodiment can identify a display position and / or display form of an object (O) displayed in a three-dimensional space.

[0142] A head mounted device (200) according to one embodiment may obtain an input corresponding to an object (O) and, based on the input corresponding to the object (O), change a specific location at which the object (O) is displayed. For example, as illustrated, the head mounted device (200) may obtain an input for moving an object (O) displayed on a second space (S2) to a third space (S3) that is relatively far away, and, accordingly, change the location at which the object (O) is displayed.

[0143] A head mounted device (200) according to one embodiment can control the output of sound data corresponding to an object (O) based on a change in the position at which the object (O) is displayed. For example, as illustrated, the head mounted device (200) can display the object (O) in a reduced size based on moving the object (O) displayed on a second space (S2) to a third space (S3) that is relatively far away, and can reduce the volume of the sound data corresponding to the object (O).

[0144] A head mounted device (200) according to one embodiment can identify at least a portion of at least one external device (590) based on a display position of an object (O). When the display position of an object (O) changes, the head mounted device (200) according to one embodiment can identify at least a portion of at least one external device (590) based on a position of the moved object (O).

[0145] According to one embodiment, a head mounted device (200) can identify at least some of at least one external device (590) adjacent to the direction from the head mounted device (200) toward the object (O) based on the location where the object (O) is displayed. For example, when the object (O) is displayed on the left side of the head mounted device (200), the head mounted device (200) can identify the external device (590) located on the left side of the head mounted device (200).

[0146] In one embodiment, the head mounted device (200) can decompose sound data corresponding to an object (O) and map it to at least a portion of at least one external device (590) adjacent to a direction from the head mounted device (200) toward the object (O). In one embodiment, the head mounted device (200) can output a sound corresponding to the sound data from a direction matching the direction toward the object (O) through a portion of at least one external device (590).

[0147] A head mounted device (200) according to one embodiment can identify whether an object (O) is displayed at a distance greater than a specified distance from the head mounted device (200). In one embodiment, if the object (O) is displayed at a distance greater than the specified distance, the head mounted device (200) can output sound data corresponding to the object (O) through at least some of the identified at least one external device (590).

[0148] In one embodiment, depending on the distance (e.g., depth) between objects (O), the volume of sound output may vary depending on the distance. For example, the volume of sound output from a first space (S1) may be perceived by the user as relatively louder than the volume of sound output from a second space (S2) that is relatively distant.

[0149] In one embodiment, the head mounted device (200) can adjust the output volume of sound data in response to the distance from the object (O). For example, the head mounted device (200) can gradually decrease the output volume as the object (O) moves away.

[0150] According to one embodiment, a head mounted device (200) may output sound data corresponding to the object (O) at a first volume through a built-in speaker (540) when an object (O) is displayed at a distance greater than a specified distance, and output the first sound data at a second volume through an identified portion of at least one external device (590). For example, the first sound data may include sound data corresponding to the object (O). For example, the first sound data may include sound data of at least a portion of sound data corresponding to the object (O).

[0151] In one embodiment, the first volume of the built-in speaker (540) and / or the second volume of the at least one external device (590) may be determined based on the distance that the object (O) is spaced from the head mounted device (200). According to one embodiment, as the object (O) is located in a space corresponding to a far distance from the head mounted device (200), the first volume of the built-in speaker (540) may become relatively smaller, and the second volume of the at least one external device (590) may become larger. For example, when the object (O) is located in the first space (S1) corresponding to the near view, the first volume of the built-in speaker (540) may be 100%, and the second volume of the at least one external device (590) may be 0%. For example, when the object (O) is located in the third space (S3) corresponding to the distance, the first volume of the built-in speaker (540) may be 0%, and the second volume of at least one external device (590) may be 100%. For example, when the object (O) is located between the first space (S1) and the third space (S3), the volume may be proportionally determined based on the position at which the object (O) is displayed spaced apart. For example, when the object (O) is located in the second space (S2), the first volume of the built-in speaker (540) may be 50%, and the second volume of at least one external device (590) may be 50%. The head mounted device (200) according to one embodiment may track the movement trajectory as the user's head moves or rotates, and adjust the ratio between the first volume and the second volume accordingly.

[0152] FIG. 12 and FIG. 13a illustrate a state in which the display form of an object (O) is changed according to one embodiment of the present disclosure. FIG. 13b illustrates a state in which sound is output using an external device (590) of a head-mounted device (200) according to one embodiment of the present disclosure.

[0153] Referring to FIG. 12, a head mounted device (200) according to one embodiment can display an object (O) (1210) in a 2D window form on a dashboard (Dash) included in a second space (S2). The head mounted device (200) according to one embodiment can change the display form of the object (O) based on a user input corresponding to the object (O). The head mounted device (200) according to one embodiment can change the object (O) (1210) displayed in a 2D window form into a 3D widget (1211) in response to an input for the object (O). For example, the user input may include a user gesture interaction that changes the display location of the object (O) (1210) from the second space (S2) to the first space (S1). For example, the user gesture interaction may include a drag or a pinch and move interaction.

[0154] A head-mounted device (200) according to one embodiment can output sound data corresponding to an object (O) (1210) displayed in a 2D form using an external device (590) based on the object (O) (1210) being displayed as a 3D widget (1211), as shown in FIG. 12 and FIG. 13b.

[0155] In one embodiment, the head mounted device (200) may convert sound data corresponding to the object (O) based on the display form of the object (O). For example, sound data corresponding to the object (O) (1210) displayed in a 2D window form may be of the 2D Mono type. For example, when the object (O) is changed to a 3D widget (1211), the head mounted device (200) may convert the sound data to the Dolby atmos type.

[0156] Referring to FIG. 13A, a head mounted device (200) according to one embodiment may display an object (1310, 1311) of a three-dimensional widget in the first space (S1) and / or the third space (S3) based on the object (O) moving from the second space (S2) to the first space (S1) and / or the third space (S3). The head mounted device (200) according to one embodiment may output sound data corresponding to the object (O) (1310, 1311) using an external device (590). In one embodiment, the head mounted device (200) may synthesize sound data corresponding to the object (O) (1310, 1311) into three-dimensional spatial sound, and may map and distribute the sound data synthesized into the three-dimensional spatial sound to each external device (590).

[0157] According to one embodiment, a head mounted device (200) can change a 2D window-shaped object (O) (1300), as illustrated in FIG. 13A, into a 3D widget (1310, 1311). For example, the 2D window-shaped object (O) can include an object that can execute a music application. For example, the 3D widget (1310, 1311) can be implemented in the shape of a CD player. According to one embodiment, based on a user interaction that changes the display position of the 2D window-shaped object (O) (1300) from the second space (S2) to the first space (S1), the head mounted device (200) can display a 3D widget (1310) in the first space (S1) instead of the 2D window-shaped object (O) (1300). According to one embodiment, the head mounted device (200) may display a 3D widget (1311) in the third space (S3) instead of the 2D window-shaped object (O) (1300) based on a user's gesture interaction that changes the display location of the 2D window-shaped object (O) (1300) from the second space (S2) to the third space (S3). For example, the user's gesture interaction may include a drag or pinch and move interaction.

[0158] According to one embodiment, a head mounted device (200) may convert sound data based on the display form of an object (O) (1311) displayed as a 3D widget on a wall surface of a third space (S3). In one embodiment, the head mounted device (200) may perform an audio synthesis operation and an operation of distributing the audio to an external device (590) so as to convert a mono sound corresponding to an object (O) (1300) in the form of a 2D window into a 5.1 channel surround sound of an external device (590).

[0159] FIG. 14 is a state diagram illustrating a state in which an event that generates sound data having a priority occurs according to one embodiment of the present disclosure.

[0160] Referring to FIG. 14, a head mounted device (200) according to one embodiment can detect an event that generates sound data having a higher priority than sound data corresponding to an object (O). For example, the event may include an incoming call event or an alarm reception event.

[0161] In one embodiment, when a head mounted device (200) receives a call interrupt event while outputting sound of sound data corresponding to an object (O) through a built-in speaker (540), the sound output through the built-in speaker (540) can be switched to an external device (590). In one embodiment, the head mounted device (200) can provide call audio control to the built-in speaker (540).

[0162] A head mounted device (200) according to one embodiment may output sound data having a priority through a built-in speaker (540) when detecting an event that generates sound data having a priority. A head mounted device (200) according to one embodiment may transmit sound data corresponding to an object (O) having a relatively low priority to at least one external device (590), and transmit a control signal for outputting a sound corresponding to the object (O) to at least one external device (590).

[0163] According to one embodiment, a head mounted device (200) can synthesize sound data into multiple channels when sound data corresponding to an object (O) is stereo sound. In one embodiment, when a call interrupt event is received, the head mounted device (200) can convert internal channel sound output through the built-in speaker (540) to an external channel corresponding to an external device (590) and synthesize and distribute the CALL reception sound into the internal channel.

[0164] FIG. 15 is a state diagram illustrating a state in which sound data corresponding to each of a plurality of objects (O1, O2, O3) is generated according to one embodiment of the present disclosure.

[0165] Referring to FIG. 15, according to one embodiment, the plurality of objects (O1, O2, O3) may include objects displayed in a first space (S1), a second space (S3), or a third space (S3).

[0166] A head mounted device (200) according to one embodiment may synthesize sound data corresponding to at least one object (O1, O2, O3) that outputs sound data in a three-dimensional space into spatial sound based on the position of each object (O1, O2, O3), and distribute and output the synthesized sound data to a built-in speaker (540) and / or at least one external device (590). In one embodiment, the head mounted device (200) may synthesize sound sources corresponding to at least one object (O1, O2, O3) displayed through a display (530) into spatial sound and output them as if they were output from the display position of each object (O1, O2, O3).

[0167] In one embodiment, the head mounted device (200) may not output sound of sound data corresponding to an object (O1, O2, O3) when the object (O1, O2, O3) is displayed through the display (530) and then the display is stopped.

[0168] For example, when the head mounted device (200) stops displaying the first object (01) and the second object (02) among the objects (O1, O2, O3) and continues to display the third object (O3), the head mounted device (200) may not output sound data corresponding to the first object (01) and sound data corresponding to the second object (02), but may output only sound data corresponding to the third object (O3). In one embodiment, the head mounted device (200) may output sound through at least one external device (590) and / or built-in speaker (540) based on the type of the sound source in the case of sound data corresponding to a sound source that does not exist in a three-dimensional space.

[0169] In one embodiment, when the head mounted device (200) identifies a sound source that is not displayed in a three-dimensional space, it may transmit sound data and a control signal corresponding to the sound source to at least one external device (590). In one embodiment, when the head mounted device (200) identifies sound data corresponding to a background sound occurring in a three-dimensional space, it may transmit sound data and a control signal to at least one external device (590).

[0170] In one embodiment, when the head mounted device (200) identifies sound data of a virtual object (e.g., an avatar) located in a three-dimensional space, it can output the sound of the sound data through the built-in speaker (540).

[0171] In one embodiment, the head mounted device (200) may output the sound of the sound data corresponding to the object (O1, O2, O3) through the built-in speaker (540) if the sound data corresponding to the object (O1, O2, O3) corresponds to a designated type requiring privacy. For example, the head mounted device (200) may output the sound data through the built-in speaker (540) if sound data corresponding to a whisper is generated in a three-dimensional space.

[0172] FIG. 16 is a state diagram showing the output state of sound according to the display position of an object (O) according to one embodiment of the present disclosure.

[0173] Referring to FIG. 16, a head mounted device (200) according to one embodiment can output sound of sound data corresponding to an object (O) based on a display position of the object (O). In one embodiment, the head mounted device (200) can output sound of sound data corresponding to an object (O) corresponding to a distance from the object (O).

[0174] In one embodiment, when an object (O) (e.g., object (O) (1211) of FIG. 12, object (O) (1310) of FIG. 13a) is displayed to be located in the foreground (e.g., first space (S1)) in a three-dimensional space, the head mounted device (200) can output sound of sound data corresponding to the object (O) (e.g., object (O) (1211) of FIG. 12, object (O) (1310) of FIG. 13a)) through the built-in speaker (540).

[0175] In one embodiment, when an object (O) (e.g., object (0) (1311) of FIG. 13a) is displayed in a three-dimensional space to be located in a distant view (e.g., third space (S3)), the head-mounted device (200) can output a sound of sound data corresponding to the object (O) (e.g., object (0) (1311) of FIG. 13a) through an external device (590).

[0176] In one embodiment, when an object (O) is displayed to be located in the middle (e.g., the second space (S2)) in a three-dimensional space, the head mounted device (200) can distribute sound data corresponding to the object (O) to the built-in speaker (540) and the external device (590) to output the sound of the sound data. In one embodiment, the head mounted device (200) can distribute sound data corresponding to the object (O) to the built-in speaker (540) and the external device (590) based on the position information of the external device (590), to output the sound of the sound data. According to one embodiment, when the external device (590) is located within a predetermined distance from the head mounted device (200), the head mounted device (200) can distribute sound data corresponding to the object (O) to the built-in speaker (540) and the external device (590), to output the sound of the sound data. According to one embodiment, the head mounted device (200) may not output sound of sound data if the external device (590) is not located within a predetermined distance from the head mounted device (200). According to one embodiment, the predetermined distance may be specified by a user or may be automatically specified by the processor (510).

[0177] In one embodiment, the head mounted device (200) may indicate that the volume of the built-in speaker (540) is 100% when there is no external device (590) located within a predetermined distance from the head mounted device (200).

[0178] In one embodiment, when there is one external device (590) located within a predetermined distance from the head mounted device (200), the volume of the built-in speaker (540) may indicate 50%, and the volume of the external device (590) may indicate 50%. In one embodiment, when there are two external devices (590) located within a predetermined distance from the head mounted device (200), the volume of the built-in speaker (540) may indicate 50%, and the volume of each external device (590) may indicate 25%.

[0179] In one embodiment, the head mounted device (200) may receive a user input for selecting a speaker to output sound data for each running application, content of the application, and / or sound source included in the content. In one embodiment, the head mounted device (200) may output sound of sound data corresponding to each application, content, and / or sound source through the built-in speaker (540) and / or external device (590) based on the user's selection input.

[0180] According to one embodiment, a head mounted device (200) can adjust a volume ratio between a built-in speaker (540) that outputs sound data corresponding to an object (O) and an external device (590) based on a user input for an object (O). For example, the head mounted device (200) can identify a user input for setting a first volume of the built-in speaker (540) that outputs sound data corresponding to the object (O) and a user input for setting a second volume of the external device (509). Based on the user input, the head mounted device (200) can output sound data through the built-in speaker (540) at the first volume and output sound data through the external device (509) at the second volume. According to one embodiment, the head mounted device (200) can store a usage history of the head mounted device (200) distributing and outputting sound to the built-in speaker (540) and the external device (590), respectively. For example, the head mounted device (200) may store usage history by generating a lookup table related to an application associated with an object (0), sound data, a plurality of first volumes of sound data output to a built-in speaker (540), the number of times sound data having each of the plurality of first volumes is output to the built-in speaker (540), a plurality of second volumes of sound data output to an external device (509) when sound data having each of the plurality of first volumes is output to the built-in speaker (540), and the number of times sound data having each of the plurality of second volumes is output to the external device (509).

[0181] A head-mounted device (200) according to one embodiment can adjust the volume ratio between a built-in speaker (540) that outputs sound data corresponding to an object (O) and an external device (590) based on a user's usage history.

[0182] According to one embodiment, a head mounted device (200) may, when outputting sound data related to an object (0), adjust the volume ratio between the built-in speaker (540) that outputs the sound data and the external device (590) based on the user's usage history. For example, the head mounted device (200) may, based on the user's input, check the first volume of sound data output to the built-in speaker (540) a certain number of times or more from a lookup table. For example, when the head mounted device (200) outputs sound data of the first volume to the built-in speaker (540), it may check the size of the volume output to the external device (590). The head mounted device (200) may output sound data through the built-in speaker (540) at the first volume, and output sound data through the external device (590) at the size of the checked volume.

[0183] FIG. 17 is a state diagram illustrating a state of separating the sound of objects (O1, O2) according to one embodiment of the present disclosure.

[0184] Referring to FIG. 17, a head mounted device (200) according to one embodiment can display at least one object (O1, O2) in a three-dimensional space through a display (530) and can adjust the volume of sound data corresponding to each object (O1, O2). For example, the head mounted device (200) can adjust the volume of sound data corresponding to at least one or more of the objects (O1, O2) based on an input corresponding to at least one or more of the objects (O1, O2).

[0185] A head mounted device (200) according to one embodiment may acquire sound data from at least one communication-connected source (e.g., a communication terminal) and output the acquired sound data through a built-in speaker (540) and / or an external device (590). For example, the head mounted device (200) may acquire sound data from terminal A and terminal B, respectively, and output the sound data through a built-in speaker (540) and / or an external device (590).

[0186] In one embodiment, the head mounted device (200) may adjust the volume of some of the sound data corresponding to at least one output source. For example, when a call interrupt event occurs in terminal A, the head mounted device (200) may adjust the volume of the sound data corresponding to terminal A and maintain the volume of the sound data corresponding to terminal B.

[0187] A head mounted device (200) according to one embodiment can receive a gesture input through a user's hand (H). In one embodiment, the head mounted device (200) can identify a gesture input corresponding to at least one object (O1, O2) or at least one source, and can adjust the volume of sound data corresponding to at least one object (O1, O2) or at least one source based on the identified gesture input. For example, the gesture input can include a pinch, a pinch & hold, or a pinch & move.

[0188] FIG. 18a and FIG. 18b are flowcharts (1800a, 1800b) regarding an operating method of a head mounted device (200, 101) according to one embodiment of the present disclosure.

[0189] Referring to FIGS. 18A and 18B , a head mounted device (200, 101) according to one embodiment may establish a communication connection with at least one external device (590, 102) including a speaker module through a communication circuit (550, 190) in operation 1810. The head mounted device (200, 101) according to one embodiment may obtain location information of at least one external device (590, 102) with which the communication connection has been established.

[0190] According to one embodiment, a head-mounted device (200, 101) can display an object (O) at a specific location in a three-dimensional space through a display (530, 160, 225) in operation 1820.

[0191] According to one embodiment, a head mounted device (200, 101) can, in operation 1830, identify whether an object (O) is displayed on a designated plane. For example, the designated plane may be a first space located at a close range from the head mounted device (200, 101) or the user.

[0192] According to one embodiment, the head mounted device (200, 101) may, if it is identified that the object (O) is not displayed on a designated plane (operation 1830-No), identify at least some of the at least one external device (590, 102) from which sound data is to be output, in operation 1843.

[0193] According to one embodiment, a head mounted device (200, 101) may map first sound data to at least a portion of at least one external device (590, 102), based on a display position or display form of an object (O), in operation 1845. According to one embodiment, a head mounted device (200, 101) may synthesize first sound data corresponding to an object (O) into spatial sound and map it to at least a portion of at least one external device (590, 102).

[0194] In one embodiment, a head mounted device (200, 101) may transmit first sound data to at least some of at least one external device (590, 102) through a communication circuit (550, 190) in operation 1847. In one embodiment, the head mounted device (200, 101) may transmit a control signal to at least some of at least one external device (590, 102) to output a sound of the first sound data.

[0195] According to one embodiment, a head mounted device (200, 101) may output first sound data through a built-in speaker (540, 155, 240) in operation 1850 when it identifies that an object (O) is displayed on a designated plane (operation 1830 - Yes).

[0196] According to one embodiment, a head mounted device (200, 101) can identify whether or not a user input corresponding to an object (O) is obtained in operation 1860.

[0197] According to one embodiment, when a head mounted device (200, 101) obtains a user input corresponding to an object (O) (operation 1860 - Yes), in operation 1865, the display position of the object (O) can be changed based on the user input.

[0198] According to one embodiment, the head mounted device (200, 101) may, if it does not obtain a user input corresponding to the object (O) (operation 1860 - No), identify whether an event that generates second sound data having priority over the first sound data is detected in operation 1870.

[0199] According to one embodiment, a head mounted device (200, 101) may, when an event that generates second sound data is detected (operation 1870 - Yes), output the second sound data through the built-in speaker (540, 155, 240) in operation 1880 and transmit a control signal to cause at least some of at least one external device (590, 102) to output the first sound data.

[0200] According to one embodiment, the head mounted device (200, 101) may, if it does not detect an event that generates second sound data (operation 1870 - Yes), identify whether a sound source that is not displayed in a three-dimensional space is identified in operation 1890.

[0201] According to one embodiment, a head mounted device (200, 101) may, when a sound source that is not displayed in a three-dimensional space is identified (operation 1890 - Yes), transmit third sound data corresponding to the sound source to at least one external device (590, 102). According to one embodiment, a head mounted device (200, 101) may transmit a control signal so that at least some of the at least one external device (590, 102) output the third sound data.

[0202] A head mounted device (200, 101) according to one embodiment may terminate a control operation if it does not identify a sound source that is not displayed in a three-dimensional space (operation 1890-No).

[0203] FIG. 19 is a flowchart (1900) regarding an operating method of a head mounted device (200, 101) according to one embodiment of the present disclosure.

[0204] Referring to FIG. 19, a head mounted device (200, 101) according to one embodiment may, in operation 1910, execute an application or task that generates sound data based on a first input of a user.

[0205] A head mounted device (200, 101) according to one embodiment can display an object (O) at a specific location in a three-dimensional space through a display (530, 160, 225) in operation 1930. A head mounted device (200, 101) according to one embodiment can display an object (O) corresponding to an executed application or task.

[0206] A head mounted device (200, 101) according to one embodiment can output at least a portion of sound data through a built-in speaker (540, 155, 240) based on the display position or display form of an object (O) in operation 1950.

[0207] According to one embodiment, a head mounted device (200, 101) may, in operation 1970, transmit a control signal to cause at least one external device (590, 102) to output at least a portion of sound data based on a display position or display form of an object (O).

[0208] According to one embodiment, a head mounted device (200, 101) can identify whether a second input of a user corresponding to an object (O) has been obtained in operation 1980.

[0209] According to one embodiment, when a head-mounted device (200, 101) obtains a second input of a user corresponding to an object (O) (operation 1980 - Yes), in operation 1990, it can change the display position or display form of the object (O) based on the second input.

[0210] According to one embodiment, the head mounted device (200, 101) may terminate the control operation if it does not obtain a second input from the user corresponding to the object (O) (operation 1980-No).

[0211] The technical tasks to be achieved in this document are not limited to the technical tasks mentioned above, and other technical tasks not mentioned will be clearly understood by those with ordinary skill in the technical field to which this document pertains from the description below.

[0212] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned can be clearly understood by a person having ordinary skill in the art to which the present disclosure belongs from the description below.

[0213] A head mounted device (200, 101) according to one embodiment of the present disclosure may include a display (530, 160, 225), a built-in speaker (540, 155, 240) configured to output sound, a communication circuit (550, 190), a memory (520, 130) storing executable instructions, and at least one processor (510, 120) accessing the memory (520, 130) to execute the instructions. The at least one processor (510, 120) may be configured to establish a communication connection with at least one external device (590, 102) including a speaker module through the communication circuit (550, 190). The at least one processor (510, 120) may be configured to display an object (O) at a specific location in a three-dimensional space through the display (530, 160, 225). The object (O) may include corresponding first sound data. The at least one processor (510, 120) may be configured to identify at least some of the at least one external device (590, 102) based on the specific location where the object (O) is displayed. The at least one processor (510, 120) may be configured to transmit the first sound data to at least some of the at least one external device (590, 102) via the communication circuit (550, 190).

[0214] In a head mounted device (200, 101) according to one embodiment, the object (O) is a screen of an application being executed, and the first sound data may correspond to the content of the application being executed.

[0215] In a head mounted device (200, 101) according to one embodiment, the object (O) is one of at least one sound source displayed on the display (530, 160, 225) by an executing application, and the first sound data may correspond to one of the at least one sound source.

[0216] In a head mounted device (200, 101) according to one embodiment, the at least one processor (510, 120) may be configured to obtain a location of the at least one external device (590, 102) at least as part of an operation of establishing a communication connection with the at least one external device (590, 102).

[0217] In a head mounted device (200, 101) according to one embodiment, the at least one processor (510, 120) may be set to output the first sound data through the built-in speaker (540, 155, 240) based on the specific location where the object (O) is displayed.

[0218] In a head mounted device (200, 101) according to one embodiment, the at least one processor (510, 120) may be configured to output the first sound data through the built-in speaker (540, 155, 240) based on whether the object (O) is displayed on a designated plane spaced apart from the head mounted device (200, 101) by a designated distance, at least as a part of an operation of outputting the first sound data through the built-in speaker (540, 155, 240).

[0219] In a head mounted device (200, 101) according to one embodiment, the at least one processor (510, 120) may be configured to identify whether the object (O) is displayed at a distance greater than a specified distance from the head mounted device (200, 101), and, based on the object (O) being displayed at a distance greater than the specified distance, output the first sound data at a first volume through the built-in speaker (540, 155, 240) as at least a part of an operation of outputting the first sound data to at least some of the at least one external device (590, 102), and transmit a control signal for outputting the first sound data at a second volume to at least some of the at least one external device (590, 102). The first volume or the second volume may be determined based on the distance the object (O) is from the head mounted device (200, 101).

[0220] In a head mounted device (200, 101) according to one embodiment, the at least one processor (510, 120) may be configured to obtain an input corresponding to the object (O), change the specific location at which the object (O) is displayed based on the input, and identify at least a part of the at least one external device (590, 102) based on the changed location of the object (O).

[0221] In a head mounted device (200, 101) according to one embodiment, the at least one processor (510, 120) may be configured to, as at least part of an operation of identifying at least some of the at least one external device (590, 102), identify at least some of the at least one external device (590, 102) adjacent to a direction in which the object (O) is displayed, based on the specific location in which the object (O) is displayed.

[0222] In a head mounted device (200, 101) according to one embodiment, the at least one processor (510, 120) may be configured to map the first sound data corresponding to the object (O) to at least some of the at least one external device (590, 102), based on the specific location where the object (O) is displayed or the display form of the object (O).

[0223] In a head mounted device (200, 101) according to one embodiment, the at least one processor (510, 120) may be configured to detect an event that generates second sound data having a priority over the first sound data, and based on the detection of the event, output the second sound data through the built-in speaker (540, 155, 240), and transmit a control signal for outputting the first sound data to at least a part of the at least one external device (590, 102).

[0224] In a head-mounted device (200, 101) according to one embodiment, the at least one processor (510, 120) identifies a sound source that is not displayed in the three-dimensional space, and the sound source includes third sound data corresponding to each of the sound sources, and transmits the third sound data to at least some of the at least one external device (590, 102) through the communication circuit (550, 190).

[0225] According to one embodiment of the present disclosure, a method of operating an electronic device may include an operation of establishing a communication connection with at least one external device (590, 102) including a speaker module through a communication circuit (550, 190). According to one embodiment, a method of operating an electronic device may include an operation of displaying an object (O) at a specific location in a three-dimensional space through a display (530, 160, 225). The object (O) may include corresponding first sound data. According to one embodiment, a method of operating an electronic device may include an operation of identifying at least some of the at least one external device (590, 102) based on the specific location at which the object (O) is displayed. According to one embodiment, a method of operating an electronic device may include an operation of transmitting the first sound data to at least some of the at least one external device (590, 102) through the communication circuit (550, 190).

[0226] An operating method of an electronic device according to one embodiment may include an operation (1830) of identifying whether the object (O) is displayed on a designated plane spaced apart from the head mounted device (200, 101) by a designated distance. An operating method of an electronic device according to one embodiment may include an operation (1850) of outputting the first sound data through a built-in speaker (540, 155, 240) set to output sound based on whether the object (O) is displayed on the designated plane.

[0227] An operating method of an electronic device according to one embodiment may include an operation (1830) of identifying whether the object (O) is displayed at a distance greater than a specified distance from the head mounted device (200, 101). In an operating method of an electronic device according to one embodiment, an operation (1850) of outputting the first sound data through the built-in speaker (540, 155, 240) based on whether the object (O) is displayed at a distance greater than the specified distance may output the first sound data at a first volume through the built-in speaker (540, 155, 240). In an operating method of an electronic device according to one embodiment, the operation (1847) of transmitting the first sound data to at least some of the at least one external device (590, 102) based on the object (O) being displayed at a distance greater than the specified distance may transmit a control signal for outputting the first sound data at a second volume to at least some of the at least one external device (590, 102). The first volume or the second volume may be set based on the distance at which the object (O) is separated from the head mounted device (200, 101).

[0228] The operating method of an electronic device according to one embodiment may include an operation (1860) of obtaining an input corresponding to the object (O). The operating method of an electronic device according to one embodiment may include an operation (1865) of changing the specific location at which the object (O) is displayed based on the input. In the operating method of an electronic device according to one embodiment, the operation (1843) of identifying at least some of the at least one external device (590, 102) may identify at least some of the at least one external device (590, 102) based on the changed display location of the object (O).

[0229] In an operating method of an electronic device according to one embodiment, an operation (1843) of identifying at least some of the at least one external device (590, 102) may identify at least some of the at least one external device (590, 102) adjacent to a direction in which the object (O) is displayed, based on the specific location at which the object (O) is displayed. The operating method of an electronic device according to one embodiment may include an operation (1845) of mapping the first sound data corresponding to the object (O) to at least some of the at least one external device (590, 102), based on the specific location at which the object (O) is displayed or the display form of the object (O).

[0230] An operating method of an electronic device according to one embodiment may include an operation (1870) of detecting an event that generates second sound data having a priority over the first sound data. An operating method of an electronic device according to one embodiment may include an operation (1880) of outputting the second sound data through the built-in speaker (540, 155, 240) based on detection of the event, and transmitting a control signal for outputting the first sound data to at least a portion of the at least one external device (590, 102).

[0231] A non-transitory computer-readable storage medium storing one or more programs according to one embodiment of the present disclosure may include an operation of establishing a communication connection with at least one external device (590, 102) including a speaker module through a communication circuit (550, 190) based on execution of an application. The storage medium according to one embodiment may include an operation of displaying an object (O) at a specific location in a three-dimensional space through a display (530, 160, 225). The object (O) may include corresponding first sound data. The storage medium according to one embodiment may include an operation of identifying at least some of the at least one external device (590, 102) based on the specific location at which the object (O) is displayed. The storage medium according to one embodiment may include an operation of transmitting the first sound data to at least some of the at least one external device (590, 102) through the communication circuit (550, 190).

[0232] A head mounted device (200, 101) according to one embodiment of the present disclosure may include a display (530, 160, 225), a built-in speaker (540, 155, 240) configured to output sound, a communication circuit (550, 190), a memory (520, 130) storing executable commands, and at least one processor (510, 120) accessing the memory (520, 130) to execute the commands. The at least one processor (510, 120) may be configured to execute an application or task that generates sound data based on a first input. The at least one processor (510, 120) may be configured to display an object (O) at a specific location in a three-dimensional space through the display (530, 160, 225) based on execution of the application or the task. The object (O) may correspond to at least a part of the sound data. The at least one processor (510, 120) may be configured to output at least a part of the sound data through the built-in speaker (540, 155, 240) based on the specific location where the object (O) is displayed or the display form of the object (O). The at least one processor (510, 120) may be configured to transmit a control signal for outputting at least a part of the sound data through the communication circuit (550, 190) to the at least one external device (590, 102) based on the specific location where the object (O) is displayed or the display form of the object (O).

[0233] In a head mounted device (200, 101) according to one embodiment, the at least one processor (510, 120) may be configured to obtain a second input corresponding to the object (O), change the specific position at which the object (O) is displayed or the display form of the object (O) based on the second input, output at least a portion of the sound data through the built-in speaker (540, 155, 240) based on the changed display position or display form of the object (O), and transmit a control signal for outputting the at least a portion of the sound data through the communication circuit (550, 190) based on the changed display position or display form of the object (O) to the at least one external device (590, 102).

[0234] A method of operating an electronic device according to one embodiment of the present disclosure may include an operation of executing an application or task that generates sound data based on a first input. A method of operating an electronic device according to one embodiment may include an operation of displaying an object (O) at a specific location in a three-dimensional space through a display (530, 160, 225) based on the execution of the application or the task. The object (O) may correspond to a sound source of at least a portion of the sound data. A method of operating an electronic device according to one embodiment may include an operation of outputting at least a portion of the sound data through a built-in speaker (540, 155, 240) based on the specific location at which the object (O) is displayed or the display form of the object (O). A method of operating an electronic device according to one embodiment may include an operation of transmitting a control signal for outputting at least a portion of the sound data to at least one external device (590, 102) through a communication circuit (550, 190) based on the specific location where the object (O) is displayed or the display form of the object (O).

[0235] A non-transitory computer-readable storage medium storing one or more programs according to one embodiment of the present disclosure may include an operation of executing an application or task that generates sound data based on a first input. According to one embodiment, the storage medium may include an operation of displaying an object (O) at a specific location in a three-dimensional space through a display (530, 160, 225) based on the execution of the application or the task. The object (O) may correspond to a sound source of at least a portion of the sound data. According to one embodiment, the storage medium may include an operation of outputting at least a portion of the sound data through a built-in speaker (540, 155, 240) based on the specific location at which the object (O) is displayed or the display form of the object (O). A storage medium according to one embodiment may include an operation of transmitting a control signal for outputting at least a portion of the sound data to at least one external device (590, 102) through a communication circuit (550, 190) based on the specific location where the object (O) is displayed or the display form of the object (O).

[0236] Electronic devices according to the embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, electronic devices, or home appliances. Electronic devices according to the embodiments disclosed in this document are not limited to the aforementioned devices.

[0237] The embodiments of this document and the terminology used herein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0238] The term "module" used in the embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0239] One embodiment of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0240] According to one embodiment, the method according to one embodiment disclosed in this document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0241] According to one embodiment, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and arranged in other components. According to one embodiment, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to one embodiment, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In the head mounted device (200; 101), display(530; 160; 225); Built-in speakers set to output sound (540; 155; 240); Communication circuit (550; 190); Memory (520; 130) for storing executable instructions; and At least one processor (510; 120) for accessing the memory (520; 130) and executing the instructions, The above instructions, when executed by the at least one processor, cause the head mounted device to: Through the above communication circuit (550; 190), a communication connection is established with at least one external device (590; 102) including a speaker module, Through the above display (530; 160; 225), an object (O) is displayed at a specific location in a three-dimensional space, and the object (O) includes corresponding first sound data. Based on the specific location where the object (O) is displayed, identifying at least a portion of the at least one external device (590; 102), Set to transmit the first sound data to at least some of the at least one external device (590; 102) through the above communication circuit (550; 190). Head mounted device (200; 101).

2. In paragraph 1, The above object (O) is the screen of the running application, The above first sound data corresponds to the content of the running application. Head mounted device (200; 101).

3. In paragraph 1, The above object (O) is one of at least one sound source displayed on the display (530; 160; 225) by the running application, The above first sound data corresponds to one of the at least one sound source, Head mounted device (200; 101).

4. In any one of paragraphs 1 to 3, The above instructions, when executed by the at least one processor, cause the head mounted device to: At least as part of the operation of establishing a communication connection with said at least one external device (590; 102), set to acquire the location of said at least one external device (590; 102); Head mounted device (200; 101).

5. In any one of paragraphs 1 to 4, The above instructions, when executed by the at least one processor, cause the head mounted device to: Based on the specific location where the object (O) is displayed, the first sound data is set to be output through the built-in speaker (540; 155; 240). Head mounted device (200; 101).

6. In paragraph 5, The above instructions, when executed by the at least one processor, cause the head mounted device to: Identifying whether the above object (O) is displayed on a designated plane spaced a designated distance from the head mounted device (200; 101), At least as a part of the operation of outputting the first sound data through the built-in speaker (540; 155; 240), based on the object (O) being displayed on the designated plane, the first sound data is set to be output through the built-in speaker (540; 155; 240). Head mounted device (200; 101).

7. In paragraph 6, The above instructions, when executed by the at least one processor, cause the head mounted device to: Identify whether the above object (O) is displayed at a distance greater than a specified distance from the head mounted device (200; 101), Based on the above object (O) being displayed at a distance greater than the specified distance: As at least a part of the operation of outputting the first sound data through the built-in speaker (540; 155; 240), outputting the first sound data at a first volume through the built-in speaker (540; 155; 240), As at least part of the operation of transmitting said first sound data to said at least some of said at least one external device (590; 102), a control signal for outputting said first sound data in a second volume is set to be transmitted to said at least some of said at least one external device (590; 102), The first volume or the second volume is determined based on the distance that the object (O) is separated from the head mounted device (200; 101). Head mounted device (200; 101).

8. In any one of paragraphs 1 to 7, The above instructions, when executed by the at least one processor, cause the head mounted device to: Obtain the input corresponding to the above object (O), Based on the above input, change the specific location where the object (O) is displayed, Based on the changed position of the above object (O), set to identify at least a part of the at least one external device (590; 102), Head mounted device (200; 101).

9. In any one of paragraphs 1 to 8, The above instructions, when executed by the at least one processor, cause the head mounted device to: At least as part of the operation of identifying at least one of said at least one external device (590; 102), Based on the specific location where the object (O) is displayed, at least some of the at least one external device (590; 102) adjacent to the direction in which the object (O) is displayed are set to be identified. Head mounted device (200; 101).

10. In paragraph 9, The above instructions, when executed by the at least one processor, cause the head mounted device to: Based on the specific location where the object (O) is displayed or the display form of the object (O), the first sound data corresponding to the object (O) is set to be mapped to at least some of the at least one external device (590; 102). Head mounted device (200; 101).

11. In any one of paragraphs 1 to 10, The above instructions, when executed by the at least one processor, cause the head mounted device to: Detecting an event that generates second sound data having priority over the first sound data, Based on the detection of the above event, Output the second sound data through the built-in speaker (540; 155; 240), Set to transmit a control signal for outputting the first sound data to at least one of the external devices (590; 102), Head mounted device (200; 101).

12. In any one of paragraphs 1 to 11, The above instructions, when executed by the at least one processor, cause the head mounted device to: Identifying sound sources that are not displayed in the above three-dimensional space, and each of the sound sources includes third sound data corresponding thereto. Through the above communication circuit (550; 190), the third sound data is transmitted to at least some of the at least one external device (590; 102), Head mounted device (200; 101).

13. In the operating method of the head mounted device (200; 101), An operation (1810) of establishing a communication connection with at least one external device (590; 102) including a speaker module through a communication circuit (550; 190); An operation (1820) of displaying an object (O) at a specific location in a three-dimensional space through a display (530; 160; 225), wherein the object (O) includes corresponding first sound data; An operation (1843) of identifying at least a portion of said at least one external device (590; 102) based on said specific location where said object (O) is displayed; and An operation (1847) for transmitting the first sound data to at least some of the at least one external device (590; 102) through the communication circuit (550; 190). How it works.

14. In paragraph 13, An operation (1830) for identifying whether the object (O) is displayed on a designated plane spaced a designated distance from the head mounted device (200; 101); and Further including an operation (1850) of outputting the first sound data through the built-in speaker (540; 155; 240) set to output sound based on the object (O) being displayed on the designated plane. How it works.

15. A storage medium storing computer-readable instructions, wherein the instructions, when executed by at least one processor (510; 120) of a head mounted device (101, 200), cause the head mounted device to perform at least one operation, the at least one operation being: An operation (1810) of establishing a communication connection with at least one external device (590; 102) including a speaker module through a communication circuit (550; 190); An operation (1820) of displaying an object (O) at a specific location in a three-dimensional space through a display (530; 160; 225), wherein the object (O) includes corresponding first sound data; An operation (1843) of identifying at least a portion of said at least one external device (590; 102) based on said specific location where said object (O) is displayed; and A storage medium including an operation (1847) of transmitting the first sound data to at least some of the at least one external device (590; 102) through the communication circuit (550; 190).

Images