Electronic device for generating or playing back audio signal, and operating method thereof

Abstract

An electronic device according to various embodiments disclosed in the present document may comprise a communication circuit, a memory, and a processor operatively connected to the communication circuit and the memory. The memory can store instructions that, when executed by the processor, enable the electronic device to: receive, through the communication circuit, two or more pieces of audio data simultaneously obtained by one or more external electronic devices; use user information acquired from the memory so as to identify user personal HRTF information acquired on the basis of reference HRTF information; identify sound source generation location information of the two or more pieces of audio data on the basis of the two or more pieces of audio data; perform inverse filtering on the two or more pieces of audio data on the basis of the user HRFT information so as to generate an audio signal; and transmit the audio signal and the sound source generation location information to another electronic device through the communication circuit.

Description

Electronic device for generating or reproducing audio signals and method of operating the same

[0001] Various embodiments disclosed in this document relate to an electronic device for generating or reproducing an audio signal and a method of operating the same.

[0002] Various electronic devices such as smart phones, tablet PCs, portable multimedia players (PMPs), personal digital assistants (PDAs), laptop personal computers, and wearable devices are becoming widespread.

[0003] Electronic devices can output sound data using wearable electronic devices such as earphones or headsets. The electronic devices can be connected to the earphones or headset via wireless communication (e.g., Bluetooth) and transmit sound data to the earphones or headset or receive external sound data acquired through a microphone installed in the earphones or headset.

[0004] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above-described matters constitute prior art related to the present disclosure.

[0005] The audio data recorded by inputting an external sound source through a microphone installed in a wearable electronic device, such as earphones or a headset worn on both ears of the user, may reflect various characteristics, such as the shape of the face and / or ears of the user wearing the wearable electronic device.

[0006] When another user plays audio data that reflects the user's various characteristics, it may be difficult for the user to feel the sense of presence directly because the other user's facial shape and / or ear shape are different from the user's.

[0007] The electronic devices of various embodiments disclosed in this document are for controlling a wearable electronic device so that external sounds of the wearable electronic device can be introduced or not introduced depending on the operating state of the electronic device.

[0008] An electronic device according to various embodiments disclosed in the present document may include a communication circuit, a memory, and a processor operatively connected to the communication circuit and the memory. The memory may store instructions that, when executed by the processor, cause the electronic device to receive two or more audio data simultaneously acquired by one or more external electronic devices through the communication circuit, to identify user personal HRTF information acquired based on reference HRTF (Head Related Transfer Function) information using user information acquired from the memory, to identify sound source generation location information of the two or more audio data based on the two or more audio data, to perform inverse filtering on the two or more audio data based on the user HRTF information to generate an audio signal, and to transmit the audio signal and the sound source generation location information to another electronic device through the communication circuit.

[0009] A method of an electronic device according to various embodiments may include an operation of receiving two or more audio data simultaneously acquired by one or more external electronic devices, an operation of confirming user personal HRTF information acquired based on reference HRTF (Head Related Transfer Function) information using user information, an operation of confirming sound source generation location information of the two or more audio data based on the two or more audio data, an operation of generating an audio signal by performing inverse filtering on the two or more audio data based on the user HRTF information, and an operation of transmitting the audio signal and the sound source generation location information to another electronic device.

[0010] An electronic device according to various embodiments may include a communication circuit, a memory, and a processor operatively connected to the communication circuit and the memory. The memory may store instructions that, when executed by the processor, cause the electronic device to receive a binaural audio signal and sound source generation location information of the audio signal from another electronic device through the communication circuit, to use user information acquired from the memory to identify user personal HRTF information acquired based on reference HRTF (Head Related Transfer Function) information, to apply the user personal HRTF information and the location information to the audio signal to generate two audio data, and to transmit the two audio data to one or more external electronic devices so that the two audio data can be played by the one or more external electronic devices.

[0011] A method of an electronic device according to various embodiments may include an operation of receiving a binaural audio signal and information on a sound source generation location of the audio signal from another electronic device through the communication circuit, an operation of confirming user personal HRTF information acquired based on reference HRTF (Head Related Transfer Function) information using user information acquired from the memory, an operation of generating two audio data by applying the user personal HRTF information and the location information to the audio signal, and an operation of transmitting the two audio data to one or more external electronic devices so that the two audio data are played by the one or more external electronic devices.

[0012] According to various embodiments, an external sound source can generate an audio signal by canceling various characteristics of a user wearing the earphones and / or headset from audio data recorded through a wearable device such as earphones and / or headset.

[0013] According to various embodiments, audio data can be generated and played back by reflecting information about an external sound source and various characteristics of a user in an audio signal, thereby providing a sense of presence through the played sound.

[0014] The technical tasks, technical features, and effects to be achieved in the present disclosure are not limited to the technical tasks, technical features, and effects mentioned above, and other technical tasks, technical features, and effects not mentioned will be clearly understood by a person having ordinary skill in the technical field to which the present invention pertains from the description below.

[0015] In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0016] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0017] FIG. 2 is a configuration diagram of an electronic device and an external electronic device (e.g., earphones) according to one embodiment.

[0018] FIG. 3 is a block diagram of an external electronic device according to one embodiment.

[0019] FIG. 4 is a drawing showing the appearance of an external electronic device according to one embodiment.

[0020] FIG. 5 is a flowchart illustrating an audio signal generation operation of an electronic device according to one embodiment.

[0021] FIG. 6 is a flowchart for explaining an audio signal reproduction operation of an electronic device according to one embodiment.

[0022] FIG. 7 is a drawing for explaining an operation of confirming or applying sound source location information of an electronic device according to one embodiment.

[0023] FIGS. 8, 9, and 10 are drawings for explaining an operation of confirming or applying sound source location information of an electronic device according to one embodiment.

[0024] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with the electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of the electronic device (104) or the server (108) via a second network (199) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0025] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0026] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0027] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0028] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0029] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0030] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0031] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0032] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0033] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0034] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0035] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0036] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0037] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0038] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0039] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0040] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0041] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0042] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0043] According to various embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0044] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0045] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0046] FIG. 2 is a block diagram (200) of an external electronic device (210 and / or 220) (e.g., electronic device (102) of FIG. 1) and an electronic device (e.g., electronic device (101) of FIG. 1) according to various embodiments.

[0047] Referring to FIG. 2, the external electronic devices (210 and / or 220) may include one or more electronic devices, such as a first external electronic device (210) and a second external electronic device (220). For example, the external electronic devices (210 and / or 220) may include headphones, earphones, and / or ear buds that can be worn by a user and provide sound to the user based on audio data received from the electronic device (101) through a communication function. Hereinafter, the first external electronic device (210) and the second external electronic device (220) included in the external electronic devices (210 and / or 220) are described as examples implemented as ear buds that can be worn on the left ear and right ear of the user, respectively, but various embodiments may not be limited thereto.

[0048] The electronic device (101) may be a portable and / or movable electronic device such as a smart phone, a tablet PC, a portable multimedia player (PMP), a personal digital assistant (PDA), a laptop PC, and a wearable device.

[0049] According to one embodiment, the electronic device (101) may be an electronic device capable of generating or playing music or images, and may receive audio data from an external electronic device (210 and / or 220) to generate an audio signal or transmit audio data to the external electronic device (210 and / or 220) to play the audio signal.

[0050] For example, the electronic device (101) may be an electronic device capable of generating music or images, and may receive and process audio data from external electronic devices (210 and 220), respectively, and generate an audio signal from which binaural sound can be restored. For example, the electronic device (101) may process audio data received from external electronic devices (210 and 220), generate an audio signal equalized by eliminating or reducing the influence of the user wearing the electronic device (101) from the audio data, and transmit the equalized audio signal to another electronic device (or referred to as a second electronic device) (e.g., the electronic device (102, 104, or 108) of FIG. 1).

[0051] For example, the electronic device (101) may process an audio signal received from another electronic device (e.g., the electronic device (102, 104, or 108) of FIG. 1) and transmit the generated audio data to the external electronic device (210 and / or 220) for playback. The generated audio data may be, for example, binaural audio data capable of restoring binaural sound through two different audio data. The two different audio data may be transmitted to the first external electronic device (210) and the second external electronic device (220) and played back, respectively, thereby restoring the binaural sound.

[0052] According to various embodiments, the external electronic device (210 and / or 220) may be wirelessly connected to the electronic device (101). For example, the electronic device (101) may communicate with the first external electronic device (210) using a first communication link (201) (e.g., the first network (198) of FIG. 1) including a short-range communication network such as Bluetooth (or BLE), WiFi direct, or IrDA (infrared data association). For example, the electronic device (101) may communicate with the second external electronic device (220) using a second communication link (202) (e.g., the first network (198) of FIG. 1) including a short-range communication network such as Bluetooth (or BLE), WiFi direct, or IrDA (infrared data association).

[0053] According to one embodiment, the first external electronic device (210) and the second external electronic device (220) may be implemented as earphones, and both devices may be independently connected to and operate with the electronic device (101), or one of the two devices may be operated as a PE (primary earbud, or primary equipment) and the other as a SE (secondary earbud, or secondary equipment). For example, when the first external electronic device (210) and the second external electronic device (220) are independently connected to and communicate with the electronic device (101), the first external electronic device (210) and the second external electronic device (220) may transmit and receive data with the electronic device (101) through the first communication link (201) and the second communication link (202), respectively. For example, when a first external electronic device (210) communicates with an electronic device (101) as a primary through a first communication link (201), a second external electronic device (220) as a secondary can sniff the first communication link (201) to obtain data transmitted from the electronic device (101) to the first external electronic device (210). Meanwhile, the first external electronic device (210) and the second external electronic device (220) can form a third communication link (203) and transmit and receive data through it.

[0054] Hereinafter, an example in which an electronic device (101) communicates with a first external electronic device (210) through a first communication link (201) and also communicates with a second external electronic device (220) through the same is described, but various embodiments are not limited thereto, and the electronic device (101) can communicate with the first external electronic device (210) and the second external electronic device (220), respectively, through independent first communication links (201) and second communication links (202) (e.g., the first network (198) of FIG. 1).

[0055] FIG. 3 is a block diagram of an external electronic device (300) (e.g., the first external electronic device (210) or the second external electronic device (220) of FIG. 2) according to various embodiments. FIG. 4 is a drawing illustrating the external appearance of the external electronic device (300).

[0056] Referring to FIG. 3, the external electronic device (300) may include a communication circuit (310), a processor (320), a memory (330), a microphone (340), a speaker (350), and a sensor (360). The components included in FIG. 3 are only some of the components included in the external electronic device (300), and the external electronic device (300) may additionally include various components (e.g., a power management circuit and / or a battery).

[0057] The processor (320) may, for example, execute software (e.g., a program) to process control commands and / or audio data received from an electronic device (e.g., an electronic device (101) of FIG. 2) connected via a communication circuit (310), and may store information based on the processing result or information generated based on the operation of various components in the memory (330) or transmit it to the electronic device. For this purpose, the processor (320) may control at least one other component (e.g., a hardware or software component) of the external electronic device (300) and perform various data processing or calculations.

[0058] The memory (330) can store various data used by at least one component (e.g., processor (320) or sensor (360)) of the external electronic device (300). The data can include, for example, input data or output data for software (e.g., program) and commands related thereto. The memory (330) can include volatile memory or non-volatile memory.

[0059] According to one embodiment, as at least part of data processing or calculation, the processor (320) may load instructions or data received from another component (e.g., a sensor (360) or a communication circuit (310)) into volatile memory, process the instructions or data loaded into the volatile memory, and store resulting data in non-volatile memory.

[0060] The communication circuit (310) may support the establishment of a communication channel through a communication link (e.g., a first communication link (201) or a second communication link (202)) between an external electronic device (300) and an electronic device (101) and / or the performance of communication through the established communication channel. The communication circuit (310) may support the establishment of a communication channel through a communication link (e.g., a third communication link (203)) with another external electronic device (e.g., a second external electronic device (220) or a first external electronic device (210)) and / or the performance of communication through the established communication channel.

[0061] According to one embodiment, the communication circuit (310) may include a wireless communication module (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module.

[0062] According to one embodiment, the communication circuit (310) can communicate with the electronic device (101) via a first communication link (201) or a second communication link (202) (e.g., a short-range wireless communication network such as Bluetooth, WiFi direct, or infrared data association (IrDA)).

[0063] According to one embodiment, the communication circuit (310) can communicate with another external electronic device (e.g., a second external electronic device (220) or a first external electronic device (210)) via a third communication link (203) (e.g., a short-range wireless communication network such as Bluetooth, WiFi direct, or infrared data association (IrDA)).

[0064] The communication circuit (310) may include an antenna module. The antenna module of the communication circuit (310) may transmit signals and / or power to or receive signals and / or power from an external source (e.g., an electronic device (101)). In one embodiment, the antenna module of the communication circuit (310) may include an antenna including a radiator formed of a conductor or conductive pattern formed on a substrate (e.g., a PCB).

[0065] In one embodiment, the antenna module may include a plurality of antennas. In this case, among the plurality of antennas, at least one antenna suitable for a communication method used in a wireless communication network, such as the first communication link (201), the second communication link (202), and / or the third communication link (203), may be selected by the communication circuit (310). A signal or power may be transmitted or received between the communication circuit (310) and the electronic device (101) via the selected at least one antenna. In some embodiments, in addition to the radiator, other components (e.g., an RFIC) may be additionally formed as part of the antenna module.

[0066] The sensor (360) may include a contact or grip sensor, an acceleration sensor, a geomagnetic sensor, and / or a gyro sensor. The contact or grip sensor may detect when the external electronic device (300) comes into contact with, for example, the user's ear for a specified period of time and / or with a specified intensity, and transmit a sensor signal to the processor (320). The acceleration sensor, geomagnetic sensor, and / or gyro sensor may detect movement and / or inertia of the external electronic device (300). The acceleration sensor and / or gyro sensor may include a circuit (e.g., an integrated circuit, IC) that controls the operation of the acceleration sensor and / or gyro sensor. For example, the circuit (e.g., an integrated circuit, IC) that controls the operation of the acceleration sensor and / or gyro sensor may be included in the external electronic device (300) and implemented by the processor (320).

[0067] According to one embodiment, the speaker (350) can output an audio signal to the outside of the external electronic device (300). The processor (320) can output an electrical signal (audio signal) processed based on audio data received from a wirelessly connected electronic device (101) as sound through the speaker (350).

[0068] According to one embodiment, the microphone (340) can convert sounds acquired from the outside into electrical signals to generate audio data. The sounds input to the microphone (340) may include sounds generated in the user's external environment, for example, when an external electronic device (300) is worn. The microphone (340) may be implemented to include multiple microphones (e.g., a first microphone (341) and a second microphone (342)).

[0069] Referring to FIG. 4, when implemented with a plurality of microphones including a first microphone (341) and a second microphone (342), the first microphone (341) and the second microphone (342) may be spaced apart from each other at a specified interval and positioned at different locations in the housing (401) of the external electronic device (300). Accordingly, the sound entering the first microphone (341) and the sound entering the second microphone (342) may have different frequencies, latencies, and / or levels (e.g., decibels (dB)) depending on the location of the external sound source.

[0070] According to one embodiment, the processor (320) may transmit one or more audio data acquired through the microphone (340) to the electronic device (101). The processor may process the audio data received from the electronic device (101) and output it through the speaker (350).

[0071] Although not illustrated, the external electronic device (300) may include a battery for supplying power required for each component. The external electronic device (300) may further include a power management circuit (not illustrated) that controls the charging of the battery using power supplied from an external power source and manages the power supplied to each component. The battery may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, and / or a fuel cell.

[0072] FIG. 5 is a flowchart for explaining an audio signal generation operation of an electronic device (e.g., the electronic device (101) of FIG. 1 or FIG. 2) according to one embodiment.

[0073] In operation 501, a processor of an electronic device (101) (e.g., a processor (120) of FIG. 1) may receive audio data generated by one or more external electronic devices (300) (e.g., a first external electronic device (210) and / or a second external electronic device (220) of FIG. 2) via a communication circuit (e.g., a communication module (190) of FIG. 1).

[0074] According to various embodiments, the electronic device (101) may be a portable and / or movable electronic device such as a smart phone, a tablet PC, a portable multimedia player (PMP), a personal digital assistant (PDA), a laptop PC, and a wearable device.

[0075] According to various embodiments, the electronic device (101) may be connected to an external electronic device (300) using a short-range wireless communication network such as Bluetooth (or BLE), WiFi direct, or IrDA (infrared data association) to receive audio data or transmit various control information including control commands.

[0076] According to one embodiment, the first external electronic device (210) and the second external electronic device (220), which are external electronic devices (300), may include headphones, earphones, and / or ear buds that can be worn on each of the user's ears, respectively.

[0077] The processor (120) can receive audio data generated by the first external electronic device (210) and the second external electronic device (220) in real time. For example, the processor (120) can receive audio data acquired by the first external electronic device (210) and the second external electronic device (220) from a memory (e.g., the memory (130) of FIG. 1). The audio data can include audio data acquired by microphones of the first external electronic device (210) and the second external electronic device (220) (e.g., the first microphone (341) and / or the second microphone (342) of FIG. 3). For example, the audio data can be acquired simultaneously when the first external electronic device (210) is worn on the user's left ear and the second external electronic device (220) is worn on the user's right ear. The audio data may include one or more first audio data obtained from one or more microphones of a first external electronic device (210) and one or more second audio data obtained from one or more microphones of a second external electronic device (220).

[0078] In operation 503, the processor (120) can check the user's personal HRTF (head related transfer function) information. For example, the user's personal HRTF information may be stored in the memory (130).

[0079] If the user's personal HRTF information cannot be verified, the processor (120) can generate the user's personal HRTF information in operation 505. Operation 505 can be implemented to be performed in advance, and the user's personal HRTF information generated accordingly can be stored in the memory (130).

[0080] HRTF information includes values ​​modeled in the frequency domain by a system in which a person perceives external sounds with two ears. For example, by converting HRTF information into the time domain and applying the generated HRIR (Head Related Impulse Response) values ​​to each mono sound source, audio data transmitted to both ears of a person can be generated, thereby allowing the user to experience binaural sound. HRTF information according to various user conditions can be organized into an HRTF database and stored in a memory (130). The HRTF database can classify and store HRTF information of various users based on user information such as nationality, age, gender, head or face, and / or ear information. The HRTF database can store impulse response values ​​according to, for example, azimuth (elevation and / or azimuth) for each classification.

[0081] The processor (120) can obtain user information, such as personal information such as the user's nationality, age, and gender, and information about the face, head, and / or ears, from the memory (120). For example, the processor (120) can extract various information, such as depth information, from user face image data obtained through an image sensor or a camera (e.g., the camera module (180) of FIG. 1), and based on this, determine information about the user's face and / or ears, such as the user's head size or face width or horizontal length, the distance between both ears, and / or the position or shape of the ears.

[0082] According to one embodiment, the processor (120) can extract the user's personal HRTF by comparing it with reference HRTF information stored in an HRTF database based on the acquired user information and use it as the user's personal HRTF information.

[0083] According to one embodiment, the processor (120) may further correct the user's personal HRTF obtained from the HRTF database and use it as the user's personal HRTF information. For example, the processor (120) may generate a designated diagnostic sound through its own speaker (e.g., the sound output module (155) of FIG. 1) and acquire the generated diagnostic sound through microphones of the first external electronic device (210) and the second external electronic device (220), thereby receiving diagnostic sound audio data, and may analyze the diagnostic sound audio data based on user information and calculate HRTF information. The processor (120) may correct the user's personal HRTF information extracted from the HRTF database based on the calculated HRTF information and use it as the user's personal HRTF information.

[0084] In operation 507, the processor (120) can identify the location of the sound source from which the audio data was generated based on the input audio data. The processor (120) can identify the location of the sound source from which the audio data was generated based on two or more audio data, i.e., the first audio data and the second audio data, respectively obtained from the first external electronic device (210) and the second external electronic device (220). For example, the processor (320) can identify the location of the sound source from which the audio data was generated based on the time difference (interaural time difference) (e.g., frequency latency difference) and / or the level difference (interaural level difference) (e.g., decibel difference) of the audio data received from the external electronic device (300) worn on the user's left ear (e.g., the first external electronic device (210) of FIG. 2) and the external electronic device worn on the user's right ear (e.g., the second external electronic device (220) of FIG. 2). The sound source location identification method will be described in more detail with reference to FIGS. 7 to 10 below.

[0085] In operation 509, the processor (120) can generate an equalized audio signal by using the user's personal HRTF information to eliminate or reduce the user's personal HRTF for the input audio data.

[0086] The processor (120) can perform inverse filtering or other audio signal processing methods on user personal HRTF information for input audio data.

[0087] User personal HRTF information can be represented as matrix A in the following mathematical expression 1, including, for example, left and right HRFT values, and in this case, the inverse filter for inverse filtering is the inverse function A in mathematical expression 2. -1 It can be expressed as a matrix value.

[0088]

[0089]

[0090] In operation 511, the processor (120) can transmit the generated audio signal together with sound source location information to another electronic device (e.g., the electronic device (102, 104, or 108) of FIG. 1) through a communication circuit (e.g., the communication module (190) of FIG. 1).

[0091] FIG. 6 is a flowchart for explaining an audio signal reproduction operation of an electronic device (e.g., the electronic device (101) of FIG. 1 or FIG. 2) according to one embodiment.

[0092] In operation 601, a processor (e.g., a processor (120) of FIG. 1) of an electronic device (101) can receive an audio signal together with sound source location information from another electronic device (or a second electronic device) (e.g., an electronic device (102 or 104) of FIG. 1) through a communication circuit (e.g., a communication module (190) of FIG. 1). The electronic device (101) can receive the audio signal from the other electronic device (102 or 104) in real time, for example. The electronic device (101) can receive the audio signal directly from the other electronic device (102 or 104) or through a server (108).

[0093] According to various embodiments, the electronic device (101) may be a portable and / or movable electronic device such as a smart phone, a tablet PC, a portable multimedia player (PMP), a personal digital assistant (PDA), a laptop PC, and a wearable device.

[0094] According to various embodiments, the other electronic device (102 or 104) may be a portable and / or movable electronic device such as a smart phone, a tablet PC, a portable multimedia player (PMP), a personal digital assistant (PDA), a laptop PC, and a wearable device. The other electronic device (102 or 104) may be connected to the electronic device (101) to transmit audio data using a short-range communication network such as Bluetooth (or BLE), WiFi direct, or IrDA (infrared data association), or a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or a WAN).

[0095] In operation 603, the processor (120) can check the user's personal HRTF (head related transfer function) information. For example, the user's personal HRTF information may be stored in the memory (130).

[0096] If the user's personal HRTF information cannot be verified, the processor (120) may generate the user's personal HRTF information in operation 605. Operation 605 may be implemented to be performed in advance, and the user's personal HRTF information generated accordingly may be stored in the memory (130).

[0097] The processor (120) can obtain user information, such as information about the user's nationality, age, gender, head or face, and / or ears, from the memory (120). For example, the processor (120) can extract various information, such as depth information, from user face image data obtained through a camera (e.g., a camera module (180) of FIG. 1), and based on this, determine information about the user's face and / or ears, such as the width of the user's face, the distance between the two ears, and / or the position or shape of the ears.

[0098] According to one embodiment, the processor (120) can extract the user's personal HRTF by comparing it with reference HRTF information stored in an HRTF database based on the acquired user information and use it as the user's personal HRTF information.

[0099] According to one embodiment, the processor (120) may further correct the user's personal HRTF extracted from the HRTF database and use it as the user's personal HRTF information. For example, the processor (120) may generate a designated diagnostic sound through its own speaker (e.g., the sound output module (155) of FIG. 1) and acquire the generated diagnostic sound through microphones of the first external electronic device (210) and the second external electronic device (220), thereby receiving diagnostic sound audio data, and may analyze the diagnostic sound audio data based on user information and calculate HRTF information. The processor (120) may correct the user's personal HRTF information extracted from the HRTF database based on the calculated HRTF information and use it as the user's personal HRTF information.

[0100] In operation 607, the processor (120) may apply sound source location information to the received audio signal to generate two audio data for generating left and right binaural sounds. For example, the processor (320) may reflect an interaural time difference (e.g., a frequency latency difference) and / or an interaural level difference (e.g., a decibel difference) according to the location of the sound source in the audio data to be transmitted to an external electronic device (300) worn on the user's left ear (e.g., a first external electronic device (210) of FIG. 2) and an external electronic device worn on the user's right ear (e.g., a second external electronic device (220) of FIG. 2). The method for reflecting the sound source location may be described in more detail with reference to FIGS. 7 to 10 below.

[0101] In operation 609, the processor (120) may perform signal processing on the input audio data using the user's personal HRTF information and generate personalized audio data. For example, the processor (120) may perform filtering or other audio signal processing methods based on the user's personal HRTF information on the input audio data.

[0102] According to one embodiment, the user's personal HRTF information can be represented by the matrix A value in the mathematical expression 1 described above, for example, to reflect HRFT values ​​for left and right audio data, and in this case, the filter for filtering can be represented by the function matrix A value of the mathematical expression 1.

[0103] According to one embodiment, user personal HRTF information can be converted into, for example, a time domain to generate HRIR, and personalized audio data can be generated by applying the HRIR of each ear to input audio data.

[0104] In operation 611, the processor (120) may transmit two audio data for generating the generated binaural sound to the first external electronic device (210) and the second external electronic device (220), which are external electronic devices (300), through a communication circuit (e.g., the communication module (190) of FIG. 1), so that the two audio data may be played back. The first external electronic device (210) and the second external electronic device (220) may include headphones, earphones, and / or ear buds that may be worn on each of the user's ears.

[0105] Each of the first external electronic device (210) and the second external electronic device (220) can process the corresponding left or right audio data among the binaural audio data received from the electronic device (101) and output it through a speaker (e.g., the speaker (350) of FIG. 3). Accordingly, a user wearing the first external electronic device (210) and the second external electronic device (220) can enjoy binaural sound with a more vivid sense of presence as his / her HRTF is applied to the sound generated at the received sound source location.

[0106] FIG. 7 is a drawing for explaining an operation of confirming or applying sound source location information of an electronic device (e.g., the electronic device (101) of FIG. 1 or FIG. 2) according to one embodiment.

[0107] Referring to FIG. 7, graph (a) may represent audio data acquired by, for example, a first external electronic device (210) worn on the user's left ear, and graph (b) may represent audio data acquired by, for example, a second external electronic device (220) worn on the user's right ear.

[0108] Referring to the drawing, depending on the distance until a sound signal generated at a location of a sound source reaches a location of a first external electronic device (210) or a second external electronic device (220), a signal frequency latency may occur and a change in the signal level may also occur. Accordingly, a difference in signal frequency latency and / or a signal level may exist between a first point (701) where audio data is acquired at a first external electronic device (210) by a sound signal generated at a location of a sound source and a second point (702) where audio data is acquired at a second external electronic device (220). Based on this difference in signal frequency latency and / or the difference in signal level, the distance to the location of the sound source where the sound signal is generated can be calculated.

[0109] According to one embodiment, the memory of the electronic device (101) (e.g., the memory (130) of FIG. 1) may store data for calculating the location of a sound source based on a frequency latency difference and / or a signal level difference of audio data obtained from the first external electronic device (210) and the second external electronic device (220), respectively. For example, the data for calculating the location of the sound source may include a HRTF database, and the HRTF database may include data for confirming sound source location information, for example, information on two audio data according to the distance and angle to the sound source, for example, information on the frequency latency difference and / or the signal level difference. The information on the two audio data included in the HRTF database may be information calculated by applying a dummy head (e.g., a mannequin head shape), for example.

[0110] According to one embodiment, the processor (120) of the electronic device (101) can identify a sound source location based on two audio data or generate binaural audio data by applying the sound source location.

[0111] According to one embodiment, the processor (120) of the electronic device (101) can determine the location of a sound source, for example, a distance and a 360-degree azimuth, based on three or more simultaneously acquired audio data, for example, by using information about a frequency latency difference and / or a signal level difference of each audio data.

[0112] According to one embodiment, the processor (120) of the electronic device (101) can determine the location of a sound source, for example, distance and 360-degree azimuth, based on two simultaneously acquired audio data.

[0113] Hereinafter, with reference to FIGS. 8, 9, and 10, a method for identifying a location of a sound source based on two pieces of audio data or generating two pieces of audio data by applying a location of a sound source will be described. Hereinafter, audio data acquired or reproduced by a first external electronic device worn on a user's left ear (e.g., the first external electronic device (210) of FIG. 1) may be referred to as first audio data or left audio data (L), and audio data acquired or reproduced by a second external electronic device worn on a user's right ear (e.g., the second external electronic device (220) of FIG. 1) may be referred to as second audio data or right audio data (R).

[0114] Referring to FIG. 8, graph (a) may represent, for example, left audio data (L) and right audio data (R) when the sound source location is 45 degrees to the left of the front (315 degrees in a 360-degree azimuth). Graph (b) may represent, for example, left audio data (L) and right audio data (R) when the sound source location is 0 degrees, i.e., forward. Graph (c) may represent, for example, left audio data (L) and right audio data (R) when the sound source location is 45 degrees to the right of the front (45 degrees).

[0115] In graph (a) of Fig. 8, when the sound source location is 45 degrees to the left in front (315 degrees in a 360-degree azimuth), it can be seen that the right audio data (R) is relatively later in terms of frequency latency than the left audio data (L) and has a relatively low level. Graph (b) shows, for example, a case where the sound source location is forward, i.e., 0 degrees, and it can be seen that there is practically no difference in frequency latency and level between the left audio data (L) and the right audio data (R). Graph (c) shows, for example, a case where the sound source location is 45 degrees to the right in front (45 degrees), and it can be seen that the right audio data (R) is relatively earlier in terms of frequency latency than the left audio data (L) and has a relatively high level.

[0116] Referring to FIG. 9, graph (a) may represent left audio data (L) and right audio data (R) when, for example, the sound source position is 90 degrees to the left (270 degrees in a 360-degree azimuth). Graph (b) may represent left audio data (L) and right audio data (R) when, for example, the sound source position is 90 degrees to the right (90 degrees).

[0117] In graph (a) of Fig. 9, for example, when the sound source location is 90 degrees to the left (270 degrees in a 360-degree azimuth), the right audio data (R) is relatively later in the audio data acquisition time according to the frequency latency than the left audio data (L) and the level is relatively low. In graph (b), for example, when the sound source location is 90 degrees to the right (90 degrees), the right audio data (R) is relatively earlier in the audio data acquisition time according to the frequency latency than the left audio data (L) and the level is relatively high.

[0118] Referring to FIG. 10, graph (a) can represent left audio data (L) and right audio data (R) when, for example, the sound source location is 45 degrees to the rear left (225 degrees in a 360-degree azimuth). Graph (b) can represent left audio data (L) and right audio data (R) when, for example, the sound source location is 180 degrees to the rear. Graph (c) can represent left audio data (L) and right audio data (R) when, for example, the sound source location is 45 degrees to the rear right (135 degrees).

[0119] In graph (a) of Fig. 10, for example, when the sound source is located 45 degrees to the left rear (225 degrees in a 360-degree azimuth), the right audio data (R) can be seen to have a relatively late audio data acquisition time according to the frequency latency and a relatively low level compared to the left audio data (L). In graph (b), for example, when the sound source is located 180 degrees to the rear, the difference in frequency latency and level between the left audio data (L) and the right audio data (R) can be seen to be practically non-existent. In graph (c), for example, when the sound source is located 45 degrees to the right rear (135 degrees), the right audio data (R) can be seen to have a relatively early audio data acquisition time according to the frequency latency and a relatively high level compared to the left audio data (L).

[0120] According to one embodiment, when comparing graph (a) of FIG. 8 with graph (a) of FIG. 10, it can be seen that when the sound source location is 45 degrees to the left in front, the signal level intensity of the high frequency region (801) of the left audio data is relatively higher than the signal level intensity of the high frequency region (1001) of the left audio data when it is 45 degrees to the left in the rear. In addition, when comparing graph (c) of FIG. 8 with graph (c) of FIG. 10, it can be seen that when the sound source location is 45 degrees to the right in front, the signal level intensity of the high frequency region (802) of the right audio data is relatively higher than the signal level intensity of the high frequency region (1002) of the right audio data when it is 45 degrees to the right in the rear. Accordingly, it is possible to determine whether the sound source location is front or rear based on the signal level intensity of the high frequency region.

[0121] According to one embodiment, the memory of the electronic device (101) (e.g., the memory (130) of FIG. 1) may store data for calculating the location of a sound source based on a signal level difference in a high-frequency region in addition to a frequency latency difference and / or a signal level of audio data obtained from each of the first external electronic device (210) and the second external electronic device (220). For example, the data for calculating the location of a sound source may include a HRTF database, and the HRTF database may further include data for confirming sound source location information, for example, information about two audio data according to the distance and angle to the sound source, for example, information about a signal level difference in a high-frequency region in addition to the frequency latency difference and / or the signal level difference. The information about the two audio data included in the HRTF database may be, for example, information calculated by applying a dummy head (e.g., a mannequin head shape).

[0122] According to one embodiment, the processor (120) of the electronic device (101) can identify a sound source location based on two audio data or generate binaural audio data by applying the sound source location.

[0123] In various embodiments, another electronic device (e.g., electronic device (101) of FIG. 1) may include a communication circuit (e.g., communication module (190) of FIG. 1), a memory (e.g., memory (130) of FIG. 1), and a processor (e.g., processor (120) of FIG. 1) operatively connected to the communication circuit and the memory. The memory may store instructions that, when executed by the processor, cause the electronic device to receive two or more audio data simultaneously acquired by one or more external electronic devices (e.g., the external electronic devices (210, 220 and / or 300) of FIG. 2, FIG. 3 or FIG. 4) through the communication circuit, identify user personal HRTF information acquired based on reference HRTF (Head Related Transfer Function) information using user information acquired from the memory, identify sound source generation location information of the two or more audio data based on the two or more audio data, perform inverse filtering on the two or more audio data based on the user HRTF information to generate an audio signal, and transmit the audio signal and the sound source generation location information to another electronic device through the communication circuit.

[0124] According to one embodiment, the memory may further store instructions that, when executed by the processor, cause the electronic device to obtain sound source generation location information of the two or more audio data based on at least one of a time difference or a level difference between the two or more audio data.

[0125] According to one embodiment, the memory may further store instructions that, when executed by the processor, cause the electronic device to receive three or more audio data simultaneously acquired by the one or more external electronic devices and to acquire sound source generation location information of the three or more audio data based on at least one of a time difference or a level difference between the three or more audio data.

[0126] According to one embodiment, the electronic device may further include an image sensor (e.g., the camera module (180) of FIG. 1). The memory may further store instructions that, when executed by the processor, cause the electronic device to obtain the user's personal HRTF information by further utilizing user information obtained from the user's image obtained through the image sensor.

[0127] According to one embodiment, the electronic device may further include a speaker (e.g., the sound output module (155) of FIG. 1). The memory may further store instructions that, when executed by the processor, cause the electronic device to generate a diagnostic sound through the speaker, receive two or more diagnostic sound audio data generated by the one or more external electronic devices according to generation of the diagnostic sound, analyze the diagnostic sound audio data using the user information to generate HRTF information, and correct the user's personal HRTF information using the generated HRTF information.

[0128] According to one embodiment, the user information includes one or more of the user's nationality, age, gender, head size, and ear shape information, and the memory may further store instructions that, when executed by the processor, cause the electronic device to obtain the reference HRTF information from the memory using the user information and obtain the user's personal HRTF information based thereon.

[0129] According to one embodiment, the memory may further store instructions that, when executed by the processor, cause the electronic device to represent the user HRFT information as a determinant and perform the inverse filtering by applying the inverse matrix of the determinant to the two or more audio data.

[0130] According to one embodiment, a method of an electronic device may include an operation of receiving two or more audio data simultaneously acquired by one or more external electronic devices, an operation of confirming user personal HRTF information acquired based on reference HRTF (Head Related Transfer Function) information using user information, an operation of confirming sound source generation location information of the two or more audio data based on the two or more audio data, an operation of generating an audio signal by performing inverse filtering on the two or more audio data based on the user HRTF information, and an operation of transmitting the audio signal and the sound source generation location information to another electronic device.

[0131] According to one embodiment, the method may further include an operation of obtaining sound source generation location information of the two or more audio data based on at least one of a time difference or a level difference between the two or more audio data.

[0132] According to one embodiment, the method may further include an operation of receiving three or more audio data simultaneously acquired by the one or more external electronic devices, and an operation of acquiring sound source generation location information of the three or more audio data based on at least one of a time difference or a level difference between the three or more audio data.

[0133] According to one embodiment, the method may further include an operation of obtaining the reference HRTF information from the memory using the user information, wherein the user information includes at least one of the user's nationality, age, gender, head size, and ear shape information, and obtaining the user's personal HRTF information based thereon.

[0134] According to one embodiment, the method may further include an operation of representing the user personal HRFT information as a determinant, and an operation of performing the inverse filtering by applying the inverse matrix of the determinant to the two or more audio data.

[0135] According to one embodiment, an electronic device (e.g., an electronic device (101) of FIG. 1) may include a communication circuit (e.g., a communication module (190) of FIG. 1), a memory (e.g., a memory (130) of FIG. 1), and a processor (e.g., a processor (120) of FIG. 1) operatively connected to the communication circuit and the memory. The memory may store instructions that, when executed by the processor, cause the electronic device to receive a binaural audio signal and sound source generation location information of the audio signal from another electronic device through the communication circuit, to use user information acquired from the memory to identify user personal HRTF information acquired based on reference HRTF (Head Related Transfer Function) information, to apply the user personal HRTF information and the location information to the audio signal to generate two audio data, and to transmit the two audio data to one or more external electronic devices so that the two audio data can be played by the one or more external electronic devices.

[0136] According to one embodiment, the electronic device further includes an image sensor (e.g., a camera module (180) of FIG. 1), and the memory may further store instructions that, when executed by the processor, cause the electronic device to obtain the user personal HRTF information by further utilizing user information obtained from an image of the user obtained through the image sensor.

[0137] According to one embodiment, the electronic device further includes a speaker (e.g., an audio output module (155) of FIG. 1), and the memory may further store instructions that, when executed by the processor, cause the electronic device to generate a diagnostic sound through the speaker, receive two or more diagnostic sound audio data generated by the one or more external electronic devices according to generation of the diagnostic sound, analyze the diagnostic sound audio data using the user information to generate HRTF information, and correct the user personal HRTF information using the generated HRTF information.

[0138] According to one embodiment, the user information includes one or more of the user's nationality, age, gender, head size, and ear shape information, and the memory may further store instructions that, when executed by the processor, cause the electronic device to obtain the reference HRTF information from the memory using the user information and obtain the user HRTF information based thereon.

[0139] According to one embodiment, the memory may further store instructions that, when executed by the processor, cause the electronic device to represent the user HRFT information as a determinant and apply the determinant and the position information to the audio signal to obtain the two audio data.

[0140] According to one embodiment, a method of an electronic device may include an operation of receiving a binaural audio signal and information on a sound source generation location of the audio signal from another electronic device through the communication circuit, an operation of confirming user personal HRTF information acquired based on reference HRTF (Head Related Transfer Function) information using user information acquired from the memory, an operation of generating two audio data by applying the user personal HRTF information and the location information to the audio signal, and an operation of transmitting the two audio data to one or more external electronic devices so that the two audio data are played by the one or more external electronic devices.

[0141] According to one embodiment, the user information may include one or more of the user's nationality, age, gender, head size, and ear shape information, and may further include an operation of obtaining the reference HRTF information using the user information and obtaining the user HRTF information based thereon.

[0142] According to one embodiment, the method may further include an operation of representing the user HRFT information as a determinant, and an operation of obtaining the two audio data by applying the determinant and the position information to the audio signal. Electronic devices according to various embodiments disclosed in the present document may be devices of various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to embodiments of the present document are not limited to the aforementioned devices.

[0143] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0144] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0145] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0146] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0147] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0148] The embodiments disclosed in this document are merely examples presented to facilitate easy explanation and understanding of the technical content, and are not intended to limit the scope of the technology disclosed in this document. Therefore, the scope of the technology disclosed in this document should be interpreted to include all modifications or variations derived based on the technical concepts of the various embodiments disclosed in this document, in addition to the embodiments disclosed herein.

Claims

1. In electronic devices, communication circuit; memory; and a processor operatively connected to the communication circuit and the memory; The above memory, when executed by the processor, causes the electronic device to: Receiving two or more audio data simultaneously acquired by one or more external electronic devices through the above communication circuit, Using the user information acquired from the above memory, the user's personal HRTF information acquired based on the reference HRTF (Head Related Transfer Function) information is verified, Based on the two or more audio data, the sound source generation location information of the two or more audio data is confirmed, Generating an audio signal by performing inverse filtering on the two or more audio data based on the user HRFT information, A device storing instructions for transmitting the above audio signal and the sound source generation location information to another electronic device through the communication circuit.

2. In paragraph 1, The above memory further stores instructions that, when executed by the processor, cause the electronic device to obtain sound source generation location information of the two or more audio data based on at least one of a time difference or a level difference between the two or more audio data.

3. In paragraph 1, The above memory, when executed by the processor, causes the electronic device to: Receiving three or more audio data simultaneously acquired by one or more external electronic devices, A device further storing instructions for obtaining sound source generation location information of three or more audio data based on at least one of a time difference or a level difference between the three or more audio data.

4. In paragraph 1, Including more image sensors, The above memory further stores instructions that, when executed by the processor, cause the electronic device to obtain the user personal HRTF information by further utilizing the user information obtained from the image of the user obtained through the image sensor.

5. In paragraph 1, Including more speakers, The above memory, when executed by the processor, causes the electronic device to: Generate diagnostic sounds through the above speakers, Receiving two or more diagnostic sound audio data generated by the one or more external electronic devices according to the occurrence of the above diagnostic sound, A device further storing instructions for analyzing the diagnostic sound audio data using the user information to generate HRTF information and correcting the user's personal HRTF information using the generated HRTF information.

6. In paragraph 1, The above user information includes one or more of the user's nationality, age, gender, head size, and ear shape information. The memory further stores instructions that, when executed by the processor, cause the electronic device to obtain the reference HRTF information from the memory using the user information and obtain the user personal HRTF information based thereon.

7. In paragraph 1, The above memory further stores instructions that, when executed by the processor, cause the electronic device to represent the user HRFT information as a determinant and perform the inverse filtering by applying the inverse matrix of the determinant to two or more pieces of audio data.

8. In the method of an electronic device, An act of receiving two or more audio data acquired simultaneously by one or more external electronic devices; An action of verifying user personal HRTF information obtained based on standard HRTF (Head Related Transfer Function) information using user information; An operation of confirming sound source generation location information of two or more audio data based on the two or more audio data; An operation of generating an audio signal by performing inverse filtering on the two or more audio data based on the user HRFT information; and A method comprising the action of transmitting the audio signal and the sound source generation location information to another electronic device.

9. In paragraph 8, A method further comprising an operation of obtaining sound source generation location information of two or more audio data based on at least one of a time difference or a level difference between the two or more audio data.

10. In paragraph 8, An operation of receiving three or more audio data simultaneously acquired by one or more external electronic devices; and A method further comprising an operation of obtaining sound source generation location information of three or more audio data based on at least one of a time difference or a level difference between the three or more audio data.

11. In paragraph 8, The above user information includes one or more of the user's nationality, age, gender, head size, and ear shape information. A method further comprising: obtaining the reference HRTF information from the memory using the user information and obtaining the user personal HRTF information based thereon.

12. In paragraph 8, An operation of representing the above user personal HRFT information as a matrix; and A method further comprising an operation of performing the inverse filtering by applying the inverse matrix of the determinant to the two or more audio data.

13. In electronic devices, communication circuit; memory; and a processor operatively connected to the communication circuit and the memory; The above memory, when executed by the processor, causes the electronic device to: Receives a binaural audio signal and information on the location of the sound source of the audio signal from another electronic device through the above communication circuit, Using the user information acquired from the above memory, the user's personal HRTF information acquired based on the reference HRTF (Head Related Transfer Function) information is verified, Generating two audio data by applying the user personal HRTF information and the location information to the above audio signal, A device storing instructions for transmitting the two audio data to one or more external electronic devices so that the two audio data can be played back by the one or more external electronic devices.

14. In paragraph 13, Including more image sensors, The above memory further stores instructions that, when executed by the processor, cause the electronic device to obtain the user personal HRTF information by further utilizing the user information obtained from the image of the user obtained through the image sensor.

15. In paragraph 13, Including more speakers, The above memory, when executed by the processor, causes the electronic device to: Generate diagnostic sounds through the above speakers, Receiving two or more diagnostic sound audio data generated by the one or more external electronic devices according to the occurrence of the above diagnostic sound, A device further storing instructions for analyzing the diagnostic sound audio data using the user information to generate HRTF information and correcting the user's personal HRTF information using the generated HRTF information.

Images