Electronic device for obtaining voice signal and operation method thereof

Abstract

One embodiment of the present invention relates to an apparatus and a method for transmitting voice data in an electronic device. The electronic device may comprise: a communication circuit; a processor; and a memory storing instructions which, when individually or collectively executed by the at least one processor, instruct the electronic device to receive audio signals from external electronic devices, perform beamforming on a frequency band in which beamforming is possible on the received audio signals, update a beamforming variable on the basis of voice components or noise components included in the received audio signals when it is determined that distortion has not occurred in the audio signals by the beamforming, and limit the update of the beamforming variable when it is determined that distortion has occurred in the audio signals by the beamforming. Other embodiments may also be possible.

Description

Electronic device for acquiring voice signals and method of operation thereof

[0001] An embodiment of the present invention relates to an electronic device for obtaining a voice signal and a method of operating the same.

[0002] Advances in information and communication technology and semiconductor technology are evolving various electronic devices into multimedia devices offering a variety of multimedia functions. These multimedia functions may include at least one of the following: voice calling, video calling, messaging, broadcasting, wireless Internet, camera, electronic payment, or content playback.

[0003] An electronic device can enhance the user's convenience in using multimedia functions by utilizing at least one other electronic device. For example, the electronic device can output audio signals (e.g., voice and / or music) through at least one audio device. The electronic device can collect audio signals through at least one audio device.

[0004] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

[0005] When an electronic device performs a voice-based function, such as a voice call, voice recording, or voice recognition, it may acquire an audio signal (e.g., voice) through at least one external electronic device. For example, the external electronic device may include a wearable device (e.g., earphone device) worn on a part of the user's body.

[0006] When an external electronic device is worn on a part of the user's body (e.g., an ear), the reception performance of voice data through the external electronic device may be relatively poor because the distance between at least one microphone included in the external electronic device and the user's mouth is relatively far.

[0007] When an electronic device acquires voice data using an external electronic device, the service quality of voice-based functions may deteriorate due to a decrease in the reception performance of the voice data by the external electronic device.

[0008] One embodiment of the present invention discloses a device and method for obtaining an audio signal using a plurality of external electronic devices in an electronic device.

[0009] The technical problems to be achieved in this document are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present invention belongs from the description below.

[0010] According to one embodiment, an electronic device may include a communication circuit, at least one processor including a processing circuit, and a memory operatively connected to the processor. According to one embodiment, the memory may store instructions that, when individually or collectively executed by the at least one processor, cause the electronic device to receive audio signals from external electronic devices. According to one embodiment, the memory may store instructions that, when individually or collectively executed by the at least one processor, cause the electronic device to identify a frequency band in which beamforming is possible in audio signals. According to one embodiment, the memory may store instructions that, when individually or collectively executed by the at least one processor, cause the electronic device to perform beamforming on an audio signal in a frequency band in which beamforming is possible. According to one embodiment, the memory may store instructions that, when individually or collectively executed by the at least one processor, cause the electronic device to determine whether distortion has occurred in an audio signal due to beamforming. According to one embodiment, the memory may store instructions that, when individually or collectively executed by at least one processor, cause the electronic device to update beamforming variables based on speech components or noise components included in audio signals when it determines that beamforming has not caused distortion in the audio signal. According to one embodiment, the memory may store instructions that, when individually or collectively executed by at least one processor, cause the electronic device to limit updating of beamforming variables when it determines that beamforming has caused distortion in the audio signal.

[0011] According to one embodiment, a method of operating an electronic device may include receiving audio signals from a plurality of external electronic devices. According to one embodiment, the method of operating an electronic device may include identifying a frequency band in which beamforming is possible in audio signals. According to one embodiment, the method of operating an electronic device may include performing beamforming on an audio signal in a frequency band in which beamforming is possible. According to one embodiment, the method of operating an electronic device may include identifying whether distortion has occurred in an audio signal due to beamforming. According to one embodiment, the method of operating an electronic device may include updating a beamforming variable based on a voice component or a noise component included in the audio signals when it is determined that distortion has occurred in the audio signal due to beamforming. According to one embodiment, the method of operating an electronic device may include limiting updating of a beamforming variable when it is determined that distortion has occurred in the audio signal due to beamforming.

[0012] According to one embodiment, a non-transitory computer-readable storage medium (or a computer program product) storing one or more programs may be described. According to one embodiment, the one or more programs may include instructions that, when individually or collectively executed by at least one processor of an electronic device, cause the following operations: receiving audio signals from a plurality of external electronic devices; identifying a frequency band in which beamforming is possible in the received audio signals; performing beamforming on an audio signal in the frequency band in which beamforming is possible; identifying whether distortion has occurred in the audio signal due to the beamforming; updating a beamforming variable based on a voice component or a noise component included in the received audio signals when it is determined that distortion has not occurred in the audio signal due to the beamforming; and limiting the updating of the beamforming variable when it is determined that distortion has occurred in the audio signal due to the beamforming.

[0013] According to one embodiment of the present invention, an electronic device can improve the quality of an audio signal by obtaining an audio signal through beamforming based on audio signals received from a plurality of external electronic devices including at least one microphone.

[0014] According to one embodiment, the electronic device can reduce distortion of audio signals caused by wireless connections between the electronic device and external electronic devices by selectively updating variables for beamforming based on whether distortion occurs in audio signals received from a plurality of external electronic devices.

[0015] The effects that can be obtained from the embodiments of the present invention are not limited to the effects mentioned above, and other effects that are not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the embodiments of the present invention belong from the description below.

[0016] In connection with the description of the drawings, the same or similar reference numerals may be used for the same or similar components.

[0017] FIG. 1 is a block diagram of an electronic device within a network environment, according to one embodiment.

[0018] Figure 2 is a block diagram of an audio module according to one embodiment.

[0019] FIG. 3 is an example of wirelessly connected electronic devices and external electronic devices according to one embodiment.

[0020] FIG. 4 is a block diagram of an electronic device and an external electronic device for obtaining an audio signal according to one embodiment.

[0021] FIG. 5 is a block diagram of an audio processing circuit of an external electronic device for obtaining an audio signal according to one embodiment.

[0022] FIG. 6 is a block diagram of an audio processing circuit of an electronic device for obtaining an audio signal according to one embodiment.

[0023] FIG. 7 is a flowchart for obtaining an audio signal from an external electronic device according to one embodiment.

[0024] FIG. 8 is a flowchart for obtaining an audio signal in an electronic device according to one embodiment.

[0025] FIG. 9 is a flowchart for confirming a frequency band in which beam forming is possible in an electronic device according to one embodiment.

[0026] FIG. 10 is an example of an audio signal obtained through beam forming in an electronic device according to one embodiment.

[0027] FIG. 11 is a flowchart for updating beam forming variables in an electronic device according to one embodiment.

[0028] FIG. 12 is an example of an audio signal obtained by selectively updating beam forming variables in an electronic device according to one embodiment.

[0029] FIG. 13 is an example of an audio signal obtained through beam forming in an electronic device located in a first region according to one embodiment.

[0030] The following examples are described in detail with reference to the attached drawings.

[0031] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to one embodiment. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with the electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of the electronic device (104) or the server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0032] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or a secondary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor)) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a specified function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0033] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of, but is not limited to, a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more thereof. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may also include a software structure.

[0034] The memory (130) can store various data used by at least one component (e.g., a processor (120) or a sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., a program (140)) and input data or output data for commands related thereto. The memory (130) can include a volatile memory (132) or a non-volatile memory (134).

[0035] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0036] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0037] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0038] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by a touch.

[0039] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0040] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0041] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0042] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., the electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0043] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0044] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0045] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least a part of a power management integrated circuit (PMIC).

[0046] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0047] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, Wi-Fi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0048] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.According to one embodiment, the subscriber identification module (196) may include multiple subscriber identification modules. For example, the multiple subscriber identification modules may store different subscriber information.

[0049] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0050] In one embodiment, the antenna module (197) may form a high-frequency (e.g., mmWave) antenna module. In one embodiment, the high-frequency (e.g., mmWave) antenna module may include a printed circuit board, an RFIC positioned on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) positioned on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band. For example, the plurality of antennas may include patch array antennas and / or dipole array antennas.

[0051] At least some of the components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)).

[0052] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0053] Figure 2 is a block diagram (200) of an audio module (170) according to one embodiment.

[0054] According to one embodiment referring to FIG. 2, the audio module (170) may include at least one of an audio input interface (210), an audio input mixer (220), an analog to digital converter (ADC) (230), an audio signal processor (240), a digital to analog converter (DAC) (250), an audio output mixer (260), or an audio output interface (270).

[0055] The audio input interface (210) can receive an audio signal corresponding to a sound acquired from the outside of the electronic device (101) as part of the input module (150) or through a microphone (e.g., a dynamic microphone, a condenser microphone, or a piezo microphone) configured separately from the electronic device (101). For example, when the audio signal is acquired from an external electronic device (102) (e.g., a headset or a microphone), the audio input interface (210) can receive the audio signal by being directly connected to the external electronic device (102) through a connection terminal (178) or wirelessly (e.g., Bluetooth communication) through a wireless communication module (192). According to one embodiment, the audio input interface (210) can receive a control signal (e.g., a volume control signal received through an input button) related to the audio signal acquired from the external electronic device (102). The audio input interface (210) includes a plurality of audio input channels, and can receive different audio signals for each corresponding audio input channel among the plurality of audio input channels. According to one embodiment, additionally or alternatively, the audio input interface (210) may receive audio signals from other components of the electronic device (101), such as a processor (120) or memory (130).

[0056] The audio input mixer (220) can synthesize a plurality of input audio signals into at least one audio signal. For example, the audio input mixer (220) can synthesize a plurality of analog audio signals input through the audio input interface (210) into at least one analog audio signal.

[0057] The ADC (230) can convert an analog audio signal into a digital audio signal. For example, the ADC (230) can convert an analog audio signal received through an audio input interface (210) or an analog audio signal synthesized through an audio input mixer (220) into a digital audio signal.

[0058] The audio signal processor (240) may perform various processing on a digital audio signal input through the ADC (230) or a digital audio signal received from another component of the electronic device (101). For example, the audio signal processor (240) may perform at least one of changing a sampling rate, applying one or more filters, interpolating, amplifying or attenuating all or part of a frequency band, noise processing (e.g., noise or echo reduction), changing a channel (e.g., switching between mono and stereo), mixing, or extracting a specified signal on one or more digital audio signals. According to one embodiment, one or more functions of the audio signal processor (240) may be implemented in the form of an equalizer.

[0059] The DAC (250) can convert a digital audio signal into an analog audio signal. For example, the DAC (250) can convert a digital audio signal processed by an audio signal processor (240) or a digital audio signal obtained from another component of the electronic device (101) (e.g., a processor (120) or a memory (130)) into an analog audio signal.

[0060] The audio output mixer (260) can synthesize a plurality of audio signals to be output into at least one audio signal. For example, the audio output mixer (260) can synthesize an audio signal converted into analog through the DAC (250) and another analog audio signal (e.g., an analog audio signal received through the audio input interface (210)) into at least one analog audio signal.

[0061] The audio output interface (270) can output an analog audio signal converted by the DAC (250) or an analog audio signal synthesized by the audio output mixer (260) to the outside of the electronic device (101) through the audio output module (155). For example, the audio output module (155) can include a speaker, such as a dynamic driver or a balanced armature driver, or a receiver. According to one embodiment, the audio output module (155) can include a plurality of speakers. In this case, the audio output interface (270) can output an audio signal having a plurality of different channels (e.g., stereo or 5.1 channels) through at least some of the speakers among the plurality of speakers. According to one embodiment, the audio output interface (270) can be directly connected to an external electronic device (102) (e.g., an external speaker or headset) through a connection terminal (178) or wirelessly through a wireless communication module (192) to output an audio signal.

[0062] According to one embodiment, the audio module (170) can generate at least one digital audio signal by synthesizing a plurality of digital audio signals using at least one function of the audio signal processor (240) without separately having an audio input mixer (220) or an audio output mixer (260).

[0063] According to one embodiment, the audio module (170) may include an audio amplifier (not shown) (e.g., a speaker amplifier circuit) capable of amplifying an analog audio signal input through the audio input interface (210) or an audio signal to be output through the audio output interface (270). According to one embodiment, the audio amplifier may be configured as a separate module from the audio module (170).

[0064] An electronic device according to an embodiment disclosed in this document may take various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance. The electronic device according to an embodiment of this document is not limited to the aforementioned devices.

[0065] It should be understood that the embodiments of this document and the terminology used herein are not intended to limit the technical features described in this document to a specific embodiment, but include various modifications, equivalents, or substitutes of the embodiment. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the item, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0066] The term "module" used in one embodiment of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. It may be a module, an integrally formed component, or a minimum unit or portion of the component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0067] An embodiment of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0068] According to one embodiment, the method according to one embodiment disclosed in the present document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or may be provided through an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smart phones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0069] According to one embodiment, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to one embodiment, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to one embodiment, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0070] FIG. 3 illustrates examples of wirelessly connected electronic devices and external electronic devices according to one embodiment. For example, the electronic device (101) of FIG. 3 may be at least partially similar to the electronic device (101) of FIG. 1 or may include other embodiments of the electronic device (101). For example, the external electronic devices (300 and / or 320) of FIG. 3 may be at least partially similar to the electronic devices (102, 104) of FIG. 1 or may include other embodiments of the electronic devices (102, 104).

[0071] According to one embodiment referring to FIG. 3, an electronic device (101) may be wirelessly connected to a plurality of external electronic devices (300 and 320) that may be worn on a part of a user's body based on a first communication method. For example, the first communication method may include Bluetooth, BLE (Bluetooth low energy), or wireless LAN.

[0072] According to one embodiment, the plurality of external electronic devices (300 and 320) may include wireless earphones that are implemented as a pair and worn on different parts of the user's body (e.g., the right ear or the left ear).

[0073] According to one embodiment, the first external electronic device (300) may include a first case (301), a housing (303) including a second case (302) at least partially coupled with the first case (301), and / or an ear tip (312) detachably coupled to the housing (303). According to one embodiment, the second external electronic device (320) may include a first case (321), a housing (323) including a second case (322) at least partially coupled with the first case (321), an ear tip (332) detachably coupled to the housing (323), a charging terminal (331), a speaker (333), and / or a wearing detection sensor (334). In the following description, the first external electronic device (300) and the second external electronic device (320) are implemented substantially identically, and the configuration of the first external electronic device (300) is described as a representative example, and a detailed description of the configuration of the second external electronic device (320) is omitted.

[0074] According to one embodiment, the housing (303) of the first external electronic device (300) may be implemented in a shape that is at least partially wearable on a user's ear. For example, at least a portion of the first case (301) may be in physical contact with a part of the human body (e.g., an ear) on which the first external electronic device (300) is worn. For example, the second case (302) may be exposed to the external environment when the first external electronic device (300) is worn on a part of the human body. For example, the ear tip (312) may be formed of an elastic material (e.g., rubber or silicone) of a size that is at least partially insertable into the user's ear (e.g., external auditory canal).

[0075] According to one embodiment, the first external electronic device (300) may include a plurality of microphones positioned in a portion of the second case (302) exposed to the external environment. For example, the plurality of microphones may include a plurality of external microphones for receiving an external sound source (or audio signal) and at least one internal microphone for acquiring user voice speech information.

[0076] According to one embodiment, the first external electronic device (300) may include a wearing detection sensor (314) disposed in a portion of the first case (301) (e.g., an inner surface). For example, the wearing detection sensor (314) may be disposed in an area of ​​the first case (301) where physical contact with the human body occurs relatively frequently. For example, the first external electronic device (300) may determine whether the first external electronic device (300) is in physical contact with the human body of the user through the wearing detection sensor (314).

[0077] According to one embodiment, the first external electronic device (300) can output sound source data to the outside through a speaker (313) placed in the internal space of the ear tip (312).

[0078] According to one embodiment, the first external electronic device (300) may include a battery in the internal space of the housing (303). The first external electronic device (300) may charge the battery based on a charging terminal (311) disposed in a portion of the first case (301).

[0079] According to one embodiment, when the first external electronic device (300) and the second external electronic device (320) are in contact with (or worn) the user's body, they may transmit audio signals acquired through a plurality of microphones (e.g., external microphones) to the electronic device (101). For example, the first external electronic device (300) may transmit audio signals acquired through a plurality of microphones (e.g., external microphones) included in the first external electronic device (300) to the electronic device (101). For example, the audio signal transmitted to the electronic device (101) may include an audio signal beamformed based on a plurality of audio signals collected through a plurality of microphones included in the first external electronic device (300). For example, the audio signal transmitted to the electronic device (101) may include a plurality of audio signals collected through a plurality of microphones included in the first external electronic device (300).

[0080] For example, the second external electronic device (320) may transmit an audio signal acquired through a plurality of microphones (e.g., external microphones) included in the second external electronic device (320) to the electronic device (101). For example, the audio signal transmitted to the electronic device (101) may include an audio signal beamformed based on a plurality of audio signals collected through a plurality of microphones included in the second external electronic device (320). For example, the audio signal transmitted to the electronic device (101) may include a plurality of audio signals collected through a plurality of microphones included in the second external electronic device (320).

[0081] According to one embodiment, the electronic device (101) can detect voice data based on audio signals received from a plurality of external electronic devices (300 and 320). For example, if it is determined that the plurality of external electronic devices (300 and 320) are in contact with (or worn) the user's body, the electronic device (101) can detect voice data by performing beamforming based on audio signals received from the plurality of external electronic devices (300 and 320). For example, the beamforming can be performed based on a beamforming variable determined based on a beamforming result performed at a previous point in time (or a beamforming result of a previous frame). For example, the beamforming performed at a previous point in time can include beamforming for a frame received immediately before performing the current beamforming.

[0082] According to one embodiment, the electronic device (101) can determine whether distortion has occurred in the beamforming result. For example, if the electronic device (101) performs packet loss concealment (PLC) on an audio signal received from at least one of the first external electronic device (300) and the second external electronic device (320), the electronic device (101) can determine that distortion has occurred in the beamforming result due to packet loss. For example, if the electronic device (101) does not perform PLC on the audio signals received from the first external electronic device (300) and the second external electronic device (320), the electronic device (101) can determine that distortion has not occurred in the beamforming result. For example, if the power of an audio signal generated through beamforming is greater than the power of audio signals before beamforming, the electronic device (101) can determine that distortion has occurred in the beamforming result due to packet loss. For example, the electronic device (101) can determine that no distortion has occurred in the beam forming result if the power of the audio signal generated through beam forming is less than or equal to the power of the audio signals before beam forming.

[0083] According to one embodiment, if it is determined that there is no distortion in the beamforming result, the electronic device (101) may generate (or update) a beamforming variable based on audio signals received from the first external electronic device (300) and the second external electronic device (320). For example, the beamforming variable may be used for beamforming performed at the next time point (or beamforming performed in the next frame). For example, if it is determined that there is speech in the audio signals received from the first external electronic device (300) and the second external electronic device (320), the electronic device (101) may generate (or update) a beamforming variable based on a covariance vector for speech updated based on the audio signals received from the first external electronic device (300) and the second external electronic device (320). For example, the covariance vector for voice may include an average value for voice components included in audio signals received from the first external electronic device (300) and the second external electronic device (320) during a specified time interval. For example, when the electronic device (101) determines that voice does not exist in the audio signals received from the first external electronic device (300) and the second external electronic device (320), the electronic device (101) may generate (or update) beamforming variables based on the covariance vector for noise updated based on the audio signals received from the first external electronic device (300) and the second external electronic device (320). For example, the covariance vector for noise may include an average value for noise components included in the audio signals received from the first external electronic device (300) and the second external electronic device (320) during a specified time interval.

[0084] According to one embodiment, the electronic device (101) may limit the generation (or updating) of beamforming variables when it is determined that distortion has occurred in the beamforming result. For example, the electronic device (101) may limit the use of audio signals that are determined to have distortion in generating (or updating) at least one of a covariance vector for speech or a covariance vector for noise used in generating (or updating) the beamforming variables.

[0085] According to one embodiment, the electronic device (101) may perform a voice-based function using voice data detected through beamforming of audio signals received from a plurality of external electronic devices (300 and 320). For example, when the electronic device (101) performs a call function with another electronic device, the electronic device (101) may transmit voice data to the other electronic device with which a call link is established. For example, when the electronic device (101) performs a voice recording function, the electronic device (101) may store (or record) voice data. For example, when the electronic device (101) performs a voice recognition function, the electronic device (101) may perform a function corresponding to the voice data.

[0086] FIG. 4 is a block diagram of an electronic device and an external electronic device for obtaining an audio signal according to one embodiment. FIG. 5 is a block diagram of an audio processing circuit of an external electronic device for obtaining an audio signal according to one embodiment. FIG. 6 is a block diagram of an audio processing circuit of an electronic device for obtaining an audio signal according to one embodiment. For example, the electronic device (101) of FIG. 4 may be at least partially similar to the electronic device (101) of FIG. 1 or FIG. 2, or may include other embodiments of the electronic device (101). For example, the external electronic device (420) of FIG. 4 may be at least partially similar to the electronic device (102, 104) of FIG. 1, or the first external electronic device (300) or the second external electronic device (320) of FIG. 3, or may include other embodiments of the electronic device (102, 104), the first external electronic device (300), or the second external electronic device (320).

[0087] According to one embodiment referring to FIGS. 4, 5, and 6, the external electronic device (420) may include at least one of a processor (400), an audio processing circuit (402), a communication circuit (404), or a memory (406). For example, the processor (400) may be substantially the same as the processor (120) of FIG. 1 or may be included in the processor (120). The audio processing circuit (402) may be substantially the same as the audio module (170) of FIG. 1 or 2 or may be included in the audio module (170). The communication circuit (404) may be substantially the same as the wireless communication module (192) of FIG. 1 or may be included in the wireless communication module (192). The memory (406) may be substantially the same as the memory (130) of FIG. 1 or may be included in the memory (130). For example, the processor (400) may include an application processor or a communication processor. For example, the processor (400) may be operatively, functionally, and / or electrically connected to at least one of the audio processing circuit (402), the communication circuit (404), or the memory (406). For example, the processor (400) may include at least one processor that includes a processing circuit. For example, the processor (400) may include various processing circuits and / or multiple processors. For example, the processor (400) may include at least one processor that includes various processing circuits. One or more processors may be configured to perform various functions individually and / or collectively in a distributed manner.When "processor," "at least one processor," and "one or more processors" are used in one embodiment of this document as being configured to perform multiple functions, this includes, but is not limited to, situations where one processor performs some of the recited functions, another processor performs other of the recited functions, and situations where a single processor can perform all of the recited functions. Furthermore, the at least one processor may comprise a combination of processors that perform various functions in a distributed manner. At least one processor may execute program instructions to perform various functions.

[0088] According to one embodiment, the processor (400) can determine whether the external electronic device (420) is in contact with (or worn on) a part of the user's body. For example, the processor (400) can determine whether the external electronic device (420) is in contact with (or worn on) a part of the user's body through the wearing detection sensor (314) of FIG. 3.

[0089] According to one embodiment, when the processor (400) determines that the external electronic device (420) is in contact with (or worn on) a part of the user's body, the processor (400) may control the audio processing circuit (402) to acquire an external audio signal. For example, when the processor (400) determines that the external electronic device (420) is in contact with (or worn on) a part of the user's body, the processor (400) may control the audio processing circuit (402) to switch a plurality of microphones included in the external electronic device (420) to an active state.

[0090] According to one embodiment, the audio processing circuit (402) can collect external audio signals via a plurality of microphones. For example, the audio processing circuit (402) can collect external audio signals via a plurality of external microphones and / or at least one internal microphone included in the external electronic device (420).

[0091] According to one embodiment, the audio processing circuit (402) can remove echo components from audio signals collected through a plurality of microphones. For example, the audio processing circuit (402) (e.g., the signal conversion module (500) of FIG. 5) can convert analog audio signals collected through a plurality of microphones into digital audio signals. For example, the audio processing circuit (402) (e.g., the AEC (502) of FIG. 5) can remove echo components (e.g., liner echo) included in digital audio signals acquired through at least one internal microphone and a plurality of external microphones. For example, the echo components can be removed through an AEC (acoustic echo cancellation) technique.

[0092] According to one embodiment, the audio processing circuit (402) can determine whether the user's voice is present. For example, the audio processing circuit (402) (e.g., the VAD (506) of FIG. 5) can determine whether the user's voice is present through at least one internal microphone or an acceleration sensor of an external electronic device (420). For example, the acceleration sensor may be placed inside the housing (303). For example, a section (e.g., a time section) in which the user's voice is present may be referred to as a voice utterance section. For example, a section (e.g., a time section) in which the user's voice is not present may be referred to as a voice non-utterance section.

[0093] According to one embodiment, the audio processing circuit (402) can detect an audio signal through beamforming based on audio signals collected through a plurality of microphones. For example, the audio processing circuit (402) (e.g., the beamforming module (504) of FIG. 5) can detect an audio signal by performing beamforming based on beamforming variables and audio signals collected through a plurality of external microphones. For example, the beamforming can be performed based on a data-based beamforming scheme or a generalized sidelobe canceller (GSC)-based beamforming scheme.

[0094] For example, the audio processing circuit (402) (e.g., the beamforming module (504) of FIG. 5) may update beamforming variables based on the presence of speech identified through the VAD (506). For example, if it is determined that speech is present, the beamforming variables may be updated based on a covariance vector for speech. For example, if it is determined that speech is not present, the beamforming variables may be updated based on a covariance vector for noise.

[0095] According to one embodiment, the audio processing circuit (402) can selectively compensate for at least a portion of the audio signal based on the quality of the audio signal detected via beamforming. For example, the audio processing circuit (402 (e.g., the signal correction module (508) of FIG. 5) can check the quality of the audio signal detected through beamforming by frequency band (or frequency). For example, the quality of the audio signal may include a signal to noise ratio (SNR). For example, the audio processing circuit (402 (e.g., the signal correction module (508) of FIG. 5)) can apply a weight corresponding to the quality of the audio signal by frequency band (or frequency) of the audio signal. For example, a part of the audio signal corresponding to a frequency band (or frequency) where the quality of the audio signal is higher than a specified reference value may not have a weight applied. For example, the audio processing circuit (402 (e.g., the signal correction module (508) of FIG. 5)) can replace (or replace) a part of the audio signal corresponding to a frequency band (or frequency) where the quality of the audio signal is lower than a specified reference value with an audio signal collected through at least one internal microphone or acceleration sensor. For example, the quality of the audio signal is lower than a specified reference value. A portion of the audio signal corresponding to a frequency band (or frequencies) higher than the value may be maintained in the beamforming result.

[0096] According to one embodiment, the audio processing circuit (402) may control the communication circuit (404) to transmit an audio signal and information related to the audio signal, which are selectively corrected based on the quality of the audio signal, to the electronic device (101). For example, the audio processing circuit (402) (e.g., the encoder (510) of FIG. 5) may encode the audio signal, which is selectively corrected based on the quality of the audio signal, based on a designated codec (e.g., a speech codec). For example, the information related to the audio signal may be included in a spare bit space of the encoded audio signal. For example, the information related to the audio signal may include at least one of information related to the presence or absence of a voice or information related to a corrected frequency band.

[0097] According to one embodiment, the communication circuit (404) can perform wireless communication between an external electronic device (420) and the electronic device (101). For example, the wireless communication can be performed based on Bluetooth, BLE (Bluetooth low energy), or wireless LAN.

[0098] According to one embodiment, the memory (406) may store various data used by at least one component of the external electronic device (420) (e.g., the processor (400), the audio processing circuit (402), and / or the communication circuit (404)). For example, the memory (406) may store various instructions that may be executed individually or collectively by the processor (400).

[0099] According to one embodiment, the external electronic device (420) may perform at least some of the operations performed by the audio processing circuit (402) on the processor (400).

[0100] According to one embodiment, the external electronic device (420) may transmit audio signals acquired through a plurality of external microphones to the electronic device (101). For example, the external electronic device (420) may remove echo components from audio signals acquired through a plurality of external microphones and transmit the removed audio signals to the electronic device (101).

[0101] According to one embodiment, the electronic device (101) may include at least one of a processor (410), an audio processing circuit (412), a communication circuit (414), or a memory (416). For example, the processor (410) may be substantially the same as the processor (120) of FIG. 1 or may be included in the processor (120). The audio processing circuit (412) may be substantially the same as the audio module (170) of FIG. 1 or 2 or may be included in the audio module (170). The communication circuit (414) may be substantially the same as the wireless communication module (192) of FIG. 1 or may be included in the wireless communication module (192). The memory (416) may be substantially the same as the memory (130) of FIG. 1 or may be included in the memory (130). For example, the processor (410) may include an application processor or a communication processor. For example, the processor (410) may be operatively, functionally, and / or electrically connected to at least one of the audio processing circuit (412), the communication circuit (414), or the memory (416). For example, the processor (410) may include at least one processor that includes a processing circuit. For example, the processor (410) may include at least one processor that includes a processing circuit. For example, the processor (410) may include various processing circuits and / or multiple processors. For example, the processor (410) may include at least one processor that includes various processing circuits. One or more processors may be configured to perform various functions individually and / or collectively in a distributed manner.When "processor," "at least one processor," and "one or more processors" are used in one embodiment of this document as being configured to perform multiple functions, this includes, but is not limited to, situations where one processor performs some of the recited functions, another processor performs other of the recited functions, and situations where a single processor can perform all of the recited functions. Furthermore, the at least one processor may comprise a combination of processors that perform various functions in a distributed manner. At least one processor may execute program instructions to perform various functions.

[0102] According to one embodiment, the processor (410) can determine whether an external electronic device (420) (e.g., the first external electronic device (300) and the second external electronic device (320) of FIG. 3) wirelessly connected to the electronic device (101) is in contact with (or worn on) a part of the user's body. For example, the processor (410) can determine whether the first external electronic device (300) and the second external electronic device (320) are in contact with (or worn on) a part of the user's body based on control information provided from the first external electronic device (300) and the second external electronic device (320).

[0103] According to one embodiment, the processor (410) may determine a voice detection method based on whether an external electronic device (420) (e.g., the first external electronic device (300) and the second external electronic device (320) of FIG. 3) wirelessly connected to the electronic device (101) is worn. For example, if the processor (410) determines that the external electronic device (420) (e.g., the first external electronic device (300) and the second external electronic device (320) of FIG. 3) wirelessly connected to the electronic device (101) is in contact with (or worn on) a part of the user's body, the processor (410) may control the audio processing circuit (412) to detect voice data based on a beamforming method. For example, if the processor (410) determines that an external electronic device (420) (e.g., the first external electronic device (300) and / or the second external electronic device (320) of FIG. 3) wirelessly connected to the electronic device (101) is not in contact with (or worn by) a part of the user's body, the processor (410) may control the audio processing circuit (412) to detect voice data based on a noise cancellation method.

[0104] According to one embodiment, the audio processing circuit (412) can decode audio signals received from an external electronic device (420) (e.g., the first external electronic device (300) and the second external electronic device (320) of FIG. 3) wirelessly connected to the electronic device (101). For example, the audio processing circuit (412) (e.g., the decoder (600) of FIG. 6) can decode an encoded audio signal received through the first external electronic device (300) and an encoded audio signal received from the second external electronic device (320). For example, the audio processing circuit (412) (e.g., the decoder (600) of FIG. 6) can synchronize an audio signal received through the first external electronic device (300) and an audio signal received from the second external electronic device (320). For example, audio signals received from external electronic devices (420) (e.g., the first external electronic device (300) and the second external electronic device (320) of FIG. 3) may include audio signals beamformed in each external electronic device (e.g., the first external electronic device (300) or the second external electronic device (320) of FIG. 3). For example, audio signals received from external electronic devices (420) (e.g., the first external electronic device (300) and the second external electronic device (320) of FIG. 3) may include audio signals collected through a plurality of microphones included in each external electronic device (e.g., the first external electronic device (300) or the second external electronic device (320) of FIG. 3).

[0105] For example, if an audio signal determined to have a packet error exists among the audio signal received through the first external electronic device (300) and the audio signal received from the second external electronic device (320), the audio processing circuit (412) (e.g., the decoder (600) of FIG. 6) may perform PLC (packet loss concealment) on the audio signal determined to have a packet error. For example, the PLC may include a signal processing method for reducing sound quality deterioration due to packet loss of the audio signal determined to have a packet error.

[0106] According to one embodiment, the audio processing circuit (412) can determine whether the user's voice exists. For example, the audio processing circuit (412) (e.g., VAD (602) of FIG. 6) can determine whether the user's voice exists in the audio signals received from the external electronic devices (300 and 320) based on information related to the presence of voice obtained from the first external electronic device (300) and the second external electronic device (320). For example, if the audio processing circuit (412) (e.g., VAD (602) of FIG. 6) obtains information indicating the presence of voice from at least one of the first external electronic device (300) or the second external electronic device (320), it can determine that the user's voice exists in the audio signals received from the external electronic devices (300 and 320). When the audio processing circuit (412) (e.g., VAD (602) of FIG. 6) obtains information indicating that there is no voice from the first external electronic device (300) and the second external electronic device (320) and / or information indicating that the presence of a voice cannot be determined, the audio processing circuit (412) may determine that the user's voice does not exist in the audio signals received from the external electronic devices (300 and 320).

[0107] According to one embodiment, when the audio processing circuit (412) determines to detect voice data based on a noise removal method in the processor (410), the audio processing circuit (412) may remove noise from an audio signal received from an external electronic device (420) determined to be worn on the user's body (e.g., the first external electronic device (300) or the second external electronic device (320) of FIG. 3). For example, the audio processing circuit (412) (e.g., the noise removal module (610) of FIG. 6) may remove noise from an audio signal received from the first external electronic device (300) or the second external electronic device (320) determined to be worn on the user's body. For example, noise removal may be performed using a noise removal method based on a deep neural network (DNN).

[0108] According to one embodiment, when the audio processing circuit (412) determines to detect voice data based on a beamforming method in the processor (410), the audio processing circuit (412) may check a frequency band (or frequency) capable of beamforming in audio signals received from an external electronic device (420) (e.g., the first external electronic device (300) and the second external electronic device (320) of FIG. 3). For example, the audio processing circuit (412) (e.g., the beam control module (604) of FIG. 6) may check whether a corrected frequency band (or frequency) exists in the audio signals received from the first external electronic device (300) and the second external electronic device (320) based on information related to the corrected frequency bands received from the first external electronic device (300) and the second external electronic device (320). For example, the audio processing circuit (412) (e.g., the beam control module (604) of FIG. 6) may determine a frequency band (or frequency) that is not corrected in the first external electronic device (300) and the second external electronic device (320) as a frequency band (or frequency) in which beamforming is possible. For example, the audio processing circuit (412) (e.g., the beam control module (604) of FIG. 6) may determine a frequency band (or frequency) that is corrected in at least one of the first external electronic device (300) or the second external electronic device (320) as a frequency band in which beamforming is restricted.

[0109] According to one embodiment, the audio processing circuit (412) may select any one of the audio signals received from the external electronic devices (420) (e.g., the first external electronic device (300) and the second external electronic device (320) of FIG. 3) corresponding to the frequency band (or frequency) in which beamforming is limited, in the case of a frequency band (or frequency) in which beamforming is limited. For example, the audio processing circuit (412) (e.g., the noise removal module (610) of FIG. 6) may select an audio signal having a low noise level among the audio signals received from the first external electronic device (300) and the second external electronic device (320) corresponding to the frequency band (or frequency) in which beamforming is limited. According to one embodiment, the audio processing circuit (412) (e.g., the noise removal module (610) of FIG. 6) may remove noise from the selected audio signal based on the noise level. For example, noise removal may be performed using a noise removal method based on a DNN.

[0110] According to one embodiment, the audio processing circuit (412) may perform beamforming based on audio signals received from external electronic devices (420) (e.g., the first external electronic device (300) and the second external electronic device (320) of FIG. 3) corresponding to a frequency band (or frequency) in which beamforming is possible. For example, the audio processing circuit (412) (e.g., the beamforming module (606) of FIG. 6) may detect an audio signal by performing beamforming based on a beamforming variable and audio signals received from the external electronic devices (300 and 320). For example, the beamforming may be performed based on a data-based beamforming method or a GSC (generalized sidelobe canceller)-based beamforming method. For example, the beamforming may be performed based on a beamforming variable determined based on a beamforming result performed at a previous point in time (or a beamforming result of a previous frame). For example, beamforming performed at a previous point in time may include beamforming for frames received immediately prior to performing the current beamforming.

[0111] According to one embodiment, the audio processing circuit (412) can determine whether distortion occurs in the audio signal due to beamforming. For example, the audio processing circuit (412) (e.g., the error detection module (608) of FIG. 6) can analyze the input and output to the beamforming module (606) by frequency band (or frequency) of the audio signal. For example, beamforming in the beamforming module (606) is a processing method for increasing (or improving) the size of the voice component relative to the size of the noise component in the audio signal, and the power of the output signal of the beamforming module (606) may be lower than the power of the input signal of the beamforming module (606). The audio processing circuit (412) (e.g., the error detection module (608) of FIG. 6) may determine that distortion has occurred in the audio signal due to beam forming if there is a frequency band (or frequency) in which the power of the audio signal included in the output of the beam forming module (606) has increased by more than a specified reference value compared to the power of the audio signal input to the beam forming module (606). For example, the audio processing circuit (412) (e.g., the error detection module (608) of FIG. 6) may determine that distortion has not occurred in the audio signal due to beam forming if there is no frequency band (or frequency) in which the power of the audio signal included in the output of the beam forming module (606) has increased by more than a specified reference value compared to the power of the audio signal input to the beam forming module (606).

[0112] For example, the audio processing circuit (412) (e.g., the error detection module (608) or the beam control module (604) of FIG. 6) can determine whether distortion has occurred in the audio signal due to beam forming based on whether PLC has been performed on the audio signal received from at least one of the first external electronic device (300) or the second external electronic device (320) corresponding to a frequency band in which beam forming is possible. For example, if the audio processing circuit (412) (e.g., the error detection module (608) or the beam control module (604) of FIG. 6) has performed PLC on the audio signal received from at least one of the first external electronic device (300) or the second external electronic device (320), it can determine that distortion has occurred in the audio signal due to beam forming. For example, if the audio processing circuit (412) (e.g., the error detection module (608) or the beam control module (604) of FIG. 6) does not perform PLC on the audio signal received from the first external electronic device (300) and the second external electronic device (320), it may be determined that no distortion has occurred in the audio signal due to beam forming.

[0113] According to one embodiment, when the audio processing circuit (412) (e.g., the beam control module (604) of FIG. 6) determines that an audio signal corresponding to a frequency band capable of beamforming is not distorted, the audio processing circuit (412) may generate (or update) a beamforming variable based on an audio signal received from an external electronic device (420) (e.g., the first external electronic device (300) and the second external electronic device (320)). For example, the beamforming variable may be used for beamforming performed at the next time point (or beamforming performed in the next frame).

[0114] For example, if the audio processing circuit (412) (e.g., the beam control module (604) of FIG. 6) determines that speech is present in audio signals received from an external electronic device (420) (e.g., the first external electronic device (300) and the second external electronic device (320)), the audio processing circuit (412) may generate (or update) beamforming variables based on a covariance vector for speech updated based on the audio signals received from the external electronic device (420) (e.g., the first external electronic device (300) and the second external electronic device (320)). For example, the covariance vector for speech may include an average value for speech components included in the audio signals received from the first external electronic device (300) and the second external electronic device (320) during a specified time interval. For example, if the audio processing circuit (412) (e.g., the beam control module (604) of FIG. 6) determines that there is no voice in the audio signals received from the external electronic device (420) (e.g., the first external electronic device (300) and the second external electronic device (320)), the audio processing circuit (412) may generate (or update) a beamforming variable based on a covariance vector for noise updated based on the audio signals received from the external electronic device (420) (e.g., the first external electronic device (300) and the second external electronic device (320)). For example, the covariance vector for noise may include an average value for noise components included in the audio signals received from the first external electronic device (300) and the second external electronic device (320) during a specified time interval.

[0115] According to one embodiment, the audio processing circuit (412) (e.g., the beam control module (604) of FIG. 6) may limit the generation (or update) of beamforming variables when it determines that an audio signal corresponding to a frequency band in which beamforming is possible is distorted. For example, the audio processing circuit (412) (e.g., the beam control module (604) of FIG. 6) may limit the use of audio signals that are determined to be distorted in generating (or updating) at least one of a covariance vector for speech or a covariance vector for noise used in generating (or updating) beamforming variables.

[0116] According to one embodiment, the audio processing circuit (412) can remove noise from an audio signal detected through beamforming. For example, the audio processing circuit (412) (e.g., the noise removal module (610) of FIG. 6) can remove noise from a beamformed audio signal output from the beamforming module (606). For example, noise removal can be performed using a noise removal method based on a deep neural network (DNN). For example, the audio processing circuit (412) (e.g., the noise removal module (610) of FIG. 6) can remove an echo component (e.g., a non-linear echo) from a beamformed audio signal output from the beamforming module (606).

[0117] In one embodiment, the audio processing circuit (412) can detect speech from a noise-removed audio signal. For example, the audio processing circuit (412) can detect the user's speech data from a noise-removed audio signal included in a section where the user's speech is determined to be present (e.g., a speech utterance section).

[0118] According to one embodiment, the processor (410) may perform a voice-based function using voice data detected based on audio signals received from external electronic devices (420) (e.g., the first external electronic device (300) and the second external electronic device (320)). For example, when the electronic device (101) performs a call function with another electronic device, the processor (410) may control the communication circuit (414) to transmit voice data to another electronic device with which a call link is established. For example, when the electronic device (101) performs a voice recording function, the processor (410) may store (or record) voice data. For example, when the electronic device (101) performs a voice recognition function, the processor (410) may perform a function corresponding to voice data.

[0119] According to one embodiment, the communication circuit (414) can perform wireless communication between the electronic device (101) and an external electronic device (420) and wireless communication between the electronic device (101) and another electronic device. For example, the wireless communication with the external electronic device (420) can be performed based on Bluetooth, BLE (Bluetooth low energy), or wireless LAN. For example, the wireless communication with another electronic device can be performed based on a 4G communication method (e.g., long term evolution (LTE)) or a 5G communication method (e.g., new radio (NR)).

[0120] According to one embodiment, the memory (416) may store various data used by at least one component of the electronic device (101) (e.g., the processor (410), the audio processing circuit (412), and / or the communication circuit (414)). For example, the memory (416) may store various instructions that may be executed individually or collectively by the processor (410).

[0121] According to one embodiment, the electronic device (101) may perform at least some of the operations performed in the audio processing circuit (412) in the processor (410).

[0122] According to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1, FIG. 2, FIG. 3 or FIG. 4) may include a communication circuit (e.g., the wireless communication module (192) of FIG. 1 or the communication circuit (414) of FIG. 4), at least one processor including a processing circuit (e.g., the processor (120) of FIG. 1, the audio module (170), the processor (410) or the audio processing circuit (412) of FIG. 3), and a memory (e.g., the memory (130) of FIG. 1 or the memory (416) of FIG. 4)) operatively connected to the processor. According to one embodiment, the memory may store instructions that, when individually or collectively executed by the at least one processor, cause the electronic device to receive audio signals from external electronic devices (e.g., the first external electronic device (300) and the second external electronic device (320) of FIG. 3 or the external electronic device (420) of FIG. 4). According to one embodiment, the memory may store instructions that, when individually or collectively executed by at least one processor, cause the electronic device to determine a frequency band in which beamforming is possible in audio signals. According to one embodiment, the memory may store instructions that, when individually or collectively executed by at least one processor, cause the electronic device to perform beamforming on an audio signal in a frequency band in which beamforming is possible. According to one embodiment, the memory may store instructions that, when individually or collectively executed by at least one processor, cause the electronic device to determine whether distortion has occurred in an audio signal due to beamforming. According to one embodiment, the memory may store instructions that, when individually or collectively executed by at least one processor, cause the electronic device to update a beamforming variable based on a speech component or a noise component included in the audio signals when it is determined that distortion has not occurred in the audio signal due to beamforming.According to one embodiment, the memory may store instructions that, when executed individually or collectively by at least one processor, cause the electronic device to limit updates of beamforming variables when it determines that distortion has occurred in an audio signal due to beamforming.

[0123] In one embodiment, the beamforming variables can be used for beamforming of an audio signal performed in the next time interval (or next frame).

[0124] According to one embodiment, the memory may store instructions that, when individually or collectively executed by at least one processor, cause the electronic device to determine that distortion has occurred in the audio signal due to beamforming if the electronic device has performed a packet loss concealment (PLC) operation on the audio signal in a frequency band capable of beamforming. According to one embodiment, the memory may store instructions that, when individually or collectively executed by at least one processor, cause the electronic device to determine that distortion has not occurred in the audio signal due to beamforming if the electronic device has not performed a PLC operation on the audio signal in a frequency band capable of beamforming.

[0125] According to one embodiment, the memory may store instructions that, when individually or collectively executed by at least one processor, cause the electronic device to determine that distortion has occurred in the audio signal due to beamforming if the power of the audio signal is determined to have increased based on the beamforming result. According to one embodiment, the memory may store instructions that, when individually or collectively executed by at least one processor, cause the electronic device to determine that distortion has not occurred in the audio signal due to beamforming if the power of the audio signal is determined to have decreased based on the beamforming result.

[0126] According to one embodiment, the memory stores instructions that, when executed individually or collectively by at least one processor, cause the electronic device to determine a frequency band for which beamforming is possible in audio signals received from external electronic devices when the electronic device determines that the external electronic devices are worn on a part of the user's body.

[0127] According to one embodiment, the memory may store instructions that, when individually or collectively executed by at least one processor, cause the electronic device to determine whether an updated frequency band exists based on another audio signal in beamformed audio signals received from each external electronic device based on control signals received from the external electronic devices. According to one embodiment, the memory may store instructions that, when individually or collectively executed by at least one processor, cause the electronic device to determine an updated frequency band based on another audio signal as a frequency band in which beamforming is restricted. According to one embodiment, the memory may store instructions that, when individually or collectively executed by at least one processor, cause the electronic device to determine an unupdated frequency band based on another audio signal as a frequency band in which beamforming is possible.

[0128] According to one embodiment, the memory may store instructions that, when executed individually or collectively by at least one processor, cause the electronic device to remove noise from a beamforming result (e.g., an audio signal detected through beamforming) for a frequency band in which beamforming is possible and an audio signal in a frequency band in which beamforming is limited. According to one embodiment, the memory may store instructions that, when executed individually or collectively by at least one processor, cause the electronic device to remove an echo component from an audio signal from which noise has been removed.

[0129] According to one embodiment, the memory may store instructions that, when executed individually or collectively by at least one processor, cause the electronic device to determine whether a designated section for beamforming is a speech generation section based on a control signal received from external electronic devices if the electronic device determines that no distortion has occurred in the audio signal due to beamforming. According to one embodiment, the memory may store instructions that, when executed individually or collectively by at least one processor, cause the electronic device to update a variable related to beamforming based on a speech component included in audio signals if the designated section for beamforming is a speech generation section. According to one embodiment, the memory may store instructions that, when executed individually or collectively by at least one processor, cause the electronic device to update a variable related to beamforming based on a noise component included in audio signals if the designated section for beamforming is not a speech generation section.

[0130] In one embodiment, the external electronic devices may include a pair of wearable devices (e.g., earphones).

[0131] FIG. 7 is a flowchart (700) for acquiring an audio signal from an external electronic device according to one embodiment. In the following embodiments, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel. For example, the external electronic device of FIG. 7 may be the external electronic device (300, 320, or 420) of FIG. 1, FIG. 2, FIG. 3, or FIG. 4.

[0132] According to one embodiment referring to FIG. 7, an external electronic device (e.g., the first external electronic device (300) and / or the second external electronic device (320) of FIG. 3) may collect an external audio signal through a plurality of microphones of the external electronic device in operation 701. For example, when the processor (400) determines that the first external electronic device (300) (or the second external electronic device (320)) is in contact with (or worn on) a part of the user's body, the processor (400) may control the audio processing circuit (402) to acquire an external audio signal. The audio processing circuit (402) may collect the external audio signal through a plurality of external microphones and / or at least one internal microphone included in the first external electronic device (300) (or the second external electronic device (320)).

[0133] According to one embodiment, an external electronic device (e.g., the first external electronic device (300) and / or the second external electronic device (320) of FIG. 3) may, in operation 703, remove an echo component from an audio signal collected through a plurality of microphones. For example, an audio processing circuit (402) (e.g., an AEC (502) of FIG. 5) may remove an echo component (e.g., a liner echo) included in audio signals acquired through at least one internal microphone and a plurality of external microphones. For example, the echo component may be removed through an AEC (acoustic echo cancellation) technique.

[0134] According to one embodiment, an external electronic device (e.g., the first external electronic device (300) and / or the second external electronic device (320) of FIG. 3) may perform beamforming based on audio signals in operation 705. For example, the audio processing circuit (402) (e.g., the beamforming module (504) of FIG. 5) may detect an audio signal by performing beamforming based on a beamforming variable and audio signals collected through a plurality of external microphones. As an example, the beamforming may be performed based on a data-based beamforming scheme or a generalized sidelobe canceller (GSC)-based beamforming scheme. As an example, the beamforming variable may include a beamforming variable generated (or updated) based on a beamforming result performed at a previous point in time.

[0135] For example, the audio processing circuit (402) (e.g., the VAD (506) of FIG. 5) can determine whether a user's voice is present through at least one internal microphone or an acceleration sensor of the external electronic device (300). For example, the audio processing circuit (402) (e.g., the beam forming module (504) of FIG. 5) can generate (or update) beam forming variables based on the presence of a voice determined through the VAD (506). For example, the beam forming variables can be generated (or updated) based on a covariance vector for a voice when it is determined that a voice is present. For example, the beam forming variables can be generated (or updated) based on a covariance vector for noise when it is determined that a voice is not present.

[0136] According to one embodiment, an external electronic device (e.g., the first external electronic device (300) and / or the second external electronic device (320) of FIG. 3) may selectively update (or correct) at least a portion of an audio signal based on the quality of the audio signal detected via beamforming, at operation 707. For example, the audio processing circuit (402 (e.g., the signal correction module (508) of FIG. 5)) can check the quality of the audio signal detected through beamforming for each frequency band. For example, the quality of the audio signal may include a signal to noise ratio (SNR). For example, the audio processing circuit (402 (e.g., the signal correction module (508) of FIG. 5)) can apply a weight corresponding to the quality of the audio signal for each frequency band of the audio signal. For example, a part of the audio signal corresponding to a frequency band where the quality of the audio signal is higher than a specified reference value may not have a weight applied. For example, the audio processing circuit (402 (e.g., the signal correction module (508) of FIG. 5)) can replace a part of the audio signal corresponding to a frequency band where the quality of the audio signal is lower than a specified reference value with an audio signal collected through at least one internal microphone or acceleration sensor. For example, a part of the audio signal corresponding to a frequency band where the quality of the audio signal is higher than a specified reference value may be maintained as a beamforming result.

[0137] According to one embodiment, an external electronic device (e.g., the first external electronic device (300) and / or the second external electronic device (320) of FIG. 3) may, in operation 709, transmit (or output) an audio signal and information related to the audio signal, which are selectively corrected based on the quality of the audio signal, to the electronic device (101). For example, the audio processing circuit (402) (e.g., the encoder (510) of FIG. 5) may encode the audio signal, which is selectively corrected based on the quality of the audio signal, based on a designated codec (e.g., a speech codec). For example, the information related to the audio signal may be included in a spare bit space of the encoded audio signal. For example, the information related to the audio signal may include at least one of information related to the presence or absence of a voice or information related to a corrected frequency band.

[0138] FIG. 8 is a flowchart (800) for acquiring an audio signal in an electronic device according to one embodiment. In the following embodiments, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel. For example, the electronic device of FIG. 8 may be the electronic device (101) of FIG. 1, FIG. 2, FIG. 3, or FIG. 4.

[0139] According to one embodiment referring to FIG. 8, an electronic device (e.g., electronic device (101)) may receive audio signals from external electronic devices (e.g., first external electronic device (300) and second external electronic device (320) of FIG. 3 or external electronic device (420) of FIG. 4) wirelessly connected to the electronic device (101) in operation 801. For example, an audio processing circuit (412) (e.g., decoder (600) of FIG. 6) may decode an encoded audio signal received through the first external electronic device (300) and an encoded audio signal received from the second external electronic device (320). For example, the audio processing circuit (412) (e.g., decoder (600) of FIG. 6) may synchronize an audio signal received through the first external electronic device (300) and an audio signal received from the second external electronic device (320). For example, audio signals received from external electronic devices (e.g., the first external electronic device (300) and the second external electronic device (320) of FIG. 3) may include audio signals beamformed from each external electronic device. For example, audio signals received from external electronic devices (e.g., the first external electronic device (300) and the second external electronic device (320) of FIG. 3) may include audio signals collected through a plurality of microphones included in each external electronic device.

[0140] For example, if an audio signal determined to have a packet error exists among the audio signal received through the first external electronic device (300) and the audio signal received from the second external electronic device (320), the audio processing circuit (412) (e.g., the decoder (600) of FIG. 6) may perform PLC (packet loss concealment) on the audio signal determined to have a packet error. For example, the PLC may include a signal processing method for reducing sound quality deterioration due to packet loss of the audio signal determined to have a packet error.

[0141] According to one embodiment, the electronic device (101) may, in operation 803, identify a frequency band in which beamforming is possible from audio signals received from external electronic devices (e.g., the first external electronic device (300) and the second external electronic device (320) of FIG. 3). For example, the processor (410) may identify whether the first external electronic device (300) and the second external electronic device (320) are in contact with (or worn on) a part of the user's body based on control information provided from the first external electronic device (300) and the second external electronic device (320). For example, if the processor (410) determines that external electronic devices (e.g., the first external electronic device (300) and the second external electronic device (320) of FIG. 3) wirelessly connected to the electronic device (101) are in contact with (or worn on) a part of the user's body, the processor (410) may control the audio processing circuit (412) to detect voice data based on a beam forming method. For example, if the processor (410) determines that at least one of the external electronic devices (e.g., the first external electronic device (300) and the second external electronic device (320) of FIG. 3) wirelessly connected to the electronic device (101) is not in contact with (or worn on) a part of the user's body, the processor (410) may control the audio processing circuit (412) to detect voice data based on a noise removal method.

[0142] For example, when the audio processing circuit (412) (e.g., the beam control module (604) of FIG. 6) determines to detect voice data based on a beam forming method in the processor (410), the audio processing circuit (412) can check a frequency band in which beam forming is possible in audio signals received from external electronic devices (e.g., the first external electronic device (300) and the second external electronic device (320) of FIG. 3). For example, the audio processing circuit (412) (e.g., the beam control module (604) of FIG. 6) can check whether a corrected frequency band (or frequency) exists in the audio signals received from the first external electronic device (300) and the second external electronic device (320) based on information related to the corrected frequency bands received from the first external electronic device (300) and the second external electronic device (320). For example, the audio processing circuit (412) (e.g., the beam control module (604) of FIG. 6) may determine a frequency band (or frequency) that is not corrected in the first external electronic device (300) and the second external electronic device (320) as a frequency band (or frequency) in which beamforming is possible. For example, the audio processing circuit (412) (e.g., the beam control module (604) of FIG. 6) may determine a frequency band (or frequency) that is corrected in at least one of the first external electronic device (300) or the second external electronic device (320) as a frequency band in which beamforming is restricted.

[0143] According to one embodiment, the electronic device (101) may perform beamforming on audio signals received from external electronic devices (e.g., the first external electronic device (300) and the second external electronic device (320) of FIG. 3) corresponding to a frequency band (or frequency) capable of beamforming, in operation 805. For example, the beamforming may be performed based on a data-based beamforming method or a GSC (generalized sidelobe canceller)-based beamforming method. For example, the beamforming may be performed based on a beamforming variable determined based on a beamforming result performed at a previous point in time (or a beamforming result of a previous frame). For example, the beamforming performed at a previous point in time may include beamforming for a frame received immediately before performing the current beamforming.

[0144] According to one embodiment, the electronic device (101) may determine whether distortion occurs in the audio signal due to beam forming at operation 807. For example, if the audio processing circuit (412) (e.g., the beam control module (604) or the error detection module (608) of FIG. 6) performs PLC on the audio signal received from at least one of the first external electronic device (300) or the second external electronic device (320), the audio processing circuit (412) may determine that the audio signal is distorted due to beam forming. If the audio processing circuit (412) (e.g., the beam control module (604) or the error detection module (608) of FIG. 6) does not perform PLC on the audio signal received from the first external electronic device (300) and the second external electronic device (320), the audio processing circuit (412) may determine that the audio signal is not distorted due to beam forming.

[0145] For example, the audio processing circuit (412) (e.g., the error detection module (608) of FIG. 6) may determine that distortion has occurred in the audio signal due to beam forming if there is a frequency band (or frequency) in which the power of the audio signal included in the output of the beam forming module (606) has increased by more than a specified reference value compared to the power of the audio signal input to the beam forming module (606). For example, the audio processing circuit (412) (e.g., the error detection module (608) of FIG. 6) may determine that distortion has not occurred in the audio signal due to beam forming if there is no frequency band (or frequency) in which the power of the audio signal included in the output of the beam forming module (606) has increased by more than a specified reference value compared to the power of the audio signal input to the beam forming module (606).

[0146] According to one embodiment, if the electronic device (101) determines that distortion has occurred in the audio signal due to beamforming (e.g., 'Yes' in operation 807), in operation 809, the electronic device may maintain the beamforming variables determined at a previous point in time. For example, the audio processing circuit (412) (e.g., the beam control module (604) of FIG. 6) may restrict the use of audio signals determined to have distortion in generating (or updating) at least one of the covariance vector for speech or the covariance vector for noise used to generate (or update) the beamforming variables.

[0147] According to one embodiment, if the electronic device (101) determines that the audio signal is not distorted through beamforming (e.g., 'No' in operation 807), in operation 811, the electronic device may update (or generate) beamforming variables based on audio signals received from external electronic devices (e.g., the first external electronic device (300) and the second external electronic device (320) of FIG. 3). For example, if the audio processing circuit (412) (e.g., the beam control module (604) of FIG. 6) determines that the audio signal is not distorted through beamforming, the audio processing circuit (412) may update (or generate) beamforming variables based on audio signals received from the first external electronic device (300) and the second external electronic device (320).

[0148] For example, if the audio processing circuit (412) (e.g., the beam control module (604) of FIG. 6) determines that speech is present in the audio signals received from the first external electronic device (300) and the second external electronic device (320), the audio processing circuit (412) may update (or generate) beamforming variables based on a covariance vector for speech updated based on the audio signals received from the first external electronic device (300) and the second external electronic device (320). As an example, the covariance vector for speech may include an average value for speech components included in the audio signals received from the first external electronic device (300) and the second external electronic device (320) during a specified time interval. For example, if the audio processing circuit (412) (e.g., the beam control module (604) of FIG. 6) determines that there is no voice in the audio signals received from the first external electronic device (300) and the second external electronic device (320), the audio processing circuit (412) may update (or generate) a beamforming variable based on a covariance vector for noise updated based on the audio signals received from the first external electronic device (300) and the second external electronic device (320). For example, the covariance vector for noise may include an average value for a noise component included in the audio signals received from the first external electronic device (300) and the second external electronic device (320) during a specified time interval.

[0149] According to one embodiment, the electronic device (101) can perform beam forming at the next time (or beam forming of the next frame) using beam forming variables that are optionally updated based on the beam forming result.

[0150] According to one embodiment, the electronic device (101) can detect voice data from an audio signal detected through beamforming. For example, the audio processing circuit (412) (e.g., the noise removal module (610) of FIG. 6) can remove noise from the beamformed audio signal output from the beamforming module (606). For example, noise removal can be performed using a noise removal method based on a deep neural network (DNN). For example, the audio processing circuit (412) (e.g., the noise removal module (610) of FIG. 6) can remove an echo component (e.g., a non-linear echo) from the beamformed audio signal output from the beamforming module (606). For example, the audio processing circuit (412) can detect the user's voice data from an audio signal from which noise has been removed, which is included in a section where the user's voice is determined to exist (e.g., a voice utterance section).

[0151] According to one embodiment, when the electronic device (101) determines to detect voice data based on a noise removal method, the electronic device (101) may remove noise from an audio signal received from an external electronic device (e.g., the first external electronic device (300) or the second external electronic device (320) of FIG. 3) determined to be worn on the user's body. The electronic device (101) may detect the user's voice data from an audio signal from which noise has been removed, which is included in a section (e.g., a voice utterance section) in which the user's voice is determined to be present.

[0152] FIG. 9 is a flowchart (900) for confirming a frequency band capable of beamforming in an electronic device according to one embodiment. For example, at least a portion of FIG. 9 may include detailed operations of operations 803 and 805 of FIG. 8. In the following embodiments, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel. For example, the electronic device of FIG. 9 may be the electronic device (101) of FIG. 1, FIG. 2, FIG. 3, or FIG. 4.

[0153] According to one embodiment referring to FIG. 9, when an electronic device (e.g., electronic device (101)) receives an audio signal from external electronic devices (e.g., first external electronic device (300) and second external electronic device (320) of FIG. 3 or external electronic device (420) of FIG. 4) wirelessly connected to the electronic device (101) (e.g., operation 801 of FIG. 8), in operation 901, it can be determined whether beamforming is possible at the i-th frequency (or i-th frequency band). For example, when the audio processing circuit (412) (e.g., the beam control module (604) of FIG. 6) determines that the first external electronic device (300) and the second external electronic device (320) wirelessly connected to the electronic device (101) are in contact with (or worn on) a part of the user's body, the audio processing circuit (412) may check whether beam forming is possible at the ith frequency (or the ith frequency band) in the audio signal received from the external electronic devices (300 and 320). For example, the ith frequency (or the ith frequency band) may represent some frequencies (or some frequency bands) among the frequencies (or frequency bands) included in the audio signal of a specified time interval (e.g., frame) received by the electronic device (101) from the external electronic devices (300 and 320). For example, i is an index representing the frequencies (or frequency bands) included in the audio signal, and may include 0 and a natural number.

[0154] For example, the audio processing circuit (412) (e.g., the beam control module (604) of FIG. 6) may determine not to perform beamforming for the i-th frequency (or the i-th frequency band) when the audio signal of the i-th frequency (or the i-th frequency band) is corrected based on information related to the corrected frequency bands received from the first external electronic device (300) and the second external electronic device (320). For example, a state in which the audio signal of the i-th frequency (or the i-th frequency band) is determined to be corrected may include a state in which at least one of the audio signal of the i-th frequency (or the i-th frequency band) received from the first external electronic device (300) or the audio signal of the i-th frequency (or the i-th frequency band) received from the second external electronic device (320) is corrected.

[0155] For example, if the audio processing circuit (412) (e.g., the beam control module (604) of FIG. 6) determines that the audio signal of the ith frequency (or the ith frequency band) is not corrected based on information related to the corrected frequency bands received from the first external electronic device (300) and the second external electronic device (320), the audio processing circuit (412) may determine to perform beamforming for the ith frequency (or the ith frequency band). For example, a state in which the audio signal of the ith frequency (or the ith frequency band) is determined to be not corrected may include a state in which the audio signal of the ith frequency (or the ith frequency band) received from the first external electronic device (300) and the audio signal of the ith frequency (or the ith frequency band) received from the second external electronic device (320) are not corrected.

[0156] According to one embodiment, when the electronic device (101) determines that beamforming is possible at the i-th frequency (or the i-th frequency band) (e.g., 'Yes' in operation 901), in operation 903, beamforming may be performed on an audio signal of the i-th frequency (or the i-th frequency band). For example, the beamforming may be performed based on a data-based beamforming method or a GSC-based beamforming method. For example, the beamforming may be performed based on a beamforming variable determined based on a beamforming result of the i-th frequency (or the i-th frequency band) performed at a previous point in time (or a previous frame).

[0157] According to one embodiment, the electronic device (101) may determine whether distortion occurs in the audio signal due to beamforming in operation 905. For example, if the audio processing circuit (412) (e.g., the beam control module (604) or the error detection module (608) of FIG. 6) performs PLC on the audio signal of the i-th frequency (or the i-th frequency band) received from at least one of the first external electronic device (300) or the second external electronic device (320), the audio processing circuit (412) may determine that the audio signal is distorted due to beamforming. If the audio processing circuit (412) (e.g., the beam control module (604) or the error detection module (608) of FIG. 6) does not perform PLC on the audio signal of the i-th frequency (or the i-th frequency band) received from the first external electronic device (300) and the second external electronic device (320), the audio processing circuit (412) may determine that the audio signal is not distorted due to beamforming.

[0158] For example, the audio processing circuit (412) (e.g., the error detection module (608) of FIG. 6) may determine that distortion has occurred in the audio signal due to beam forming if there is a frequency band (or frequency) in which the power of the audio signal of the i-th frequency (or i-th frequency band) included in the output of the beam forming module (606) has increased by a specified reference value or more compared to the power of the audio signal of the i-th frequency (or i-th frequency band) input to the beam forming module (606). For example, the audio processing circuit (412) (e.g., the error detection module (608) of FIG. 6) may determine that distortion has not occurred in the audio signal due to beam forming if there is no frequency band (or frequency) in which the power of the audio signal of the i-th frequency (or i-th frequency band) included in the output of the beam forming module (606) has increased by a specified reference value or more compared to the power of the audio signal of the i-th frequency (or i-th frequency band) input to the beam forming module (606).

[0159] According to one embodiment, when the electronic device (101) determines that beamforming is not possible at the i-th frequency (or the i-th frequency band) (e.g., 'No' in operation 901), in operation 907, the electronic device (101) may select any one of the audio signals of the i-th frequency (or the i-th frequency band) received from the first external electronic device (300) and the second external electronic device (320). For example, the electronic device (101) may select an audio signal having a low noise level among the audio signals received from the first external electronic device (300) and the second external electronic device (320) corresponding to the i-th frequency (or the i-th frequency band) for which beamforming is restricted.

[0160] According to one embodiment, the electronic device (101) can remove noise from a selected audio signal based on a noise level. The electronic device (101) can detect the user's voice data from the noise-removed audio signal included in a section where the user's voice is determined to be present (e.g., a voice utterance section).

[0161] According to one embodiment, the electronic device (101) can repeatedly perform operations 901 to 907 of FIG. 9 for each frequency (or frequency band) included in an audio signal of a specified time interval (e.g., frame) received by the electronic device (101) from external electronic devices (300 and 320).

[0162] According to one embodiment, the electronic device (101) can detect voice data through beamforming based on audio signals collected by each of a plurality of external electronic devices (300 and 320) through a plurality of microphones. In this case, the electronic device (101) can perform beamforming in a three-dimensional plane by utilizing a plurality of microphones included in the plurality of external electronic devices (300 and 320), thereby improving the quality (e.g., SNR) of the voice data, as illustrated in FIG. 10.

[0163] FIG. 10 is an example of an audio signal obtained through beam forming in an electronic device according to one embodiment.

[0164] According to one embodiment referring to FIG. 10, the electronic device (101) can detect voice data through first beam forming (1000) based on audio signals collected through microphones of one external electronic device (e.g., the first external electronic device (300) or the second external electronic device (320) of FIG. 3) wirelessly connected to the electronic device (101).

[0165] According to one embodiment, the electronic device (101) can detect voice data through second beamforming (1010) based on audio signals collected through microphones of external electronic devices (e.g., the first external electronic device (300) and the second external electronic device (320) of FIG. 3) wirelessly connected to the electronic device (101).

[0166] According to one embodiment, the noise level (1020) included in the result of performing the first beamforming (1000) may be higher than the noise level (1030) included in the result of performing the second beamforming (1010). For example, the quality (e.g., SNR) of voice data when performing the first beamforming (1000) may include a first value (e.g., about 8.49 dB). The quality (e.g., SNR) of voice data when performing the second beamforming (1010) may be improved to a second value (e.g., about 18.67 dB) higher than the first value.

[0167] FIG. 11 is a flowchart (1100) for updating beamforming variables in an electronic device according to one embodiment. For example, at least a portion of FIG. 11 may include detailed operations for operation 811 of FIG. 8. In the following embodiments, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel. For example, the electronic device of FIG. 11 may be the electronic device (101) of FIG. 1, FIG. 2, FIG. 3, or FIG. 4.

[0168] According to one embodiment referring to FIG. 11, when an electronic device (e.g., electronic device (101)) determines that there is no distortion in an audio signal through beamforming (e.g., 'No' in operation 807 of FIG. 8), in operation 1101, it may determine whether audio signals received from external electronic devices (e.g., the first external electronic device (300) and the second external electronic device (320) of FIG. 3) are detected in a voice generation section. For example, when the audio processing circuit (412) (e.g., VAD (602) of FIG. 6) obtains information indicating the presence of a voice from at least one of the first external electronic device (300) or the second external electronic device (320), it may determine that a user's voice is present in the audio signals received from the external electronic devices (300 and 320). When the audio processing circuit (412) (e.g., VAD (602) of FIG. 6) obtains information indicating that there is no voice from the first external electronic device (300) and the second external electronic device (320) and / or information indicating that the presence of a voice cannot be determined, the audio processing circuit (412) may determine that the user's voice does not exist in the audio signals received from the external electronic devices (300 and 320).

[0169] According to one embodiment, when the electronic device (101) determines that audio signals received from external electronic devices (e.g., the first external electronic device (300) and the second external electronic device (320) of FIG. 3) are detected in a voice generation section (e.g., 'Yes' in operation 1101), in operation 1103, the electronic device may update a covariance vector for voice based on the audio signals received from the external electronic devices (e.g., the first external electronic device (300) and the second external electronic device (320) of FIG. 3). For example, when the audio processing circuit (412) (e.g., the beam control module (604) of FIG. 6) performs beamforming on an audio signal of an ith frequency (or an ith frequency band), the electronic device may update a covariance vector for voice based on a voice component included in the audio signal of the ith frequency (or the ith frequency band).

[0170] According to one embodiment, the electronic device (101) may update (or generate) beamforming variables based on a covariance vector for the voice, at operation 1105.

[0171] According to one embodiment, when the electronic device (101) determines that audio signals received from external electronic devices (e.g., the first external electronic device (300) and the second external electronic device (320) of FIG. 3) are not detected in the voice generation section (e.g., 'No' in operation 1101), in operation 1107, the electronic device may update a covariance vector for noise based on the audio signals received from the external electronic devices (e.g., the first external electronic device (300) and the second external electronic device (320) of FIG. 3). For example, when the audio processing circuit (412) (e.g., the beam control module (604) of FIG. 6) performs beamforming on an audio signal of an ith frequency (or an ith frequency band), the electronic device may update a covariance vector for noise based on a noise component included in the audio signal of the ith frequency (or the ith frequency band).

[0172] According to one embodiment, the electronic device (101) may update (or generate) beamforming variables based on a covariance vector for noise, in operation 1109.

[0173] According to one embodiment, the electronic device (101) may perform beamforming on audio signals received from the first external electronic device (300) and the second external electronic device (320) at the next time point (or next frame) using beamforming variables updated (or generated) based on the covariance vector for voice or the covariance vector for noise.

[0174] FIG. 12 is an example of an audio signal obtained by selectively updating beam forming variables in an electronic device according to one embodiment.

[0175] According to one embodiment referring to FIG. 12, when the electronic device (101) performs beamforming based on beamforming variables updated (or generated) based on audio signals determined to have packet errors, the quality of the audio signal may be degraded due to the beamforming. For example, when the electronic device (101) updates beamforming variables based on a covariance vector for noise based on audio signals determined to have packet errors (1210), the amount of noise removal may be reduced (about 5 dB) (1212) compared to when the beamforming variables are updated based on audio signals in which packet errors do not occur (1200). For example, when the electronic device (101) updates beamforming variables based on a covariance vector for voice based on audio signals in which packet errors do not occur (1220), distortion may occur in voice data (1222) compared to when the beamforming variables are updated based on audio signals in which packet errors do not occur (1200).

[0176] According to one embodiment, the electronic device (101) can selectively update (or generate) beamforming variables based on whether audio signals received from a plurality of external electronic devices (300 and 320) are distorted in order to reduce distortion of audio signals due to beamforming. For example, if no packet error occurs in the audio signals received from the plurality of external electronic devices (300 and 320), the electronic device (101) can perform beamforming using an updated beamforming vector based on the audio signals received from the plurality of external electronic devices (300 and 320).

[0177] FIG. 13 is an example of an audio signal obtained through beam forming in an electronic device located in a first region according to one embodiment.

[0178] According to one embodiment referring to FIG. 13, the electronic device (101) can detect voice data through beamforming based on audio signals collected through microphones of at least one external electronic device (e.g., the first external electronic device (300) and / or the second external electronic device (320) of FIG. 3) wirelessly connected to the electronic device (101) in a first area (e.g., inside a vehicle).

[0179] For example, the electronic device (101) can detect voice data through first beam forming (1300) based on audio signals collected through microphones of an external electronic device (e.g., the first external electronic device (300) or the second external electronic device (320) of FIG. 3) wirelessly connected to the electronic device (101).

[0180] For example, the electronic device (101) can detect voice data through second beam forming (1310) based on audio signals collected through microphones of external electronic devices (e.g., the first external electronic device (300) and the second external electronic device (320) of FIG. 3) wirelessly connected to the electronic device (101).

[0181] According to one embodiment, the noise level (1302) included in the result of performing the first beamforming (1300) may be higher than the noise level (1312) included in the result of performing the second beamforming (1310). For example, the quality (e.g., SNR) of voice data when performing the first beamforming (1300) may include a first value (e.g., about 21.6 dB). The quality (e.g., SNR) of voice data when performing the second beamforming (1310) may be improved to a second value (e.g., about 27.15 dB) higher than the first value.

[0182] According to one embodiment, a method of operating an electronic device (e.g., the electronic device (101) of FIG. 1, FIG. 2, FIG. 3, or FIG. 4) may include an operation of receiving audio signals from a plurality of external electronic devices (e.g., the first external electronic device (300) and the second external electronic device (320) of FIG. 3 or the external electronic device (420) of FIG. 4). According to one embodiment, the method of operating the electronic device may include an operation of identifying a frequency band in which beamforming is possible in audio signals. According to one embodiment, the method of operating the electronic device may include an operation of performing beamforming on audio signals in a frequency band in which beamforming is possible. According to one embodiment, the method of operating the electronic device may include an operation of identifying whether distortion has occurred in the audio signal due to beamforming. According to one embodiment, the method of operating the electronic device may include an operation of updating a beamforming variable based on a voice component or a noise component included in the audio signals when it is determined that distortion has not occurred in the audio signal due to beamforming. According to one embodiment, a method of operating an electronic device may include an operation of limiting an update of a beamforming variable when it is determined that distortion has occurred in an audio signal due to beamforming.

[0183] In one embodiment, the beamforming variables can be used for beamforming of an audio signal performed in the next time interval (or next frame).

[0184] According to one embodiment, the operation of determining whether distortion has occurred in the audio signal may include an operation of determining that distortion has occurred in the audio signal due to beamforming when a PLC (packet loss concealment) operation has been performed on the audio signal in a frequency band where beamforming is possible. According to one embodiment, the operation of determining whether distortion has occurred in the audio signal may include an operation of determining that distortion has not occurred in the audio signal due to beamforming when a PLC operation has not been performed on the audio signal in a frequency band where beamforming is possible.

[0185] According to one embodiment, the operation of determining whether distortion has occurred in the audio signal may include an operation of determining that distortion has occurred in the audio signal due to beamforming if it is determined that the power of the audio signal has increased based on the beamforming result. According to one embodiment, the operation of determining whether distortion has occurred in the audio signal may include an operation of determining that distortion has not occurred in the audio signal due to beamforming if it is determined that the power of the audio signal has decreased based on the beamforming result.

[0186] According to one embodiment, a method of operating an electronic device may include an operation of identifying a frequency band in which beamforming is possible from audio signals received from external electronic devices when it is determined that the external electronic devices are worn on a part of a user's body.

[0187] According to one embodiment, the operation of determining a frequency band capable of beamforming may include an operation of determining whether an updated frequency band exists based on another audio signal in beamformed audio signals received from each external electronic device based on control signals received from the external electronic devices. According to one embodiment, the operation of determining a frequency band capable of beamforming may include an operation of determining a frequency band updated based on another audio signal as a frequency band in which beamforming is restricted. According to one embodiment, the operation of determining a frequency band capable of beamforming may include an operation of determining a frequency band that has not been updated based on another audio signal as a frequency band capable of beamforming.

[0188] According to one embodiment, a method of operating an electronic device may include an operation of removing noise from a beamforming result (e.g., an audio signal detected through beamforming) for a frequency band in which beamforming is possible and an audio signal in a frequency band in which beamforming is limited. According to one embodiment, the method of operating an electronic device may include an operation of removing an echo component from an audio signal from which noise has been removed.

[0189] According to one embodiment, the operation of updating the beamforming variable may include an operation of checking whether a designated section for beamforming is a speech generation section based on a control signal received from external electronic devices when it is determined that no distortion has occurred in the audio signal due to beamforming. According to one embodiment, the operation of updating the beamforming variable may include an operation of updating a variable related to beamforming based on a speech component included in audio signals when the designated section for beamforming is a speech generation section. According to one embodiment, the operation of updating the beamforming variable may include an operation of updating a variable related to beamforming based on a noise component included in audio signals when the designated section for beamforming is not a speech generation section.

[0190] The embodiments of the present invention disclosed in this specification and drawings are merely specific examples presented to easily explain the technical contents according to the embodiments of the present invention and to help understand the embodiments of the present invention, and are not intended to limit the scope of the embodiments of the present invention. Therefore, the scope of the embodiments of the present invention should be interpreted to include all changes or modified forms derived based on the technical idea of ​​the embodiments of the present invention, in addition to the embodiments disclosed herein.

Claims

1. In an electronic device (101), Communication circuit (192 or 414), At least one processor (120 or 410) comprising a processing circuit, and When executed individually or collectively by at least one processor (120 or 410), the electronic device (101) Receive audio signals from external electronic devices through the above communication circuit (192 or 414), Check the frequency band in which beamforming is possible from the received audio signals, Perform beamforming on an audio signal in a frequency band where beamforming is possible, Check whether the audio signal is distorted due to the above beam forming, If it is determined that no distortion occurs in the audio signal due to the above beamforming, the beamforming variable is updated based on the voice component or noise component included in the received audio signals, An electronic device including a memory (130 or 416) storing instructions for limiting updating of beamforming variables when it is determined that distortion has occurred in an audio signal due to the beamforming.

2. In paragraph 1, The above beam forming variable is an electronic device used for beam forming of an audio signal performed in the next time interval.

3. In paragraph 1 or 2, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: When a PLC (packet loss concealment) operation is performed on an audio signal in a frequency band where the above beamforming is possible, it is determined that distortion has occurred in the audio signal due to the beamforming. An electronic device including instructions for determining that no distortion occurs in an audio signal due to beamforming when a PLC operation is not performed on an audio signal in a frequency band in which beamforming is possible.

4. In paragraph 1 or 2, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: If it is determined that the power of the audio signal has increased based on the above beamforming result, it is determined that distortion has occurred in the audio signal due to the beamforming. An electronic device including instructions for determining that no distortion occurs in the audio signal due to the beamforming when it is determined that the power of the audio signal has decreased based on the beamforming result.

5. In any one of paragraphs 1 to 4, The above memory, when executed individually or collectively by the at least one processor, causes the electronic device to: An electronic device storing instructions for checking a frequency band in which beamforming is possible from audio signals received from the external electronic devices when the external electronic devices are determined to be worn on a part of the user's body.

6. In any one of paragraphs 1 to 5, An electronic device in which audio signals received from the external electronic devices include audio signals beam-formed based on audio signals collected through a plurality of microphones in each external electronic device.

7. In paragraph 6, The above memory, when executed individually or collectively by the at least one processor, causes the electronic device to: Based on the control signals received from the external electronic devices, it is determined whether there is an updated frequency band based on another audio signal in the beamformed audio signals received from each external electronic device, The updated frequency band based on the above other audio signal is determined to be a frequency band to which the beamforming is limited, An electronic device storing instructions for determining a non-updated frequency band as a frequency band capable of beamforming based on the above-described other audio signal.

8. In paragraph 7, The above memory, when executed individually or collectively by the at least one processor, causes the electronic device to: The beamforming result for the frequency band where the beamforming is possible and the noise of the audio signal in the frequency band where the beamforming is limited are removed, An electronic device storing instructions for removing echo components from an audio signal from which noise has been removed.

9. In paragraph 1, The above memory, when executed individually or collectively by the at least one processor, causes the electronic device to: If it is determined that no distortion occurs in the audio signal due to the above beamforming, it is determined whether the designated section for beamforming is a voice generation section based on the control signal received from the external electronic devices, If the designated section for the beam forming is a voice generation section, the variables related to beam forming are updated based on the voice components included in the received audio signals, An electronic device storing instructions for updating variables related to beam forming based on noise components included in the received audio signals when the designated section for the beam forming is not a voice generation section.

10. In the operating method of the electronic device (101), An operation of receiving audio signals from multiple external electronic devices; An operation of checking a frequency band in which beamforming is possible in the received audio signals, An operation for performing beamforming on an audio signal in a frequency band in which beamforming is possible, An operation to check whether distortion occurs in the audio signal due to the above beam forming; When it is determined that no distortion occurs in the audio signal due to the above beamforming, an operation of updating the beamforming variable based on the voice component or noise component included in the received audio signals; A method including an operation of limiting the update of the beam forming variable when it is determined that distortion has occurred in the audio signal due to the beam forming.

11. In paragraph 10, The above beam forming variables are used in beam forming of an audio signal performed in the next time interval.

12. In paragraph 10 or 11, The operation to check whether distortion has occurred in the above audio signal is as follows: When a PLC (packet loss concealment) operation is performed on an audio signal in a frequency band where the beamforming is possible, an operation for determining that distortion has occurred in the audio signal due to the beamforming, and A method including an operation for determining that no distortion occurs in the audio signal due to the beamforming, if a PLC operation is not performed on the audio signal in a frequency band in which the beamforming is possible.

13. In paragraph 10 or 11, The operation to check whether distortion has occurred in the above audio signal is as follows: If it is determined that the power of the audio signal has increased based on the above beamforming result, an operation for determining that distortion has occurred in the audio signal due to the beamforming, and A method including an operation of determining that no distortion occurs in the audio signal due to the beamforming when it is determined that the power of the audio signal has decreased based on the beamforming result.

14. In any one of paragraphs 10 to 13, A method further comprising an action of checking a frequency band in which beamforming is possible in audio signals received from the external electronic devices, when it is determined that the external electronic devices are worn on a part of the user's body.

15. In paragraph 10, The operation of updating the above beam forming variables is: If it is determined that no distortion occurs in the audio signal due to the above beamforming, an operation of checking whether a designated section for beamforming is a voice generation section based on a control signal received from the external electronic devices; If the designated section for the beam forming is a voice generation section, an operation of updating a variable related to beam forming based on a voice component included in the received audio signals, and A method comprising an operation of updating a variable related to beam forming based on a noise component included in the received audio signals, when the designated section for the beam forming is not a voice generation section.

Images