Electronic device for communicating with tws and method for controlling the same

Abstract

An electronic device is disclosed. The electronic device according to the present disclosure comprises: a front camera; a rear camera; a microphone; a memory; a communication module; and at least one processor. The memory may store instructions that, when executed by the at least one processor, instruct the electronic device to: receive a first audio signal from an external electronic device via the communication module; acquiring a second audio signal via the microphone; determining whether a voice of a user wearing the external electronic device is included in the first audio signal; based on the voice of the user being included in the first audio signal, extracting a voice signal of the user included in the first audio signal; changing the second audio signal by using the voice signal of the user; based on the fact that the voice of the user is not included in the first audio signal, whether the front camera or the rear camera works or not is determined; and enhancing the ambient audio signal included in the second audio signal or controlling the weight of the second audio signal based on the operation of the front camera or the rear camera.

Description

Technical Field

[0001] Various embodiments of this disclosure relate to electronic devices that communicate with TWS and methods for controlling them. Background Technology

[0002] More and more services and additional features are being offered through electronic devices, such as smartphones or other portable electronic devices. To meet the diverse needs of users and improve the efficiency of electronic device usage, communication service operators and equipment manufacturers are competing to develop electronic devices with differentiated and diversified functions. Therefore, the various functions offered through electronic devices are becoming increasingly sophisticated.

[0003] This electronic device can use short-range wireless technology, such as Bluetooth, to connect to external devices, such as laptops, headphones, or headsets, to exchange information. For example, the electronic device can connect to headphones via Bluetooth to output music or video sound through the headphones.

[0004] The headphones come in the form of true wireless stereo (TWS) wireless headphones (or wireless input / output devices) that can be inserted into the user's ears to suit their needs. Each wireless headphone includes a microphone in both the left and right earcups, enabling binaural recording and more realistic sound recording.

[0005] The above information is provided as relevant technical information to aid in understanding this disclosure. No claim or judgment is made regarding whether any of the above content constitutes relevant technical background for this disclosure. Summary of the Invention

[0006] Solutions to technical problems

[0007] According to an embodiment, the electronic device may include a front-facing camera, a rear-facing camera, a microphone, a communication module including communication circuitry, at least one processor including processing circuitry, and a memory storing instructions, which, when executed by the at least one processor, cause the electronic device to perform the following operations.

[0008] According to an implementation, the memory may store instructions that, when executed by at least one processor, cause the electronic device to receive a first audio signal from an external electronic device via a communication module.

[0009] According to the implementation, when the instructions are executed by at least one processor, the electronic device acquires a second audio signal via a microphone.

[0010] According to an implementation, when executed by at least one processor, the instructions cause the electronic device to identify whether the first audio signal includes the voice of a user wearing an external electronic device.

[0011] According to an implementation, when executed by at least one processor, the instructions cause the electronic device to extract the user's voice signal included in the first audio signal based on the user's voice being included in the first audio signal.

[0012] According to the implementation, when the instructions are executed by at least one processor, the electronic device causes the second audio signal to be modified based on the user's voice, which is included in the first audio signal.

[0013] According to the implementation, when the instructions are executed by at least one processor, the electronic device identifies whether the front-facing camera or the rear-facing camera is working based on the fact that the user's voice is not included in the first audio signal.

[0014] According to the implementation, when executed by at least one processor, the instructions cause the electronic device to enhance the weight of the ambient audio signal included in the second audio signal or control the weight of the second audio signal based on the operation of the front or rear camera.

[0015] According to an embodiment, a method for controlling an electronic device may include receiving a first audio signal from an external electronic device via a communication module of the electronic device.

[0016] According to an embodiment, a method for controlling an electronic device may include acquiring a second audio signal via the microphone of the electronic device.

[0017] According to an embodiment, a method for controlling an electronic device may include identifying whether a first audio signal includes the voice of a user wearing an external electronic device.

[0018] According to an embodiment, a method for controlling an electronic device may include extracting the user's voice signal included in a first audio signal based on the user's voice included in the first audio signal.

[0019] According to an embodiment, a method for controlling an electronic device may include, based on a user’s voice included in a first audio signal, altering a second audio signal using the user’s voice signal.

[0020] According to an embodiment, a method for controlling an electronic device may include identifying whether a front-facing camera or a rear-facing camera is working based on the user's voice not being included in a first audio signal.

[0021] According to an embodiment, a method for controlling an electronic device may include enhancing an ambient audio signal included in a second audio signal or controlling the weight of the second audio signal based on the operation of a front-facing camera or a rear-facing camera.

[0022] According to an embodiment, a non-transitory computer-readable recording medium stores one or more programs, the one or more programs storing instructions that cause an electronic device to receive a first audio signal from an external electronic device via the electronic device's communication module.

[0023] According to one implementation, one or more programs may store instructions that cause the electronic device to acquire a second audio signal through the microphone of the electronic device.

[0024] According to one implementation, one or more programs may store instructions that enable an electronic device to identify whether a first audio signal includes the voice of a user wearing an external electronic device.

[0025] According to one implementation, one or more programs may store instructions that cause an electronic device to extract the user's voice signal included in the first audio signal based on the user's voice included in the first audio signal.

[0026] According to one implementation, one or more programs may store instructions that cause an electronic device to include a user's voice in a first audio signal and to modify a second audio signal using the user's voice signal.

[0027] According to one implementation, one or more programs may store instructions that cause the electronic device to identify whether the front camera or the rear camera of the electronic device is working, based on the fact that the user's voice is not included in the first audio signal.

[0028] According to an implementation, one or more programs may store instructions that cause the electronic device to enhance the ambient audio signal included in the second audio signal or control the weight of the second audio signal based on the operation of the front or rear camera. Attached Figure Description

[0029] Figure 1 This is a block diagram illustrating an electronic device in a network environment according to an embodiment; Figure 2 This is a block diagram illustrating an audio module according to an embodiment of the present disclosure; Figure 3 This is a view showing the network environment between an electronic device and an external electronic device according to an embodiment of the present disclosure; Figure 4 This is a block diagram illustrating an external electronic device according to an embodiment of the present disclosure; for example, the external electronic device may be a wearable electronic device including TWS; Figure 5 This is a flowchart illustrating, according to an embodiment, the operation of processing audio signals by an electronic device; Figure 6 This is a flowchart illustrating the operation of an electronic device according to an embodiment of the present disclosure that processes audio signals according to conditions; Figure 7 This is a view illustrating the operation of an electronic device according to an embodiment of the present disclosure to identify whether a user's voice is included in an audio signal based on TWS; Figure 8 This is a view illustrating audio signal processing operations performed by an electronic device according to an embodiment of the present disclosure, depending on whether the user speaks and whether the front / rear camera is working; Figure 9 This is a view illustrating audio signal processing operations performed by an electronic device according to embodiments of the present disclosure, based on whether the user speaks and whether the front / rear camera is active; and Figure 10 This is a view illustrating audio signal processing operations performed by an electronic device according to an embodiment of the present disclosure, depending on whether the user speaks and whether the front / rear camera is working. Detailed Implementation

[0030] Figure 1 This is a block diagram illustrating an electronic device 101 in a network environment 100 according to an embodiment. (Refer to...) Figure 1 In network environment 100, electronic device 101 can communicate with at least one of electronic devices 102 via a first network 198 (e.g., a short-range wireless communication network), or with electronic device 104 or server 108 via a second network 199 (e.g., a long-range wireless communication network). According to an embodiment, electronic device 101 can communicate with electronic device 104 via server 108. According to an embodiment, electronic device 101 may include a processor 120, memory 130, input module 150, sound output module 155, display module 160, audio module 170, sensor module 176, interface 177, connection terminal 178, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, user identification module (SIM) 196, or antenna module 197. In an embodiment, at least one of the components (e.g., connection terminal 178) may be omitted from electronic device 101, or one or more other components may be added to electronic device 101. According to an implementation, some of the components (e.g., sensor module 176, camera module 180, or antenna module 197) may be integrated into a single component (e.g., display module 160).

[0031] Processor 120 may execute software (e.g., program 140) to control at least one other component (e.g., hardware or software component) of electronic device 101 coupled to processor 120, and may perform various data processing or calculations. According to embodiments, as at least part of data processing or calculation, processor 120 may store commands or data received from another component (e.g., sensor module 176 or communication module 190) in volatile memory 132, process the commands or data stored in volatile memory 132, and store the result data in non-volatile memory 134. According to embodiments, processor 120 may include a main processor 121 (e.g., a central processing unit (CPU) or application processor (AP)) or an auxiliary processor 123 (e.g., a graphics processing unit (GPU), neural processing unit (NPU), image signal processor (ISP), sensor hub processor, or communication processor (CP)), which may operate independently of or in conjunction with the main processor 121. For example, when electronic device 101 includes a main processor 121 and an auxiliary processor 123, the auxiliary processor 123 may be configured to use lower power than the main processor 121 or be designated for a specific function. The auxiliary processor 123 may be implemented separately from the main processor 121 or as part of the main processor 121.

[0032] The auxiliary processor 123 can control at least some functions or states of components of the electronic device 101 (e.g., display module 160, sensor module 176, or communication module 190) in place of the main processor 121 when the main processor 121 is inactive (e.g., in a sleep state), or control at least one related function or state together with the main processor 121 when the main processor 121 is active (e.g., executing an application). According to embodiments, the auxiliary processor 123 (e.g., an image signal processor or a communication processor) can be implemented as part of another component (e.g., camera module 180 or communication module 190) functionally related to the auxiliary processor 123. According to embodiments, the auxiliary processor 123 (e.g., a neural processing unit) can include hardware structures dedicated to processing artificial intelligence models. The artificial intelligence model can be generated through machine learning. This learning can be performed, for example, by the electronic device 101 performing artificial intelligence or via a separate server (e.g., server 108). The learning algorithm can include, but is not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. The artificial intelligence model can include multiple layers of artificial neural networks. Artificial neural networks can be deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), deep Q-networks, or combinations of two or more thereof, but are not limited to these. Artificial intelligence models may additionally or alternatively include software structures in addition to hardware structures.

[0033] Memory 130 may store various data used by at least one component of electronic device 101 (e.g., processor 120 or sensor module 176). The various data may include, for example, input or output data of software (e.g., program 140) and associated commands. Memory 130 may include volatile memory 132 or non-volatile memory 134.

[0034] Program 140 may be stored as software in memory 130 and may include, for example, an operating system (OS) 142, middleware 144, or application 146.

[0035] Input module 150 can receive commands or data from outside electronic device 101 (e.g., a user) to be used by other components of electronic device 101 (e.g., processor 120). Input module 150 may include, for example, a microphone, mouse, keyboard, keys (e.g., buttons), or digital pen (e.g., stylus).

[0036] The audio output module 155 can output audio signals to the outside of the electronic device 101. The audio output module 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as playing multimedia or playing recordings. The receiver can be used to receive incoming calls. According to an embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0037] Display module 160 can visually provide information to the outside of electronic device 101 (e.g., to a user). Display 160 may include, for example, a display, a holographic device, or a projector, and control circuitry for controlling a corresponding one of the display, holographic device, and projector. According to an embodiment, display 160 may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by a touch.

[0038] The audio module 170 can convert sound into electrical signals and vice versa. According to an embodiment, the audio module 170 can acquire sound via the input module 150, or output sound via the sound output module 155 or headphones of an external electronic device (e.g., electronic device 102) that is directly (e.g., wired) or wirelessly connected to the electronic device 101.

[0039] Sensor module 176 can detect the operating state of electronic device 101 (e.g., power or temperature) or the environmental state outside electronic device 101 (e.g., user state), and then generate an electrical signal or data value corresponding to the detected state. According to embodiments, sensor module 176 may include, for example, a gesture sensor, gyroscope sensor, atmospheric pressure sensor, magnetic sensor, accelerometer, grip sensor, proximity sensor, color sensor, infrared (IR) sensor, biometric sensor, temperature sensor, humidity sensor, or illuminance sensor.

[0040] Interface 177 may support one or more specified protocols for direct (e.g., wired) or wireless connection of electronic device 101 to external electronic device (e.g., electronic device 102). Depending on the implementation, interface 177 may include, for example, a High Definition Multimedia Interface (HDMI), a Universal Serial Bus (USB) interface, a Secure Digital (SD) card interface, or an audio interface.

[0041] Connection end 178 may include a connector through which electronic device 101 can be physically connected to an external electronic device (e.g., electronic device 102). According to embodiments, connection end 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0042] The tactile module 179 can convert electrical signals into mechanical stimulation (e.g., vibration or motion) or electrical stimulation, which a user can perceive through his touch or kinesthesia. According to embodiments, the tactile module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulator.

[0043] Camera module 180 can capture still or moving images. Depending on the implementation, camera module 180 may include one or more lenses, an image sensor, an image signal processor, or a flash.

[0044] The power management module 188 can manage the power supply to the electronic device 101. According to an embodiment, the power management module 188 can be implemented as at least part of, for example, a power management integrated circuit (PMIC).

[0045] Battery 189 can supply power to at least one component of electronic device 101. According to embodiments, battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0046] Communication module 190 can support the establishment of a direct (e.g., wired) or wireless communication channel between electronic device 101 and external electronic devices (e.g., electronic device 102, electronic device 104, or server 108), and perform communication via the established communication channel. Communication module 190 may include one or more communication processors that operate independently of processor 120 (e.g., application processor (AP)) and support direct (e.g., wired) or wireless communication. According to embodiments, communication module 190 may include wireless communication module 192 (e.g., cellular communication module, short-range wireless communication module, or Global Navigation Satellite System (GNSS) communication module) or wired communication module 194 (e.g., local area network (LAN) communication module or power line communication (PLC) module). A corresponding one of these communication modules can communicate via a first network 198 (e.g., a short-range communication network, such as Bluetooth). TM The communication module 192 can communicate with external electronic device 104 via a Wi-Fi Direct or Infrared Data Association (IrDA) network or a second network 199 (e.g., a long-range communication network such as a traditional cellular network, 5G network, next-generation communication network, the Internet, or a computer network (e.g., a local area network (LAN) or a wide area network (WAN))). Various types of communication modules can be implemented as a single component (e.g., a single chip) or as multiple components separate from each other (e.g., multiple chips). The wireless communication module 192 can use user information (e.g., International Mobile Subscriber Identity (IMSI)) stored in the user identification module 196 to identify or authenticate electronic device 101 within a communication network (such as a first network 198 or a second network 199).

[0047] Wireless communication module 192 can support 5G networks following 4G networks and next-generation communication technologies, such as New Radio (NR) access technology. NR access technology can support enhanced mobile broadband (eMBB), massive machine-type communication (mMTC), or ultra-reliable low-latency communication (URLLC). Wireless communication module 192 can support high-frequency bands (e.g., millimeter-wave bands) to achieve, for example, high data transmission rates. Wireless communication module 192 can support various technologies used to ensure performance in high-frequency bands, such as beamforming, massive MIMO, full-dimensional MIMO (FD-MIMO), array antennas, analog beamforming, or massive antennas. Wireless communication module 192 can support various requirements specified in electronic device 101, external electronic device (e.g., electronic device 104), or network system (e.g., second network 199). According to the implementation, the wireless communication module 192 may support the following peak data rates (e.g., 20 Gbps or higher), loss coverage (e.g., 164 dB or lower), or U-plane latency (e.g., 0.5 ms or less for both downlink (DL) and uplink (UL), or 1 ms or less for round trip).

[0048] Antenna module 197 can transmit signals or power to or from an external source (e.g., an external electronic device). According to one embodiment, antenna module 197 may include an antenna comprising a radiator formed on a substrate (e.g., a printed circuit board (PCB)) by a conductor or conductive pattern. According to another embodiment, antenna module 197 may include multiple antennas (e.g., an antenna array). In this case, at least one antenna suitable for a communication scheme used in a communication network (such as a first network 198 or a second network 199) can be selected from the multiple antennas by, for example, communication module 190. Signals or power can then be transmitted or received between communication module 190 and an external electronic device via the selected at least one antenna. According to another embodiment, antenna module 197 may further include other components besides the radiator (e.g., a radio frequency integrated circuit (RFIC)) as part of it.

[0049] According to an embodiment, antenna module 197 can form a millimeter-wave antenna module. According to an embodiment, the millimeter-wave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high-frequency band (e.g., millimeter-wave band), or a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving signals of a specified high-frequency band.

[0050] At least some of the aforementioned components can be interconnected and communicate signals (e.g., commands or data) between them via an inter-peripheral communication scheme (e.g., bus, general purpose input / output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)).

[0051] According to an implementation, commands or data can be sent or received between electronic device 101 and external electronic device 104 via server 108 connected to a second network 199. Each of the external electronic devices 102 or 104 can be the same as or a different type of device as electronic device 101. According to an implementation, all or some of the operations to be performed at electronic device 101 can be performed at one or more of the external electronic devices 102, 104, or 108. For example, if electronic device 101 is to automatically or in response to a request from a user or another device to perform a function or service, electronic device 101 can request one or more external electronic devices to perform at least a portion of the function or service, and not, or in addition to, performing the function or service. The one or more external electronic devices receiving the request can perform at least a portion of the requested function or service or additional functions or services related to the request, and transmit the result of the performance to electronic device 101. Electronic device 101 can provide the result as at least part of a response to the request, whether or not the result is further processed. For this purpose, technologies such as cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing can be used. Electronic device 101 can use, for example, distributed computing or mobile edge computing to provide ultra-low latency services. In another embodiment, external electronic device 104 may include an Internet of Things (IoT) device. Server 108 may be an intelligent server using machine learning and / or neural networks. Depending on the embodiment, external electronic device 104 or server 108 may be included in a second network 199. Electronic device 101 may be applied to intelligent services based on 5G communication technology or IoT-related technologies (e.g., smart homes, smart cities, smart cars, or healthcare).

[0052] Figure 2 This is a block diagram illustrating the audio module 170 according to an embodiment. (Refer to...) Figure 2 The audio module 170 may include, for example, an audio input interface 210, an audio input mixer 220, an analog-to-digital converter (ADC) 230, an audio signal processor 240, a digital-to-analog converter (DAC) 250, an audio output mixer 260, or an audio output interface 270.

[0053] The audio input interface 210 can receive audio signals corresponding to sounds acquired from outside the electronic device 101 via a microphone (e.g., a dynamic microphone, condenser microphone, or piezoelectric microphone) configured as part of the input module 150 or separate from the electronic device 101. For example, if the audio signal is acquired from an external electronic device 102 (e.g., headphones or a microphone), the audio input interface 210 can be directly connected to the external electronic device 102 via connection terminal 178, or wirelessly via wireless communication module 192 (e.g., Bluetooth). TM The audio input interface 210 is connected to receive audio signals. According to one embodiment, the audio input interface 210 can receive control signals (e.g., volume adjustment signals received via input buttons) associated with audio signals acquired from an external electronic device 102. The audio input interface 210 may include multiple audio input channels, and can receive different audio signals via corresponding channels. According to another embodiment, additionally or alternatively, the audio input interface 210 may receive audio signals from another component of the electronic device 101 (e.g., processor 120 or memory 130).

[0054] The audio input mixer 220 can combine multiple input audio signals into at least one audio signal. For example, according to an embodiment, the audio input mixer 220 can combine multiple analog audio signals input via the audio input interface 210 into at least one analog audio signal.

[0055] ADC 230 can convert analog audio signals into digital audio signals. For example, according to an embodiment, ADC 230 can convert analog audio signals received via audio input interface 210, or analog audio signals synthesized via audio input mixer 220, into digital audio signals.

[0056] The audio signal processor 240 can perform various processes on digital audio signals received via the ADC 230 or from another component of the electronic device 101. For example, according to embodiments, the audio signal processor 240 can perform operations on one or more digital audio signals, such as changing the sampling rate, applying one or more filters, interpolation, amplifying or attenuating a whole or part of a frequency band, noise processing (e.g., attenuating noise or echo), changing channels (e.g., switching between mono and stereo), mixing, or extracting a specified signal. According to embodiments, one or more functions of the audio signal processor 240 can be implemented in the form of an equalizer.

[0057] DAC 250 can convert digital audio signals into analog audio signals. For example, according to an embodiment, DAC 250 can convert digital audio signals processed by audio signal processor 240 or digital audio signals obtained from another component of electronic device 101 (e.g., processor 120 or memory 130) into analog audio signals.

[0058] The audio output mixer 260 can combine multiple audio signals to be output into at least one audio signal. For example, according to an embodiment, the audio output mixer 260 can combine an analog audio signal converted by the DAC 250 and another analog audio signal (e.g., an analog audio signal received via the audio input interface 210) into at least one analog audio signal.

[0059] Audio output interface 270 can output analog audio signals converted by DAC 250, or analog audio signals synthesized by audio output mixer 260, to the outside of electronic device 101 via sound output module 155. Sound output module 155 may include, for example, a speaker, such as a moving coil driver or a balanced armature driver, or a receiver. According to an embodiment, sound output module 155 may include multiple speakers. In this case, audio output interface 270 can output audio signals having multiple different channels (e.g., stereo channels or 5.1 channels) via at least some of the multiple speakers. According to an embodiment, audio output interface 270 can be directly connected to external electronic device 102 (e.g., external speakers or headphones) via connection terminal 178 or wirelessly connected via wireless communication module 192 to output audio signals.

[0060] According to the implementation, the audio module 170 can generate at least one digital audio signal by synthesizing multiple digital audio signals using at least one function of the audio signal processor 240, without separately including the audio input mixer 220 or the audio output mixer 260.

[0061] According to one embodiment, the audio module 170 may include an audio amplifier (not shown) (e.g., a speaker amplifier circuit) capable of amplifying analog audio signals input via the audio input interface 210 or outputting audio signals via the audio output interface 270. According to another embodiment, the audio amplifier may be configured as a separate module from the audio module 170.

[0062] Figure 3 This is a view showing the network environment between an electronic device and an external electronic device according to an embodiment of the present disclosure.

[0063] Reference Figure 3 Electronic device 101 (e.g., Figure 1The electronic device 101 can be connected to external electronic devices 310 and 330. The external electronic devices 310 and 330 may include a first device 310 and a second device 330 that are separated on the left and right sides in a TWS manner.

[0064] According to the implementation, the first device 310 can establish a first communication link 301 with the electronic device 101, and the second device 330 can establish a third communication link 305 with the electronic device 101. For example, a second communication link 303 can be established (or connected) between the first device 310 and the second device 330. The first communication link 301 and the third communication link 305 can be the same or similar communication methods. The method by which the electronic device 101, the first device 310, and the second device 330 each establish communication links can be called an independent link mechanism.

[0065] According to the implementation, the second communication link 303 can be used to send / receive control signals and information between the first device 310 and the second device 330, and may not be used for data packet transmission. The first device 310 can send first audio data (e.g., audio packet 1) acquired by itself to the electronic device 101 through the first communication link 301, and the second device 330 can send second audio data (e.g., audio packet 2) acquired by itself to the electronic device 101 through the third communication link 305. The electronic device 101 can synchronize the first audio data and the second audio data based on the index number included in the first audio data and the second audio data, and store the synchronized first audio data and the second audio data in the memory 130. The electronic device 101 can output the first audio data and the second audio data through the sound output module 155.

[0066] According to the implementation, electronic device 101 and external electronic devices 310 and 330 can establish a communication connection using a mechanism other than an independent link mechanism.

[0067] According to the implementation, the first device 310 can establish a first communication link 301 with the electronic device 101, establish (or connect to) a second communication link 303 with the second device 330, and send information about the first communication link 301 (e.g., first communication link information) to the second device 330. The second device 330 can establish (or connect to) a third communication link 305 with the electronic device 101 based on the first communication link information. This can be called a sniffing mechanism.

[0068] According to an implementation, the first communication link information may include information for connecting to the first communication link 301, such as a Bluetooth address, FHS packet information, or a link key. For example, the first communication link 301 and the third communication link 305 may be on the same channel or using the same communication mechanism. The second device 330 may receive (e.g., sniff) information (or packets) sent / received between the first device 310 and the electronic device 101 via the third communication link 305. For example, the electronic device 101 may identify the first device 310 connected via the first communication link 301 and the second device 330 connected via the third communication link 305 as the same device.

[0069] According to the implementation, the second communication link 303 can be used to send / receive control signals and information between the first device 310 and the second device 330, and may not be used for data packet transmission. The first device 310 can send first audio data (e.g., audio packet 1) acquired by itself to the electronic device 101 through the first communication link 301, and the second device 330 can directly send second audio data (e.g., audio packet 2) acquired by itself to the electronic device 101 through the third communication link 305. The first device 310 and the second device 330 can send audio data acquired by the first device 310 and the second device 330 to the electronic device 101 through the first communication link 301 or the third communication link 305, respectively. The electronic device 101 can synchronize the first audio data and the second audio data based on the index number included in the first audio data and the second audio data, and store the synchronized first audio data and the second audio data in the memory 130. The electronic device 101 can output the first audio data and the second audio data through the sound output module 155.

[0070] According to an implementation, the electronic device 101 may be connected to one of the first device 310 and the second device 330, rather than both.

[0071] According to the implementation, the first device 310 may be a master device (PE) that can communicate directly with the electronic device 101, and the second device 330 may be a slave device (SE) that does not communicate directly with the electronic device 101.

[0072] According to the implementation, when the first device 310 is the master device and the second device 330 is the slave device, the first device 310 can establish a first communication link 301 with the electronic device 101, and can establish (or connect to) a second communication link 303 with the second device 330. This can be referred to as a relay mechanism.

[0073] According to the implementation method, when a communication connection is established using a relay mechanism, the first communication link 301 and the second communication link 303 can be different links (or channels) or different communication mechanisms. For example, the first communication link 301 can be connected via Bluetooth, and the second communication link 303 can be connected via Bluetooth Low Energy.

[0074] According to an implementation, the second device 330 can send second audio data (e.g., audio packet 2) acquired by itself to the first device 310 via the second communication link 303. For example, the first device 310 can synchronize the first audio data (e.g., audio packet 1) acquired by itself with the second audio data (e.g., audio packet 1 + audio packet 2) acquired from the second device 330, and can send the synchronized first audio data to the electronic device 101 via the first communication link 301. For example, the electronic device 101 can store the received first and second audio data in a memory (e.g., ...). Figure 1 The memory 130) is used for storage. Alternatively, for example, electronic device 101 may synchronize and store the first and second audio data based on index numbers included in each audio data. For example, electronic device 101 may use a speaker (e.g., Figure 1 The sound output module 155 outputs the first audio data and the second audio data.

[0075] Figure 4 This is a block diagram illustrating an external electronic device according to an embodiment of the present disclosure. For example, the external electronic device may be a wearable electronic device including TWS.

[0076] Reference Figure 4 External electronic devices according to various implementations (e.g., Figure 2 The wearable electronic device 200 may include a first device (e.g., Figure 3 First device 310), second device (e.g., Figure 3 At least one of the second device 330 and the housing. In the following, the external electronic device described in this disclosure may refer to the first device 310 or the second device 330.

[0077] According to an embodiment, the first device 310 may include at least one of a first sensor module 311, a first microphone 313, a first speaker 315, a first charging module 317, a first interface 319, a first communication module 321, a first processor 323, a first touch sensor 325, a first memory 327, or a first battery 329. According to an embodiment, at least one of these components may be omitted from the first device 310, or one or more other components may be added to the first device 310. In an embodiment, some of these components may be integrated into a single component.

[0078] According to an embodiment, the first sensor module 311 is used to determine whether the first device 310 is worn, and may include at least one of a proximity sensor, a touch sensor, an accelerometer sensor, or a gyroscope sensor. For example, the first sensor module 311 may be a proximity sensor or a touch sensor that detects an object approaching the first device 310. For example, when the first sensor module 311 is a proximity sensor or a touch sensor, the first sensor module 311 may be arranged in the area inserted into the user's ear. Alternatively, the first sensor module 311 may be an accelerometer sensor that measures the dynamic forces (such as acceleration, vibration, and impact) of an object. Alternatively, the first sensor module 311 may be a gyroscope sensor that measures the angular velocity of an object.

[0079] According to one embodiment, the first microphone 313 can convert sound into an electrical signal. According to one embodiment, the first microphone 313 can acquire sound (or audio) and convert it into an electrical signal. According to one embodiment, the first microphone 313 may include an external microphone capable of receiving external sound and an internal microphone capable of receiving sound reflected from the user's ear (e.g., reverberation or echo). The first speaker 315 can convert electrical signals into sound. The first speaker 315 can output audio (or sound) signals to the outside of the first device 310. For example, the first speaker 315 may include a receiver. The first speaker 315 can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. According to one embodiment, the receiver can be implemented separately from the first speaker 315 or as part of the first speaker 315. According to one embodiment, the number of first microphones 313 and the number of first speakers 315 can be varied as needed.

[0080] According to an embodiment, the first charging module 317 can manage the power supplied to the first device 310. For example, the first charging module 317 can charge the first battery 329 using the power received through the first interface 319. The first charging module 317 can be implemented as at least a portion of a power management integrated circuit (PMIC). The first interface 319 may include a connector through which the first device 310 can be physically connected to the housing 350.

[0081] According to the implementation method, the first communication module 321 can communicate with an external electronic device (e.g., Figure 1The first communication module 321 establishes a wireless communication channel (either an electronic device 101 or a second device 330) and can support communication via the established communication channel. The first communication module 321 can connect to an external electronic device or to an access point or network via Bluetooth, Bluetooth Low Energy, Wi-Fi, ANT+ (Adaptive Network Topology), LTE, 5G, or Narrowband Internet of Things (NB-IoT). The first communication module 321 can receive acoustic signals from the external electronic device or send sensing information (or sensing signals) or acoustic signals to the external electronic device.

[0082] According to embodiments, the first processor 323 can execute software to control at least one other component (e.g., hardware or software component) of the first device 310 connected to the first processor 323, and can perform various data processing or calculations. According to embodiments, as at least part of the data processing or calculations, the first processor 323 can store commands or data received from another component (e.g., the first sensor module 311 or the first communication module 321) in a first memory 327, process the commands or data stored in the first memory 327, and store the result data in the first memory 327. According to embodiments, the first processor 323 may include a main processor (e.g., a central processing unit (CPU) or an application processor) or an auxiliary processor (e.g., a sensor hub processor or a communication processor), which may operate independently of or in conjunction with the main processor. For example, when the first device 310 includes a main processor and an auxiliary processor, the auxiliary processor may be configured to use lower power than the main processor or be dedicated to a specific function. The auxiliary processor may be implemented separately from the main processor or as part of the main processor.

[0083] According to an implementation, the first processor 323 can connect to the electronic device 101 via a first communication link and can connect to the second device 330 via a second communication link. For example, the first and second communication links can be different links (or channels) or can be connected using different communication mechanisms. The first processor 323 can send information about the first communication link (e.g., Bluetooth address, File System Hierarchy (FHS) packet information, or link key) to the second device 330. For example, when the first processor 323 receives a recording instruction from the electronic device 101, it can instruct the second device 330 to turn on its microphone. The first processor 323 can instruct the second device 330 to operate the microphone (or issue a microphone turn-on command), and after a predetermined time has elapsed, the first processor 323 can operate the first microphone 313 to receive (or acquire) audio. In an implementation, the first processor 323 can monitor (or analyze) the transmission environment (or wireless transmission environment) by exchanging transmit buffer status or signal strength with the second device 330. The transmit buffer status can refer to the current remaining buffer space of the transmit buffer of the controller of the first device 310 or the current remaining buffer space of the transmit buffer of the controller of the second device 330. The signal strength may include the signal strength of the first device 310 (e.g., Received Signal Strength Indicator (RSSI)) or the signal strength of the second device 330.

[0084] According to an implementation, the first processor 323 can determine the bit rate of the audio based on the transmission environment. For example, when the transmit buffer state is less than or equal to a first threshold or the signal strength exceeds a signal threshold, the first processor 323 can determine that it is a strong electric field (e.g., a good transmission environment) and determine a first bit rate. Alternatively, when the transmit buffer state exceeds the first threshold or the signal strength is less than or equal to the signal threshold, the first processor 323 can determine that it is a weak electric field (e.g., a poor transmission environment) and determine a second bit rate. The first bit rate may be greater than the second bit rate. In an implementation, when determining the bit rate, the first processor 323 can determine the packet type based on the determined bit rate. The first processor 323 can encode and store the audio based on the determined bit rate. The first processor 323 can encode the audio by assigning the same index number (or index value, index information) to the audio acquired at the same time as the second device 330. The first processor 323 can send the stored first audio data (or audio packets) to the electronic device 101, or it can send the stored first audio data together with the second audio data obtained from the second device 330 to the electronic device 101.

[0085] According to an embodiment, the first touch sensor 325 may be a sensor for controlling the first device 310. For example, when a touch is detected by the first touch sensor 325 while the first device 310 is outputting sound, the first device 310 may stop playback. After playback stops, the first device 310 may start playback again when a touch is detected by the first touch sensor 325. For example, the first touch sensor 325 may be arranged in the external area of ​​the first device 310 that is not inserted into the user's ear so as to receive touch input while the user is wearing the first device 310. Touch input may include, for example, single touch, multi-touch, swipe, or flick. In an embodiment, touch recognition via the first touch sensor 325 may be performed in various ways. For example, touch input may be recognized by at least one of capacitive, resistive, infrared, or ultrasonic methods. According to an embodiment, the first device 310 may include physical buttons or optical keys.

[0086] According to an implementation, the first memory 327 (or buffer) may store various data used by at least one component of the first device 310 (e.g., the first sensor module 311 or the first processor 323). The various data may include, for example, input or output data of software (e.g., a program) and associated commands.

[0087] According to one embodiment, the first battery 329 can supply power to at least one component of the first device 310. According to another embodiment, the first battery 329 may include a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0088] According to an embodiment, the second device 330 may include at least one of a second sensor module 331, a second microphone 333, a second speaker 335, a second charging module 337, a second interface 339, a second communication module 341, a second processor 343, a second touch sensor 345, a second memory 347, or a second battery 349. In an embodiment, at least one of these components may be omitted from the second device 330, or one or more other components may be added to the second device 330. In an embodiment, some of these components may be integrated into a single component. Since components included in the first device 310 and components included in the second device 330 are the same or similar, "first" and "second" may be added before the components to distinguish them. Components identical to those in the first device 310 are briefly described.

[0089] According to an embodiment, the second sensor module 331 can be used to determine whether the second device 320 is worn, and may include at least one of a proximity sensor, a touch sensor, an accelerometer sensor, or a gyroscope sensor. The second microphone 333 can convert sound into an electrical signal. The second speaker 335 can convert the electrical signal into sound. The second speaker 335 can output audio (or sound) signals to the outside of the second device 330. The second speaker 335 may include a receiver. The second speaker 335 can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. According to an embodiment, the receiver can be implemented separately from the second speaker 335 or as part of the second speaker 335.

[0090] According to an embodiment, the second charging module 337 can manage, at least partially, the power supplied to the second device 330 as part of the PMIC. The second charging module 337 can charge the second battery 349 using the power received through the second interface 339. The second interface 339 may include a connector through which the second device 330 can be physically connected to the housing 350.

[0091] According to an implementation, the second communication module 341 can communicate with an external electronic device (e.g., Figure 1 The electronic device 101 or the first device 310 establishes a wireless communication channel and can support communication via the established communication channel. The second communication module 341 can receive acoustic signals from an external electronic device or can send sensing information (or sensing signals) or acoustic signals to an external electronic device.

[0092] According to various embodiments, the second processor 343 can execute software to control at least one other component (e.g., hardware or software component) of the second device 330 connected to the second processor 343, and can perform various data processing or calculations. According to various embodiments, the second processor 343 can connect to the first device 310 via a second communication link, or can connect to the electronic device 101 via a first communication link based on first communication link information received from the first device 310. When the second processor 343 receives a microphone-on command (or microphone-on instruction) from the first device 310, the second processor 343 can turn on the second microphone 333 after a predetermined time to receive (or acquire) audio. The second processor 343 can monitor (or analyze) the transmission environment (or wireless transmission environment) by exchanging transmit buffer status or signal strength with the first device 310. The transmit buffer status can refer to the current remaining buffer space of the transmit buffer of the controller of the first device 310 or the current remaining buffer space of the transmit buffer of the controller of the second device 330. The signal strength can include the signal strength of the first device 310 (e.g., RSSI) or the signal strength of the second device 330.

[0093] According to an implementation, the second processor 343 can determine the audio bit rate based on the transmission environment. For example, when the transmit buffer state is less than or equal to a first threshold or the signal strength exceeds a signal threshold, the second processor 343 can determine that it is a strong electric field (e.g., a good transmission environment) and determine the first bit rate. Alternatively, when the transmit buffer state exceeds the first threshold or the signal strength is less than or equal to the signal threshold, the second processor 343 can determine that it is a weak electric field (e.g., a poor transmission environment) and determine the second bit rate. The first bit rate may be greater than the second bit rate. When the bit rate is determined, the second processor 343 can determine the packet type based on the determined bit rate. The second processor 343 can encode and store the audio based on the determined bit rate. The second processor 343 can encode the audio by assigning the same index number (or index value, index information) as the first device 310 to the audio acquired at the same time as the first device 310. The second processor 343 can send the stored second audio data (or audio packets) to the first device 310 or the electronic device 101.

[0094] According to an embodiment, the second touch sensor 345 may be a sensor for controlling the second device 330. For example, when a touch is detected by the second touch sensor 345 while the second device 330 is outputting sound, the second device 330 may stop playback. After playback has stopped, the second device 330 may start playback again when a touch is detected by the second touch sensor 345. The second memory 347 (or buffer) may store various data used by at least one component of the second device 330 (e.g., the second sensor module 331 or the second processor 343). The various data may include, for example, input or output data of software (e.g., a program) and associated commands. The second battery 349 may supply power to at least one component of the second device 330.

[0095] Figure 5 This is a flowchart schematically illustrating the operation of an electronic device processing audio signals according to an embodiment.

[0096] Reference Figure 5 In operation 510, electronic devices (e.g., Figure 1 Electronic devices 101 Figure 1 Processor 120 or Figure 3 Electronic device 101) from external electronic device (e.g., Figure 1 Electronic devices 104 or Figure 3 External electronic devices 310 and 330 receive the first audio signal.

[0097] According to the implementation, the external electronic device can be a wearable electronic device that performs wireless communication (e.g., Bluetooth communication) with the electronic device. For example, the external electronic device can be a true wireless stereo (TWS) earphone with separate left and right sides.

[0098] According to the implementation method, when a user performs a video recording function or a video call function while wearing an external electronic device for wireless communication, the electronic device receives a first audio signal from the external electronic device.

[0099] According to one embodiment, the external electronic device includes an internal microphone disposed internally and / or an external microphone disposed externally. According to one embodiment, the external electronic device generates a first audio signal by combining audio signals acquired from the internal microphone and audio signals acquired from the external microphone. According to one embodiment, the external electronic device transmits the first audio signal to the electronic device.

[0100] According to one embodiment, the external electronic device may include a voice pickup unit (VPU). According to another embodiment, the VPU is a technique for separating user speech from ambient audio, and the external electronic device may include an accelerometer for detecting vibrations of the user's vocal cords based on vibrations of the user's skull and tissues.

[0101] According to one embodiment, the electronic device can receive information detected by an accelerometer (or VPU) of an external electronic device. According to one embodiment, the electronic device can receive a first audio signal and information detected by the accelerometer from the external electronic device. According to one embodiment, the information detected by the accelerometer may include sensing values ​​detected by the accelerometer of the external electronic device during the time period in which the first audio signal is acquired by the microphone of the external electronic device. According to one embodiment, the information detected by the accelerometer may include information about segments identified as vocal cord vibrations within the sensing values ​​detected by the accelerometer.

[0102] According to an implementation, the electronic device can receive a user's voice signal extracted from a first audio signal by an external electronic device (e.g., a VPU).

[0103] According to the implementation, in operation 520, the electronic device uses a microphone (e.g., Figure 1 The input module 150) acquires the second audio signal.

[0104] According to one embodiment, the second audio signal can be acquired independently of the first audio signal. According to another embodiment, the second audio signal can be acquired within the same time period as the first audio signal.

[0105] According to an implementation, the electronic device can synchronize the first audio signal with the second audio signal based on the delay caused by receiving the first audio signal. For example, the electronic device can synchronize the first audio signal and the second audio signal by comparing the waveforms of the first audio signal and the second audio signal with respect to the time points of the first audio signal and the second audio signal that are identified as the same sound (e.g., user voice or ambient sound).

[0106] According to the implementation, in operation 530, the electronic device identifies whether the first audio signal includes the voice of a user wearing an external electronic device.

[0107] According to an implementation, the electronic device can identify a first portion of the user's voice in a first audio signal acquired through the microphone of the external electronic device, based on information detected by the accelerometer of the external electronic device.

[0108] According to the implementation method, the electronic device can identify whether the user's vocal cords are vibrating and the segment of the user's vocal cords vibrating based on the sensing value detected by the acceleration sensor of the external electronic device.

[0109] According to the implementation method, such as Figure 7 As shown, the electronic device can identify segments in the first audio signal that correspond to segments in which the user's vocal cords vibrate as segments that include the user's speech. According to an embodiment, when no segment is identified as a user's vocal cord vibration in the sensing values ​​detected by the accelerometer of an external electronic device, the electronic device can identify that the first audio signal does not include the user's speech.

[0110] Figure 7 This is a view illustrating the operation of an electronic device according to an embodiment of the present disclosure to identify whether a user's voice is included in an audio signal based on TWS.

[0111] Reference Figure 7 Electronic devices (e.g.) Figure 1 Electronic devices 101 Figure 1 Processor 120 or Figure 3 Electronic device 101) can be accessed from external electronic devices (e.g., Figure 1 Electronic devices 104 or Figure 3 External electronic devices 310 and 330 receive a first audio signal 710 and a VPU signal 720. According to an embodiment, the VPU signal 720 may be a sensed value sensed by an accelerometer of the external electronic device.

[0112] According to the implementation, the electronic device can identify segments 730 and 731 of the user's vocal cord vibration based on the VPU signal 720. For example, the electronic device can identify segments of the VPU signal 720 that have a set value or greater as segments 730 and 731 of the user's vocal cord vibration.

[0113] According to the implementation, the electronic device can identify the segments 730 and 731 where the user's vocal cords vibrate as the segments where the user speaks, and can identify the remaining segments 740 and 741 as the segments where the user does not speak.

[0114] return Figure 5 According to an implementation, in operation 540, the electronic device extracts the user's voice signal included in the first audio signal based on the user's voice, and can use the user's voice signal to change the second audio signal.

[0115] According to an embodiment, in operation 550, the electronic device identifies whether the front-facing camera or the rear-facing camera is active, and based on whether the front-facing camera or the rear-facing camera is active, enhances the weight of the ambient audio signal included in the second audio signal or controls the weight of the second audio signal. According to an embodiment, operations 540 and 550 are... Figure 5 The execution is shown as sequential, but can be selective depending on whether the first audio signal includes the user's voice.

[0116] According to an implementation, the electronic device can extract a user's voice signal, including segments of audio signals acquired through a microphone of an external electronic device that are identified as vocal cord vibrations.

[0117] According to an implementation, the electronic device can transmit signals from a first audio signal (e.g., ...). Figure 7 Remove the first part (e.g., 710) from the original text. Figure 7 The second part of the first audio signal other than 731 and 731) (e.g., Figure 7 The audio signals of 740 and 741 are used to extract the user's voice.

[0118] According to an implementation, an electronic device can extract a user's voice by removing the audio signal acquired by the external microphone of the external electronic device from a first audio signal received from an external electronic device.

[0119] According to the implementation, the electronic device can acquire the user's voice signal and environmental signal included in the audio signal based on the following equations (1) to (4).

[0120] A All = A T + A P Equation (1)

[0121] A All It can refer to all audio signals acquired through external electronic devices (e.g., TWS) and the microphone of electronic devices. AT can refer to audio signals acquired through external electronic devices. PIt can refer to audio signals acquired through the microphone of an electronic device.

[0122] A T = A Ti + A Te Equation (2)

[0123] A Ti It can be an audio signal acquired through the VPU of an external electronic device and an internal microphone, A Te The audio signal can be obtained through the external microphone of an external electronic device. According to the implementation, referring to equation (2), the audio signal obtained through the external electronic device can be the sum of the audio signal obtained through the internal microphone of the external electronic device and the audio signal obtained through the external microphone of the external electronic device.

[0124] A user = A T – A Te Equation (3)

[0125] A other = A All – A user Equation (4)

[0126] A user It could be the voice signal of a user wearing an external electronic device, A other It can be an ambient audio signal outside the user's environment. According to an embodiment, referring to equation (3), the user's voice signal can be obtained by removing the audio signal acquired by the external microphone of the external electronic device from the audio acquired by the external electronic device. According to an embodiment, referring to equation (4), A other It can be obtained by removing the user's voice signal from all audio signals.

[0127] According to the implementation, when the electronic device extracts user speech from the first audio signal, the electronic device can enhance the user speech component in the first audio signal by performing operations such as bandwidth extension (BWE).

[0128] According to the implementation method, when the first audio signal does not include the user's voice, the electronic device can change the first audio signal and / or the second audio signal based on whether the front camera or the rear camera is working.

[0129] According to one embodiment, when user voice is included in a first audio signal, the electronic device can modify a second audio signal to enhance the user's voice signal. According to another embodiment, when user voice is included in a first audio signal, the electronic device can modify a second audio signal to enhance the user's voice signal in the segment including the user's voice.

[0130] According to one embodiment, when user voice is included in a first audio signal, the electronic device can increase the weight of the combination of the first audio signal and the second audio signal to enhance the user's voice signal. According to another embodiment, when user voice is included in a first audio signal, the electronic device can increase the weight of the combination of the user's voice signal extracted from the first audio signal and the second audio signal to enhance the user's voice signal.

[0131] According to one embodiment, when the user's voice is included in the first audio signal and the front-facing camera is active, the electronic device can modify the first audio signal and / or the second audio signal to enhance the user's voice signal. According to another embodiment, when the user's voice is included in the first audio signal and the rear-facing camera is active, the electronic device can modify the first audio signal and / or the second audio signal to enhance the user's voice signal.

[0132] According to the implementation method, the electronic device can increase the weight of the first audio signal and the second audio signal by including the user's voice in the first audio signal.

[0133] According to the implementation, the electronic device can enhance the user's voice signal by modifying the second audio signal based on the signal of the acquired user voice segment amplified in the second audio signal by the VPU received from the external electronic device.

[0134] According to the implementation method, refer to Figure 6 , Figure 8 and Figure 10 The operation in the segment that includes the user's voice signal in the first audio signal received from an external electronic device is described in more detail.

[0135] According to the implementation, the electronic device can increase the weight of the ambient audio signal included in the second audio signal based on the fact that the user's voice is not included in the first audio signal and the front camera is working.

[0136] According to the implementation, the electronic device can enhance the ambient audio signal by amplifying the signal outside the acquired user voice segment in the second audio signal based on the VPU received from the external electronic device.

[0137] According to an implementation, the electronic device can obtain an audio signal (e.g., A) by adding an audio signal acquired from an external electronic device and an audio signal acquired through the microphone of the electronic device. All Remove the user's voice signal extracted from the first audio signal (e.g., A) user This is used to enhance ambient audio signals.

[0138] According to the implementation, the electronic device can enhance the ambient audio signal by adding a segment of the audio signal acquired through the microphone (excluding the user's voice) to a corresponding segment (or a segment excluding the user's voice) of the audio signal acquired from an external electronic device.

[0139] According to the implementation, the electronic device can maintain or reduce the weight of the second audio signal based on the fact that the front-facing camera is working and the user's voice is not included in the first audio signal when performing a video call. Therefore, the electronic device can provide high-quality video call functionality even when the user does not speak during the video call, without amplifying ambient sound.

[0140] According to the implementation method, refer to Figure 6 and Figure 9 A more detailed description of the operation in the segment of the first audio signal that does not include the user's voice when the front-facing camera of an electronic device is in operation.

[0141] According to the implementation, the electronic device can increase the weight of the ambient audio signal included in the second audio signal based on the fact that the user's voice is not included in the first audio signal and the rear camera is working.

[0142] According to the implementation method, refer to Figure 6 and Figure 10 A more detailed description of the operation in the segment of the first audio signal that does not include the user's voice when the rear camera of the electronic device is in operation.

[0143] According to one embodiment, the user can perform recovery when the audio signal is distorted during the operation of enhancing the user's voice or ambient audio signal. According to another embodiment, when the audio signal is distorted during the operation of enhancing the user's voice, the electronic device can recover the signal in a specific frequency range that was removed during the operation of extracting the user's voice signal from the first audio signal via bandwidth extension (BWE).

[0144] According to an implementation, when an audio signal is distorted during operation of an enhanced ambient audio signal, the electronic device can identify a first segment of the first audio signal where distortion has occurred based on the similarity or correlation of the frequency changes of the first audio signal over time. For example, the electronic device can identify the similarity or correlation between each of the audio signals of the first channel (e.g., the right channel) and the second channel (e.g., the left channel) of an external electronic device recorded via BLE operation in the frequency or time domain.

[0145] According to the implementation method, when distortion occurs only in one channel, the electronic device can copy the audio signal of the other channel that is not distorted and restore the audio signal of the distorted channel to the energy level of the distorted channel.

[0146] According to the implementation, when both channels are distorted and need to be restored, restoration can be performed by executing packet loss concealment (PLC) logic embedded in an external electronic device (e.g., TWS).

[0147] According to one embodiment, the electronic device can recover the first segment based on segments other than the first segment. According to another embodiment, in the first audio signal, the audio signal is stored for each segment of the buffer unit, and when the audio signal of one buffer is distorted, the distorted audio signal can be recovered based on the audio signal of the previous buffer.

[0148] According to one embodiment, the electronic device can identify distortion in a first segment of a first audio signal based on the similarity or correlation of signal values ​​over time between an audio signal obtained by combining a first audio signal acquired from an external electronic device and a second audio signal acquired through the microphone of the electronic device, and a user's voice signal extracted from the first audio signal. According to another embodiment, the electronic device can recover the first segment based on segments other than the first segment.

[0149] Figure 6 This is a flowchart illustrating the operation of an electronic device according to an embodiment of the present disclosure, which processes audio signals according to conditions.

[0150] Reference Figure 6 In operation 601, electronic devices (e.g., Figure 1 Electronic devices 101 Figure 1 Processor 120 or Figure 3 The electronic device 101 can acquire the first audio signal and the second audio signal.

[0151] According to the implementation method, the electronic device can be powered from an external electronic device (e.g., Figure 1 Electronic devices 104 or Figure 3 External electronic devices 310 and 330 receive a first audio signal. According to an embodiment, the operation of the electronic device receiving the first audio signal is similar to... Figure 5 The operation 510 is essentially the same, therefore its repeated description is omitted. According to the implementation, the electronic device can use a microphone (e.g., Figure 1 The input module 150) acquires the second audio signal. According to the embodiment, the operation of the electronic device acquiring the second audio signal is similar to... Figure 5 The operation is basically the same as 520, so its repeated description is omitted.

[0152] According to an implementation, in operation 602, the electronic device can identify whether the user's voice is included in the first audio signal.

[0153] According to an implementation, the electronic device can identify a first portion of the user's voice in a first audio signal acquired through the microphone of the external electronic device, based on information detected by the accelerometer of the external electronic device.

[0154] According to the implementation method, the electronic device can identify whether the user's vocal cords are vibrating and the segment of the user's vocal cords vibrating based on the sensing value detected by the acceleration sensor of the external electronic device.

[0155] According to the implementation method, whether the user's voice is included in the description of the first audio signal is related to... Figure 5 Operation 530 and Figure 7 The descriptions are basically the same, so repeated descriptions are omitted.

[0156] According to the implementation, when the first audio signal is identified as including the user's voice ("Yes" in operation 602), in operation 603, the electronic device can extract the user's voice signal from the first audio signal.

[0157] According to an implementation, the electronic device can remove a second portion of the first audio signal from the first audio signal, excluding the segment identified as the user's vocal cord vibration (e.g., ...). Figure 7 The audio signals of 740 and 741 are used to extract the user's voice.

[0158] According to an implementation, an electronic device can extract a user's voice by removing the audio signal acquired by the external microphone of the external electronic device from a first audio signal received from an external electronic device.

[0159] According to an implementation, in operation 604, the electronic device can increase the weights to enhance the user's voice signal. For example, the electronic device can increase the weights of the user's voice signal included in the first audio signal and / or the user's voice signal included in the second audio signal, based on the fact that the user's voice is included in the first audio signal.

[0160] According to the implementation method, the electronic device can increase the weight of the user's voice signal in the first audio signal and the second audio signal when they are combined.

[0161] According to the implementation, the electronic device can enhance the user's voice signal by modifying the second audio signal based on the signal of the acquired user voice segment amplified in the second audio signal by the VPU received from the external electronic device.

[0162] According to the implementation method, such as Figure 8 As shown, when the user's voice is included in the first audio signal and the front-facing camera is working, the electronic device can change the first audio signal and / or the second audio signal to enhance the user's voice signal.

[0163] Figure 8 This is a view illustrating the audio signal processing operations performed by an electronic device according to an embodiment of the present disclosure, depending on whether the user speaks and whether the front / rear camera is working.

[0164] Reference Figure 8 When a user 10 wearing an external electronic device 104 (e.g., TWS) activates (810) the front camera of electronic device 101, the user 10's voice 820 and ambient sound 830 can be acquired by electronic device 101 and external electronic device 104.

[0165] According to the implementation, the electronic device 101 can change the weight of the segment in the first audio signal received from the external electronic device 104 that identifies the user 10's voice 820 based on information obtained by the VPU of the external electronic device 104, so as to enhance the user 10's voice 820.

[0166] According to the implementation method, such as Figure 10 As shown, when the first audio signal includes the user's voice and the rear camera is working, the electronic device can change the second audio signal to enhance the user's voice signal.

[0167] Figure 10 This is a view illustrating audio signal processing operations performed by an electronic device according to an embodiment of the present disclosure, depending on whether the user speaks and whether the front / rear camera is working.

[0168] Reference Figure 10 (a) When a user 10 wearing an external electronic device 104 (e.g., TWS) activates (1010) the rear camera 182 of the electronic device 101, the user 10's voice 1020 and ambient sound 1030 can be acquired by the electronic device 101 and the external electronic device 104.

[0169] According to the implementation, the electronic device 101 can change the weight of the segment of the user 10's voice 1020 identified in the first audio signal received from the external electronic device 104 based on information obtained by the VPU of the external electronic device 104 in order to enhance the user 10's voice 1020.

[0170] return Figure 6 According to the implementation method, in operation 605, the electronic device can mix the first audio signal and the second audio signal according to the weights.

[0171] According to the implementation, the electronic device can mix the first audio signal and the second audio signal based on weights that are changed to enhance the user's voice. For example, the electronic device can mix (or merge) the user's voice signal extracted from the first audio signal acquired from an external electronic device with the second audio signal acquired through a microphone.

[0172] According to the implementation, the electronic device can mix (or generate) an enhanced audio of the user's voice signal by amplifying the signal of the acquired user voice segment in a second audio signal acquired through a microphone based on the VPU received from an external electronic device.

[0173] According to an implementation, when it is determined that the user's voice is not included in the first audio signal ("No" in operation 602), in operation 606, the electronic device can identify whether the currently operating camera is a front-facing camera. For example, when the user's voice is not included in the first audio signal while one of the electronic device's front-facing and rear-facing cameras is operating, the electronic device can identify whether the currently operating camera is a front-facing camera. According to an implementation, when the front-facing camera is not operating, the rear-facing camera may be operating.

[0174] According to the implementation, when the camera in operation is not a front-facing camera ("No" in operation 606), the electronic device can identify that the camera in operation is a rear-facing camera.

[0175] According to the implementation, when the camera in operation is a rear camera, in operation 607, the electronic device can increase the weight to enhance the ambient audio signal.

[0176] According to the implementation, the electronic device can enhance the ambient audio signal by amplifying the signal outside the user's voice segment in the entire audio signal acquired through the microphone, based on the VPU received from the external electronic device.

[0177] According to an implementation, an electronic device can enhance an audio signal by removing the user's voice signal extracted from the audio signal acquired from an external electronic device from an audio signal acquired by adding the entire audio signal acquired from the entire audio signal acquired from the external electronic device and the entire audio signal acquired from the microphone of the electronic device.

[0178] According to the implementation, the electronic device can enhance the ambient audio signal by adding the segment of the entire audio signal acquired through the microphone, excluding the user's voice, to the corresponding segment (or the segment excluding the user's voice) of the audio signal acquired from an external electronic device.

[0179] According to the implementation method, such as Figure 10 As shown, the electronic device can increase the weight of the ambient audio signal included in the second audio signal based on the fact that the user's voice is not included in the first audio signal and the rear camera is working.

[0180] Reference Figure 10(b) When the user 10 wearing an external electronic device (e.g., TWS) activates (1010) the rear camera 182 of the electronic device 101, there is no sound of the user 10, only ambient sound 1030 can be obtained by the electronic device 101 and the external electronic device 104.

[0181] According to the implementation, the electronic device 101 can enhance the ambient sound 1030 by changing the weight of segments in the audio signal received from the external electronic device that do not identify the user 10's voice, based on information obtained by the VPU of the external electronic device.

[0182] return Figure 6 According to the implementation method, the electronic device can continue to operate 605 and mix audio according to weights.

[0183] According to the implementation, the electronic device can mix audio based on weights that are changed to enhance ambient sound. For example, the electronic device can mix (or merge) a second audio signal acquired through a microphone with a first audio signal acquired by an external electronic device.

[0184] According to the implementation, the electronic device can mix (or generate) audio in which ambient sounds have been enhanced by amplifying signals from a second audio signal acquired through a microphone based on a VPU received from an external electronic device, in amplifying signals from segments other than the user's voice segment.

[0185] According to the implementation, when the front-facing camera in the camera is working ("Yes" in operation 606), in operation 608, the electronic device can identify whether a video call function is being performed.

[0186] According to the implementation, when the video call function is not performed ("No" in operation 608), the electronic device can continue to operate 607 to increase the weight to enhance the ambient audio signal.

[0187] According to the implementation method, such as Figure 9 As shown, when the video call function is not performed, the electronic device can increase the weight of the ambient audio signal included in the second audio signal based on the fact that the first audio signal does not include the user's voice and the front camera is working.

[0188] Figure 9 This is a view illustrating audio signal processing operations performed by an electronic device according to an embodiment of the present disclosure, depending on whether the user speaks and whether the front / rear cameras 181 / 182 are working.

[0189] Reference Figure 9(a) When the user 10 wearing an external electronic device 104 (e.g., TWS) activates (910) the front camera 181 of the electronic device 101, there is no voice of the user 10, only ambient sound 920 can be obtained by the electronic device 101 and the external electronic device 104.

[0190] According to the implementation, the electronic device 101 can enhance the ambient sound 920 by changing the weight of segments in the audio signal received from the external electronic device that do not identify the user 10's voice based on information obtained by the VPU of the external electronic device.

[0191] return Figure 6 According to the implementation method, the electronic device can continue to operate 605 and mix audio according to weights.

[0192] According to the implementation, when a video call function is being performed ("Yes" in operation 608), in operation 609, the electronic device can maintain or reduce the weight of the second audio signal.

[0193] According to the implementation method, such as Figure 9 As shown, when performing a video call, the electronic device can maintain or reduce the weight of the second audio signal based on the fact that the front-facing camera is working and the user's voice is not included in the first audio signal.

[0194] Reference Figure 9 (b) When the user 10 wearing an external electronic device (e.g., TWS) activates (910) the front camera 181 of the electronic device 101 by performing a video call, there is no voice of the user 10, only ambient sound 920 can be obtained by the electronic device 101 and the external electronic device.

[0195] According to the implementation, the electronic device 101 can maintain the weight of the second audio signal acquired by the electronic device 101. Therefore, the electronic device can provide high-quality video call functionality even when the user does not speak during the video call and does not amplify ambient sound.

[0196] return Figure 6 According to the implementation method, the electronic device can continue to operate 605 and mix audio according to weights.

[0197] According to an implementation, the electronic device can store (e.g., record) the mixed audio in a memory (e.g., Figure 1 In the memory 130), it can be synchronized with the video and stored (e.g., recorded) in the memory (e.g., Figure 1 It can be stored in the memory 130, or it can be sent (e.g., via video call) to an external electronic device.

[0198] As described above, realistic recordings or calls can be made by separating and amplifying user voice or ambient sounds using external electronic devices (TWS).

[0199] According to the implementation method, electronic devices (e.g., Figure 1 Electronic device 101) may include a front-facing camera 181 (included in Figure 1 (in the camera module 180), and the rear camera 182 (included in) Figure 1 (in the camera module 180), microphone (e.g., Figure 1 The input module 150), and the communication module including the communication circuit (e.g., Figure 1 The communication module 190), including at least one processor (e.g., processing circuitry), Figure 1 The processor 120) and the memory for storing instructions (e.g., Figure 1 The memory 130), when executed by at least one processor, causes the electronic device to perform the following operations.

[0200] According to the implementation, when the instructions are executed by at least one processor, the electronic device receives a first audio signal from an external electronic device via a communication module.

[0201] According to the implementation, when the instructions are executed by at least one processor, the electronic device acquires a second audio signal via a microphone.

[0202] According to an implementation, when executed by at least one processor, the instructions cause the electronic device to identify whether the first audio signal includes the voice of a user wearing an external electronic device.

[0203] According to an implementation, when executed by at least one processor, the instructions cause the electronic device to extract the user's voice signal included in the first audio signal based on the user's voice being included in the first audio signal.

[0204] According to the implementation, when the instructions are executed by at least one processor, the electronic device causes the second audio signal to be modified based on the user's voice, which is included in the first audio signal.

[0205] According to the implementation, when the instructions are executed by at least one processor, the electronic device identifies whether the front-facing camera or the rear-facing camera is working based on the fact that the user's voice is not included in the first audio signal.

[0206] According to the implementation, when executed by at least one processor, the instructions cause the electronic device to enhance the weight of the ambient audio signal included in the second audio signal or control the weight of the second audio signal based on the operation of the front or rear camera.

[0207] According to the implementation, when the instructions are executed by at least one processor, the electronic device receives information detected by an acceleration sensor of an external electronic device.

[0208] According to an implementation, when executed by at least one processor, the instructions cause the electronic device to identify, based on information, a first portion of the user's voice included in a first audio signal acquired through a microphone of an external electronic device.

[0209] According to an implementation, when executed by at least one processor, the instructions cause the electronic device to extract the user's voice by removing a second portion of the audio signal other than a first portion from the first audio signal.

[0210] According to an implementation, when executed by at least one processor, the instructions cause the electronic device to increase the weight of the user's voice signal included in the second audio signal based on the user's voice being included in the first audio signal.

[0211] According to an implementation, when executed by at least one processor, the instructions cause the electronic device to increase the weight of the user's voice signal included in the second audio signal by merging the user's voice signal in the first audio signal with the second audio signal.

[0212] According to an implementation, when executed by at least one processor, the instructions cause the electronic device to enhance the ambient audio signal included in the second audio signal by merging the first audio signal with the second audio signal, based on the fact that the first audio signal does not include user voice and the rear camera is working.

[0213] According to the implementation, when the instructions are executed by at least one processor, the electronic device reduces or maintains the weight of the second audio signal based on the fact that, when performing a video call function, the first audio signal does not include the user's voice and the front-facing camera is working.

[0214] According to an implementation, when the instructions are executed by at least one processor, the electronic device identifies distortion in a first segment of the first audio signal based on the similarity of the frequency changes of the first audio signal over time.

[0215] According to an implementation, when the instruction is executed by at least one processor, the electronic device recovers the first segment based on segments other than the first segment in the first audio signal.

[0216] According to an implementation, when executed by at least one processor, the instructions cause the electronic device to synchronize the first audio signal and the second audio signal based on the delay caused by receiving the first audio signal.

[0217] According to the implementation method, the external electronic device may include true wireless stereo (TWS).

[0218] According to an embodiment, a method for controlling an electronic device may include: receiving a first audio signal from an external electronic device via the communication module of the electronic device.

[0219] According to an embodiment, a method for controlling an electronic device may include acquiring a second audio signal via the microphone of the electronic device.

[0220] According to an implementation, a method for controlling an electronic device may include identifying whether a first audio signal includes the voice of a user wearing an external electronic device.

[0221] According to an embodiment, a method for controlling an electronic device may include extracting the user's voice signal included in a first audio signal based on the user's voice included in the first audio signal.

[0222] According to an embodiment, a method for controlling an electronic device may include modifying a second audio signal based on a user's voice included in a first audio signal.

[0223] According to an embodiment, a method for controlling an electronic device may include identifying whether a front-facing camera or a rear-facing camera is working based on the user's voice not being included in a first audio signal.

[0224] According to an embodiment, a method for controlling an electronic device may include enhancing an ambient audio signal included in a second audio signal or controlling the weight of the second audio signal based on the operation of a front-facing camera or a rear-facing camera.

[0225] According to an embodiment, the method of controlling the electronic device may further include receiving information detected by an acceleration sensor of an external electronic device.

[0226] According to an implementation, extracting the user's voice signal included in the first audio signal may include identifying, based on information, a first portion of the first audio signal acquired through a microphone of an external electronic device that includes the user's voice.

[0227] According to an implementation, extracting a user's speech signal included in a first audio signal may include extracting the user's speech by removing a second portion of the audio signal other than a first portion from the first audio signal.

[0228] According to an implementation, enhancing the weight of the ambient audio signal or the control of the second audio signal included in the second audio signal may include increasing the weight of the user's voice signal included in the second audio signal based on the user's voice being included in the first audio signal.

[0229] According to an implementation, enhancing the weight of the ambient audio signal or the control of the second audio signal included in the second audio signal may include increasing the weight of the user's voice signal included in the second audio signal by merging the user's voice signal of the first audio signal with the second audio signal.

[0230] According to an implementation, enhancing the ambient audio signal included in the second audio signal or controlling the weight of the second audio signal may include enhancing the ambient audio signal included in the second audio signal by merging the first audio signal with the second audio signal, based on the fact that the first audio signal does not include user voice and the rear camera is working.

[0231] According to the implementation, enhancing the weight of the ambient audio signal included in the second audio signal or controlling the weight of the second audio signal may include reducing or maintaining the weight of the second audio signal based on the fact that, when performing a video call function, the first audio signal does not include the user's voice and the front camera is working.

[0232] According to an embodiment, the method for controlling an electronic device may further include identifying distortion in a first audio signal based on the similarity of frequency changes over time, and recovering the distorted segment in the first audio signal based on segments other than the distorted segment in the first audio signal.

[0233] According to an embodiment, the method of controlling an electronic device may further include synchronizing a first audio signal and a second audio signal based on a delay caused by receiving a first audio signal.

[0234] According to the implementation method, the external electronic device may include true wireless stereo (TWS).

[0235] According to an embodiment, a non-transitory computer-readable recording medium stores one or more programs, the one or more programs storing instructions that cause an electronic device to receive a first audio signal from an external electronic device via the electronic device's communication module.

[0236] According to one implementation, one or more programs may store instructions that cause the electronic device to acquire a second audio signal through the microphone of the electronic device.

[0237] According to one implementation, one or more programs may store instructions that enable an electronic device to identify whether a first audio signal includes the voice of a user wearing an external electronic device.

[0238] According to one implementation, one or more programs may store instructions that cause an electronic device to extract the user's voice signal included in the first audio signal based on the user's voice included in the first audio signal.

[0239] According to one implementation, one or more programs may store instructions that cause an electronic device to include a user's voice in a first audio signal and to modify a second audio signal using the user's voice signal.

[0240] According to one implementation, one or more programs may store instructions that cause the electronic device to identify whether the front camera or the rear camera of the electronic device is working, based on the fact that the user's voice is not included in the first audio signal.

[0241] According to an implementation, one or more programs may store instructions that cause the electronic device to enhance the ambient audio signal included in the second audio signal or control the weight of the second audio signal based on the operation of the front or rear camera.

[0242] According to one implementation, one or more programs may store instructions that cause an electronic device to receive information detected by an acceleration sensor of an external electronic device.

[0243] According to one implementation, one or more programs may store instructions that cause an electronic device to identify, based on information, a first portion of a user's voice included in a first audio signal acquired through a microphone of an external electronic device.

[0244] According to an implementation, one or more programs may store instructions that cause an electronic device to extract a user's voice by removing a second portion of the audio signal other than a first portion from a first audio signal.

[0245] According to one implementation, one or more programs may store instructions that cause an electronic device to receive information detected by an acceleration sensor of an external electronic device.

[0246] According to one implementation, one or more programs may store instructions that cause an electronic device to identify, based on information, a first portion of a user's voice included in a first audio signal acquired through a microphone of an external electronic device.

[0247] According to an implementation, one or more programs may store instructions that cause an electronic device to extract a user's voice by removing a second portion of the audio signal other than a first portion from a first audio signal.

[0248] According to an implementation, one or more programs may store instructions to enable an electronic device to increase the weight of the user's voice signal included in a second audio signal based on the user's voice being included in a first audio signal.

[0249] According to an implementation, one or more programs may store instructions that cause an electronic device to increase the weight of the user's voice signal included in the second audio signal by merging the user's voice signal in the first audio signal with the second audio signal.

[0250] According to one implementation, one or more programs may store instructions that cause an electronic device to enhance an ambient audio signal included in a second audio signal by merging the first audio signal with a second audio signal, based on the premise that a first audio signal does not include user voice and a rear camera is working.

[0251] According to one implementation, one or more programs may store instructions that cause the electronic device to reduce or maintain the weight of a second audio signal based on the fact that, when performing a video call function, the first audio signal does not include the user's voice and the front-facing camera is working.

[0252] According to an implementation, one or more programs may store instructions that enable an electronic device to identify distortion in a first segment of a first audio signal based on the similarity of the frequency changes of the first audio signal over time.

[0253] According to one implementation, one or more programs may store instructions that cause an electronic device to recover the first segment based on segments other than the first segment in the first audio signal.

[0254] According to an implementation, one or more programs may store instructions that cause an electronic device to synchronize a first audio signal and a second audio signal based on a delay caused by receiving a first audio signal.

[0255] According to the implementation method, the external electronic device may include true wireless stereo (TWS).

[0256] The electronic device according to the embodiments can be one of various types of electronic devices. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer equipment, portable multimedia devices, portable medical devices, cameras, wearable electronic devices, or home appliances. According to embodiments of this disclosure, the electronic device is not limited to those described above.

[0257] It should be understood that the various embodiments of this disclosure and the terminology used therein are not intended to limit the technical features set forth herein to a particular embodiment, and include various changes, equivalents, or substitutions to corresponding embodiments. Regarding the description of the drawings, similar reference numerals may be used to refer to similar or related elements. It should be understood that, unless the relevant context clearly indicates otherwise, the singular form of a noun corresponding to an item may include one or more things. As used herein, each of phrases such as “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B, or C,” “at least one of A, B, and C,” and “at least one of A, B, or C” may include all possible combinations of the items listed together in the corresponding phrase. As used herein, terms such as “first” and “second” or “first” and “second” may be used simply to distinguish corresponding components from another component and do not limit the components in other respects (e.g., importance or order). It should be understood that if an element (e.g., the first element) is referred to as being “connected” to another element (e.g., the second element), an element (e.g., the first element) is referred to as being “connected to” (e.g., the second element), an element (e.g., the first element) is referred to as being “connected to” (e.g., the second element), or an element (e.g., the first element) is referred to as being “connected to” (e.g., the second element), regardless of whether the terms “operably” or “communicably” are used, this means that the element can be connected to another element directly (e.g., wired), wirelessly, or via a third element.

[0258] As used herein, the term "module" can include units implemented in hardware, software, or firmware, and is used interchangeably with other terms such as "logic," "logic block," "part," or "circuit." A module can be a single integral component, or its smallest unit or part, adapted to perform one or more functions. For example, depending on the implementation, a module can be implemented as an application-specific integrated circuit (ASIC).

[0259] Embodiments of this disclosure can be implemented as software (e.g., program 140) including one or more instructions stored in a machine-readable storage medium (e.g., internal memory 136 or external memory 138). For example, a processor (e.g., processor 120) of the machine (e.g., electronic device 101) can invoke at least one of the one or more instructions stored in the storage medium and execute it with or without one or more other components under the control of the processor. This allows the machine to operate to perform at least one function according to the invoked at least one instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. The term "non-transitory" simply means that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), but the term does not distinguish between data that is semi-permanently stored in the storage medium and data that is temporarily stored in the storage medium.

[0260] According to various embodiments of this disclosure, the methods can be included in and provided in a computer program product. The computer program product can be traded as a commodity between a seller and a buyer. The computer program product can be distributed in the form of a machine-readable storage medium (e.g., an optical disc read-only memory (CD-ROM)) or via an app store (e.g., the Play Store). TM Online distribution (e.g., downloading or uploading), or direct distribution between two user devices (e.g., smartphones). If distributed online, at least a portion of the computer program product may be temporarily generated or at least temporarily stored in a machine-readable storage medium, such as the memory of the manufacturer's server, the app store's server, or a relay server.

[0261] According to various embodiments, each of the above components (e.g., a module or program) may include a single entity or multiple entities. Some of the multiple entities may be arranged in different components. According to various embodiments, one or more of the above components may be omitted, or one or more other components may be added. Alternatively or additionally, multiple components (e.g., modules or programs) may be integrated into a single component. In this case, according to various embodiments, the integrated component may still perform one or more functions of each of the multiple components in the same or similar manner as before integration by the corresponding one of the multiple components. According to various embodiments, operations performed by a module, program, or other component may be performed sequentially, in parallel, repeatedly, or heuristically, or one or more operations may be performed in a different order or omitted, or one or more other operations may be added.

Claims

1. Electronic devices (101, 310), including: Front camera (181); Rear camera (182); Microphone (150); The communication module (190) includes communication circuitry; At least one processor (120), including processing circuitry; and A memory (130) stores instructions that, when executed by the at least one processor (120), cause the electronic device (101, 310) to: The first audio signal is received from external electronic devices (102, 104, 108, 310, 330) through the communication module (190); The second audio signal is acquired through the microphone (150); Identify whether the first audio signal includes the voice of a user wearing the external electronic device (102, 104, 108, 310, 330); Based on the user's voice (1020) included in the first audio signal, the user's (10) voice signal included in the first audio signal is extracted, and the second audio signal is modified using the user's (10) voice signal; and Based on the fact that the user's (10) voice (1020) is not included in the first audio signal, identify whether the front camera (181) or the rear camera (182) is working, and based on the working of the front camera (181) or the rear camera (182), enhance the environmental audio signal included in the second audio signal or control the weight of the second audio signal.

2. The electronic device according to claim 1, in, When the instructions are executed by the at least one processor (120), the electronic device (101, 310) causes: Receive information detected by the acceleration sensors of the external electronic devices (102, 104, 108, 310, 330); Based on the information, the first part of the user's (1020) voice is identified in the first audio signal acquired by the microphone (150) of the external electronic device (102, 104, 108, 310, 330); as well as The user's (1020) voice is extracted by removing the second part of the audio signal other than the first part from the first audio signal.

3. The electronic device according to claim 1 or 2, in, When the instructions are executed by the at least one processor (120), the electronic device causes: The weight of the user's (1020) voice signal included in the second audio signal is increased based on the user's (10) voice signal (1020) being included in the first audio signal.

4. The electronic device according to claim 3, in, When the instructions are executed by the at least one processor (120), the electronic device (101, 310) causes: By merging the voice signal of the user (10) in the first audio signal with the second audio signal, the weight of the voice signal of the user (10) included in the second audio signal is increased.

5. The electronic device according to any one of claims 1 to 4, in, When the instructions are executed by the at least one processor (120), the electronic device (101, 310) causes: Based on the fact that the first audio signal does not include the user's (1020) voice and the rear camera (182) is working, the ambient audio signal included in the second audio signal is enhanced by merging the first audio signal with the second audio signal.

6. The electronic device according to any one of claims 1 to 4, in, When the instructions are executed by the at least one processor (120), the electronic device (101, 310) causes: Based on the fact that the first audio signal does not include the user's (1020) voice and the front camera (181) is working when performing the video call function, the weight of the second audio signal is reduced or maintained.

7. The electronic device according to any one of claims 1 to 6, in, When the instructions are executed by the at least one processor (120), the electronic device (101, 310) causes: Based on the similarity of the frequency changes of the first audio signal over time, distortion is identified in a first segment of the first audio signal; and The first segment is recovered based on segments other than the first segment in the first audio signal.

8. The electronic device according to any one of claims 1 to 7, in, When the instructions are executed by the at least one processor (120), the electronic device (101, 310) causes: The first audio signal and the second audio signal are synchronized based on the delay caused by receiving the first audio signal.

9. The electronic device according to any one of claims 1 to 8, in, The external electronic devices (102, 104, 108, 310, 330) include true wireless stereo (TWS) functionality.

10. A method for controlling electronic devices (101, 310), the method comprising: The first audio signal (510) is received from external electronic devices (102, 104, 108, 310, 330) via the communication module (190) of the electronic devices (101, 310); The second audio signal is acquired (520) through the microphone (150) of the electronic device; Identify (530) whether the first audio signal includes the voice (1020) of the user (10) wearing the external electronic device (102, 104, 108, 310, 330); The voice (1020) of the user (10) is included in the first audio signal: Extract (540) the user's voice signal included in the first audio signal; and The second audio signal is altered using the user's voice signal. Based on the fact that the user's (10) voice (1020) is not included in the first audio signal: Identify whether the front camera (181) or the rear camera (182) is working; and Based on the operation of the front camera (181) or the rear camera (182), the ambient audio signal included in the second audio signal or the weight of the second audio signal is enhanced (550).

11. The method of claim 10, further comprising: The system receives information detected by the accelerometers of the external electronic devices (102, 104, 108, 310, 330). Extracting the user's voice signal included in the first audio signal includes: Based on the information, a first portion of the user's (1020) voice is identified in the first audio signal acquired through the microphone (150) of the external electronic device (102, 104, 108, 310, 330); and The user's (1020) voice is extracted by removing the second part of the audio signal other than the first part from the first audio signal.

12. The method according to claim 10 or 11, in, Enhancing the ambient audio signal included in the second audio signal or controlling the weights of the second audio signal includes: The weight of the user's (1020) voice signal included in the second audio signal is increased based on the user's (10) voice signal (1020) being included in the first audio signal.

13. The method according to any one of claims 10 to 12, in, Enhancing the ambient audio signal included in the second audio signal or controlling the weights of the second audio signal includes: Based on the fact that the first audio signal does not include the user's (1020) voice and the rear camera (182) is working, the ambient audio signal included in the second audio signal is enhanced by merging the first audio signal with the second audio signal.

14. The method according to any one of claims 10 to 12, in, Enhancing the ambient audio signal included in the second audio signal or controlling the weights of the second audio signal includes: Based on the fact that the first audio signal does not include the user's (1020) voice and the front camera (181) is working when performing the video call function, the weight of the second audio signal is reduced or maintained.

15. A non-transitory computer-readable recording medium storing one or more programs, said one or more programs storing instructions that cause electronic devices (101, 310): The first audio signal is received from external electronic devices (102, 104, 108, 310, 330) through the communication module (190) of the electronic devices (101, 310); The second audio signal is acquired through the microphone (150) of the electronic devices (101, 310); Identify whether the first audio signal includes the voice (1020) of the user (10) wearing the external electronic device (102, 104, 108, 310, 330); Based on the user's (10) voice (1020) included in the first audio signal, the user's (10) voice signal included in the first audio signal is extracted, and the second audio signal is modified using the user's (10) voice signal; and Based on the fact that the user's (1020) voice is not included in the first audio signal, identify whether the front camera (181) or the rear camera (182) of the electronic device (101, 310) is working, and based on the working of the front camera (181) or the rear camera (182), enhance the environmental audio signal included in the second audio signal or control the weight of the second audio signal.

Images