Sound signal processing method and device, electronic equipment and computer readable storage medium

Abstract

The invention provides a sound signal processing method and device, electronic equipment and a computer readable storage medium. The method comprises the following steps: collecting a first sound signal which is currently output, and collecting a second sound signal which is currently input; wherein the first sound signal comprises sound output signals of a plurality of sound channels; performing sound channel combination on the sound output signals of the plurality of sound channels to obtain a single sound channel output signal; performing echo cancellation of the call sound signal on the monaural output signal by taking the call sound signal as a reference to obtain a third sound signal; performing echo cancellation of the first sound signal on the second sound signal by taking the sound output signal of each sound channel as a reference to obtain a fourth sound signal; and carrying out sound mixing on the third sound signal and the fourth sound signal to obtain a sound mixing signal, and sending the sound mixing signal to the target terminal so as to share the media sound signal in the call process. According to the invention, the echo cancellation effect and the audio sharing effect in the call process can be improved.

Description

Technical Field

[0001] This application relates to the field of signal processing, and more particularly to a sound signal processing method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] With the rapid development of technology, communication technology is also constantly innovating and upgrading. Video and voice calls can provide high-definition, smooth picture and sound quality, greatly enriching the user's communication experience. Users' demands for voice calls are also increasing; for example, during a call, they may want others to see a shared local screen while simultaneously allowing others to hear audio played by local media applications. Because local audio creates an echo as it travels through the air during playback, this echo can be present in the sent-to-other voice, severely affecting the audio sharing effect. Therefore, echo cancellation is necessary during audio calls.

[0003] In related technologies, call applications are based on mono audio calls, so echo cancellation for call voice is also implemented using mono echo cancellation methods. However, local audio usually uses multi-channel playback, so echo cancellation will miss some echo signals from some channels, resulting in poor echo cancellation effect and poor audio sharing during the call. Summary of the Invention

[0004] This application provides a sound signal processing method, apparatus, electronic device, and computer-readable storage medium, which can improve the echo cancellation effect and audio sharing effect during a call.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a sound signal processing method, the method comprising:

[0007] Acquire the currently output first sound signal and the currently input second sound signal;

[0008] The first audio signal includes multiple audio output signals, and the audio output signals include: a sub-media audio signal of one channel of the media audio signal having multiple audio channels, and a call audio signal sent by the target terminal.

[0009] The audio output signals from the multiple channels are combined to obtain a mono output signal.

[0010] Using the call audio signal as a reference, echo cancellation of the call audio signal is performed on the mono output signal to obtain a third audio signal;

[0011] Using the sound output signal of each of the aforementioned channels as a reference, the echo of the first sound signal is canceled on the second sound signal to obtain the fourth sound signal;

[0012] The third and fourth audio signals are mixed to obtain a mixed signal, and the mixed signal is sent to the target terminal to share the media audio signal during the call.

[0013] This application provides a sound signal processing device, including:

[0014] The acquisition module is used to acquire the currently output first sound signal and the currently input second sound signal; wherein, the first sound signal includes sound output signals of multiple channels, and the sound output signals include: a sub-media sound signal of one channel of the media sound signal having multiple channels, and a call sound signal sent by the target terminal;

[0015] The merging module is used to merge the audio output signals of the multiple channels to obtain a mono output signal.

[0016] The first cancellation module is used to perform echo cancellation on the mono output signal of the call audio signal with reference to obtain a third audio signal;

[0017] The second cancellation module is used to perform echo cancellation on the second sound signal based on the sound output signal of each of the channels, thereby obtaining the fourth sound signal.

[0018] The mixing module is used to mix the third audio signal and the fourth audio signal to obtain a mixed audio signal, and send the mixed audio signal to the target terminal to share the media audio signal during the call.

[0019] This application provides an electronic device, the electronic device comprising:

[0020] Memory is used to store executable instructions or computer programs.

[0021] The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the sound signal processing method provided in the embodiments of this application.

[0022] This application provides a computer-readable storage medium storing computer-executable instructions or computer programs for implementing the sound signal processing method provided in this application when executed by a processor.

[0023] This application provides a computer program product, including computer-executable instructions or a computer program, which, when executed by a processor, implements the sound signal processing method provided in this application.

[0024] The embodiments of this application have the following beneficial effects:

[0025] In this embodiment, a first audio signal is acquired, and a second audio signal is acquired. The first audio signal includes multiple channel audio output signals, which further include a sub-media audio signal from one channel of a multi-channel media audio signal and a call audio signal sent by the target terminal. Then, the multiple channel audio output signals are combined to obtain a mono output signal. On one hand, using the call audio signal as a reference, echo cancellation is performed on the mono output signal to obtain a third audio signal. On the other hand, using the audio output signals of each channel as references, echo cancellation is performed on the second audio signal to obtain a fourth audio signal. The third and fourth audio signals are then mixed to obtain a mixed signal, which is sent to the target terminal. This allows sharing local media audio (i.e., media audio signals) during a call. Thus, when sharing media audio during a call, multi-channel call voice echo cancellation is achieved, avoiding the omission of echo signals in some channels and improving the echo cancellation and audio sharing effects during the call. Attached Figure Description

[0026] Figure 1 This is a schematic diagram illustrating the application mode of the sound signal processing method provided in the embodiments of this application;

[0027] Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0028] Figure 3A This is a first flowchart illustrating the sound signal processing method provided in an embodiment of this application;

[0029] Figure 3B This is a second flowchart illustrating the sound signal processing method provided in the embodiments of this application;

[0030] Figure 3C This is a schematic diagram of the third process of the sound signal processing method provided in the embodiments of this application;

[0031] Figure 3D This is a schematic diagram of the fourth process of the sound signal processing method provided in the embodiments of this application;

[0032] Figure 3E This is a fifth flowchart illustrating the sound signal processing method provided in the embodiments of this application;

[0033] Figure 3F This is a sixth flowchart illustrating the sound signal processing method provided in this application embodiment;

[0034] Figure 3G This is a schematic diagram of the seventh process of the sound signal processing method provided in the embodiments of this application;

[0035] Figure 4 This is a block diagram of a shared audio stereo echo cancellation embodiment provided in this application.

[0036] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0038] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0039] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0040] It should be noted that the data collection and processing in this application (e.g., acquiring loopback stereo signals through the operating system loopback acquisition interface) should strictly comply with the requirements of relevant national laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0041] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0042] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0043] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0044] 1) Mono: This is an audio format, also known as mono audio. In mono audio, all sound data is combined into a single audio channel for playback.

[0045] 2) Stereo: This is an audio format that transmits audio signals through the left and right channels separately to create a stereo effect.

[0046] 3) Loopback Stereo: This is a stereo audio stream acquired through the operating system's loopback acquisition interface. The application sends the stereo signal to the operating system, which mixes the audio from different applications to obtain the mixed stereo signal. The application can then use the loopback acquisition interface to acquire the mixed stereo signal back.

[0047] 4) Gain Control: Used to adjust the volume level or gain level of the audio signal. Gain control allows you to amplify or reduce the audio signal to achieve the desired volume effect.

[0048] This application provides a sound signal processing method, apparatus, electronic device, storage medium, and program product that can improve echo cancellation and audio sharing effects during calls.

[0049] The following describes exemplary applications of the electronic devices provided in the embodiments of this application. These electronic devices can be implemented as various types of terminals such as laptops, tablets, desktop computers, set-top boxes, smartphones, smart speakers, smartwatches, smart TVs, in-vehicle terminals, wearable devices, virtual reality (VR) devices, and aircraft, or as servers. The following will describe exemplary applications when the device is implemented as a terminal.

[0050] See Figure 1 , Figure 1 This is a schematic diagram illustrating the application mode of the sound signal processing method provided in the embodiments of this application, for example. Figure 1 The system involves server 200, network 300, target terminal 400-1, and local terminal 400-2. Target terminal 400-1 and local terminal 400-2 are connected to server 200 via network 300, which can be a wide area network (WAN), a local area network (LAN), or a combination of both. A calling application can be installed on target terminal 400-1 (corresponding to the first user) and local terminal 400-2 (corresponding to the second user). Figure 1 The call application installed on the local terminal 400-2 can be referred to as call application 410-2, and the call application installed on the target terminal 400-1 can be referred to as call application 410-1. The local terminal 400-2 and the target terminal 400-1 can make calls and transmit call voice data packets based on the call application.

[0051] For example, a communication connection is established between a calling application 410-2 installed on local terminal 400-2 and a calling application 410-1 installed on target terminal 400-1. A second user conducts an audio / video call with the first user corresponding to local terminal 400-2 and target terminal 400-1 based on this communication connection. During the audio / video call, target terminal 400-1 collects the first user's call audio signal and sends a call audio data packet containing the call audio signal to server 200; server 200 forwards the call audio data packet to local terminal 400-2; local terminal 400-2 receives the call audio data packet sent by server 200; it collects the first audio signal to be output and the currently input second audio signal. The first audio signal includes multiple channel audio output signals and includes the media audio signal from local terminal 400-2 and the call audio signal from the call audio data packet sent by target terminal 400-1. The system performs channel merging on multiple audio output signals to obtain a mono output signal. Using the call audio signal as a reference, it performs echo cancellation on the mono output signal to obtain a third audio signal. Using the audio output signals of each channel as references, it performs echo cancellation on the second audio signal to obtain a fourth audio signal. The third and fourth audio signals are then mixed to obtain a mixed signal, which is sent to server 200. Server 200 forwards the mixed signal to target terminal 400-1. Target terminal 400-1 receives the mixed signal from server 200 and plays it. In this way, users can share media audio signals during a call.

[0052] In some embodiments, the embodiments of this application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence (AI), smart transportation, assisted driving, autonomous driving, autonomous driving, online conferencing, instant messaging, etc.

[0053] In some embodiments, the server (e.g., server 200) can be a standalone physical server, a server cluster consisting of multiple physical servers, or a distributed system. The terminal and the server can be connected directly or indirectly via wired or wireless communication, and this application embodiment does not impose any limitations.

[0054] In some embodiments, the sound signal processing method provided in this application can be implemented using cloud technology. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or local area network to realize data computation, storage, processing, and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form a resource pool, be used on demand, and is flexible and convenient. Cloud computing technology will become an important support. The backend services of the technology network system require a large amount of computing and storage resources. As an example, a server (e.g., server 200) can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0055] In some embodiments, the sound signal processing method provided in this application can be implemented using artificial intelligence (AI) technology. AI is a theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities.

[0056] See Figure 2 , Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may be... Figure 1 Local terminal 400-2 in the middle, Figure 2 The illustrated electronic device 500 includes at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. The various components in the electronic device 500 are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 440.

[0057] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0058] User interface 430 includes one or more output devices 431 that enable the presentation of audio call content, including one or more speakers and / or one or more visual displays. User interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0059] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 410.

[0060] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.

[0061] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0062] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, for implementing various basic business functions and handling hardware-based tasks.

[0063] The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including Bluetooth, WiFi, and Universal Serial Bus (USB).

[0064] Presentation module 453 is configured to enable the presentation of audio call information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 431 associated with user interface 430 (e.g., a display screen, a speaker, etc.).

[0065] The input processing module 454 is used to detect and translate one or more user inputs or interactions from one or more input devices 432.

[0066] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2 A sound signal processing device 455 stored in memory 450 is shown. This device can be software in the form of programs and plug-ins, and includes the following software modules: acquisition module 4551, merging module 4552, first cancellation module 4553, second cancellation module 4554, and mixing module 4555. These modules are logically connected and can therefore be arbitrarily combined or further separated according to their implemented functions. The functions of each module will be described below.

[0067] In some embodiments, the terminal or server can implement the sound signal processing method provided in this application by running various computer-executable instructions or computer programs. For example, computer-executable instructions can be microprogram-level commands, machine instructions, or software instructions. Computer programs can be native programs or software modules in an operating system; they can be native applications (APPs), i.e., programs that need to be installed in the operating system to run, such as online meeting APPs or instant messaging APPs; or they can be applets that can be embedded in any APP, i.e., programs that only need to be downloaded to a browser environment to run. In summary, the aforementioned computer-executable instructions can be any form of instruction, and the aforementioned computer programs can be any form of application, module, or plugin.

[0068] The audio signal processing method provided in this application will be described in conjunction with exemplary applications and implementations of the terminal provided in the embodiments of this application.

[0069] The following describes the audio signal processing method provided in the embodiments of this application. As mentioned above, the electronic device implementing the audio signal processing method of the embodiments of this application can be a terminal, a server, or a combination of both. Next, taking an electronic device as an example as a terminal, the audio signal processing method provided in the embodiments of this application will be described. See also... Figure 3A , Figure 3A This is a first flowchart illustrating the sound signal processing method provided in this application embodiment, which will be combined with... Figure 3A The steps shown are explained.

[0070] In step 301, the terminal acquires the currently output first sound signal and the currently input second sound signal.

[0071] Here, the first audio signal includes multiple audio output signals, which include: a sub-media audio signal of one channel in a media audio signal with multiple channels, and a call audio signal sent by the target terminal.

[0072] The terminal (i.e., the user's local terminal) and the target terminal (i.e., the remote terminal) can establish a communication connection based on their respective installed calling applications. The local terminal and the target terminal then conduct audio and video calls based on this connection. During the audio and video call, the local terminal receives the call audio signal (also known as the remote voice signal) sent by the target terminal. After receiving the call audio signal from the target terminal, the local terminal needs to play it back. Since media audio (i.e., media audio signal) is also shared during the call, the local terminal outputs both the target terminal's call audio signal and the media audio signal. Based on this, the local terminal collects the first audio signal, which includes both the target terminal's call audio signal and the media audio signal. Because it is a call, the local terminal also collects the second audio signal currently input locally based on a sound acquisition unit (such as a microphone).

[0073] The media audio signal comprises sub-media audio signals with multiple channels. Since the media audio signal is played across multiple channels, the first audio signal, which includes the call audio signal from the target terminal and the media audio signal, is composed of audio output signals from multiple channels. For example, the multiple channels may include a left channel and a right channel; therefore, the first audio signal can be a stereo signal composed of the audio output signals from the left and right channels. The second audio signal includes the near-end user's voice signal corresponding to the local terminal and the echo signal formed by the first audio signal through the multi-channel audio playback unit. Because the second audio signal acquired by the local terminal contains echo signals, echo cancellation is required. The media audio signal refers to media audio played by applications other than the call application on the local terminal. Media audio can be audio from an audio / video stream played in a video player (e.g., audio from an audio / video stream played in real-time by a web browser's video playback plugin), audio played by a music player, etc.

[0074] In some embodiments, see Figure 3B , Figure 3B This is a schematic diagram of the second process of the sound signal processing method provided in the embodiments of this application. Figure 3A The illustrated step 301 can be achieved through Figure 3B Steps 3011 to 3013 are implemented, and will be explained in detail below.

[0075] In step 3011, the call audio signal and the sub-media audio signal of each channel are mixed to obtain the audio output signal of the channel.

[0076] Here, the first audio signal includes multiple channel audio output signals because the media audio signal includes multiple channel sub-media audio signals. For each channel of the media audio signal, the sub-media audio signal of each channel is mixed with the call audio signal to obtain the audio output signal of each channel. Therefore, the number of channels corresponding to the first audio signal is the same as the number of channels corresponding to the media audio signal. For example, taking a media audio signal that includes two channel sub-media audio signals, the media audio signal has two channel sub-media audio signals, left and right. In the left and right channels, the call audio signal (i.e., the remote voice signal from the remote user) is mixed with the sub-media audio signals of each channel to obtain the audio output signal of each channel.

[0077] In step 3012, the loopback acquisition interface is invoked to acquire the sound output signals of each channel, and the first sound signal of the current output is obtained.

[0078] Here, the loopback acquisition interface can be provided by the audio acquisition engine of the local terminal's operating system. The loopback acquisition interface acquires the sound signal played by the multi-channel audio playback unit. For example, by calling the loopback acquisition interface provided by the local terminal's operating system audio engine, the sound output signals of the two channels after mixing are acquired to obtain the currently output loopback stereo signal, which is the first sound signal.

[0079] For example, taking the first audio signal as an example that includes audio output signals from two channels, the formula for calculating the audio output signal of the left channel of the first audio signal is as follows:

[0080] x1=i+j1

[0081] Where x1 represents the left channel audio output signal, i represents the call audio signal sent by the target terminal, and j1 represents the left channel signal in the media audio signal.

[0082] The formula for calculating the right channel audio output signal of the first audio signal is as follows:

[0083] x2=i+j2

[0084] Where x2 represents the right channel audio output signal, i represents the call audio signal sent by the target terminal, and j2 represents the right channel signal in the media audio signal.

[0085] In step 3013, the sound acquisition unit is invoked to acquire the currently input near-end call sound signal and the echo signal formed when the first sound signal is played, so as to obtain the second sound signal.

[0086] For example, a multi-channel audio playback unit is invoked to play a first sound signal. After playback, the first sound signal propagates through the air to form an echo signal. The multi-channel audio playback unit can be a stereo speaker of the local terminal. The sound acquisition unit can be a microphone of the local terminal. The near-end call sound signal is the voice signal of the near-end user corresponding to the local terminal. The microphone is invoked to acquire the currently input near-end call sound signal and the echo signal formed when the first sound signal is played, thus obtaining the second sound signal.

[0087] In this embodiment of the application, by calling the loopback acquisition interface provided by the operating system audio engine, the audio output signals of multiple channels after mixing can be acquired to obtain the first audio signal currently output, which facilitates subsequent echo cancellation based on the first audio signal.

[0088] Continue to refer to Figure 3A In step 302, the audio output signals of multiple channels are combined to obtain a mono output signal.

[0089] Here, since call applications often conduct audio calls based on mono signals, it is necessary to combine the audio output signals of multiple channels in the first audio signal into a mono output signal. Channel combining refers to using a synthesizer or mixer to convert the audio output signals of multiple channels into a single audio output signal, and the resulting single-channel audio output signal also includes the audio output signal content of multiple channels.

[0090] In some embodiments, see Figure 3C , Figure 3C This is a schematic diagram of the third process of the sound signal processing method provided in the embodiments of this application. Figure 3A The illustrated step 302 can be achieved through Figure 3C Steps 3021 to 3022 are implemented, and will be explained in detail below.

[0091] In step 3021, for each channel, the amplitude of the channel's audio output signal is attenuated to obtain the attenuated audio signal of the channel.

[0092] To combine multiple audio output signals into a single mono output signal for audio calls based on the mono signal, the amplitude attenuation parameter can be determined based on the number of channels, and the amplitude of the multiple audio output signals can be attenuated. For example, taking a first audio signal that includes two audio output channels, the amplitude of the left and right audio output signals in the first audio signal is attenuated separately. Since the first audio signal has two channels, the amplitude of the left and right audio output signals is halved, resulting in attenuated audio signals for both channels.

[0093] In step 3022, the attenuated sound signals of multiple channels are added together to obtain a mono output signal.

[0094] The attenuated audio signals from each channel are summed to obtain a mono output signal containing the call audio signal and media audio signal sent by the target terminal.

[0095] For example, taking the first audio signal as an example that includes audio output signals from two channels, the attenuated audio signals of the left and right channels are added together. The formula for calculating channel merging is as follows:

[0096]

[0097] Where k represents the mono output signal. This indicates the attenuated audio signal in the left channel. This represents the attenuated audio signal of the right channel, and 'i' represents the call audio signal sent by the target terminal. This indicates the attenuation of the media audio signal.

[0098] In this embodiment of the application, by merging the audio output signals of multiple channels in the first audio signal, the audio output signals of multiple channels are converted into a mono output signal, and the mono output signal also includes the audio output signal content of multiple channels, thereby improving the channel utilization rate.

[0099] Continue to refer to Figure 3A In step 303, using the call audio signal as a reference, echo cancellation of the call audio signal is performed on the mono output signal to obtain the third audio signal.

[0100] For example, the third audio signal does not contain the remote voice signal and is approximately the same as the media audio signal. The call audio signal sent by the remote user corresponding to the target terminal (i.e., the remote voice signal) is used as a reference signal. Echo cancellation processing is then performed on the mono output signal obtained by channel combining of the first audio signal to eliminate the remote voice signal in the mono output signal, resulting in a third audio signal that approximates the media audio signal.

[0101] In some embodiments, see Figure 3D , Figure 3D This is a schematic diagram of the fourth process of the sound signal processing method provided in the embodiments of this application. Figure 3A Step 303 shown can be achieved through Figure 3D Steps 3031 to 3032 are implemented, and will be explained in detail below.

[0102] In step 3031, the characteristics of the call audio signal are analyzed to obtain the audio signal characteristics.

[0103] For example, characteristic analysis can include time-domain and frequency-domain analysis of the audio signal, and the resulting audio signal characteristics can include the amplitude and frequency of the call audio signal. Characteristic analysis can be performed as follows: The call audio signal is sampled at a different rate to obtain a target call audio echo signal with the same sampling rate as the mono output signal. The amplitude or frequency of the target call audio echo signal is then analyzed to obtain the characteristics of the target call audio echo signal.

[0104] In step 3032, sound data that matches the characteristics of the sound signal is removed from the mono output signal to obtain a third sound signal.

[0105] Based on the obtained characteristics of the target call echo signal, sound data matching the characteristics of the target call echo signal is eliminated from the mono output signal, thereby obtaining a third sound signal that approximates the media sound signal. For example, taking a first sound signal comprising a two-channel sound output signal as an example, the third sound signal is represented as follows:

[0106]

[0107] Where l represents the third audio signal, j1 represents the left channel signal in the media audio signal, and j2 represents the right channel signal in the media audio signal.

[0108] In this embodiment, the echo cancellation of the call audio signal sent by the target terminal is performed on the mono output signal using the call audio signal as a reference, thereby improving the echo cancellation effect of the far-end voice signal.

[0109] Continue to refer to Figure 3A In step 304, the echo of the first sound signal is canceled on the second sound signal with reference to the sound output signal of each channel to obtain the fourth sound signal.

[0110] For example, the fourth audio signal does not contain the first audio signal, and the fourth audio signal approximates the voice signal of the near-end user corresponding to the local terminal. Using the audio output signals of each channel in the first audio signal as reference signals, the echo signal in the second audio signal captured by the microphone is eliminated to obtain the fourth audio signal that approximates the voice of the near-end user.

[0111] In some embodiments, see Figure 3E , Figure 3E This is a schematic diagram of the fifth process of the sound signal processing method provided in the embodiments of this application. Figure 3A Step 304 shown can be achieved through Figure 3E Steps 3041 to 3043 are implemented, and will be explained in detail below.

[0112] In step 3041, echo signal estimation is performed on the sound output signal of each channel to obtain the echo signal of each channel.

[0113] In some embodiments, for each channel corresponding to the first sound signal, the following processing is performed: obtaining the filter parameters of the channel; and estimating the echo signal of the sound output signal of the channel based on the filter parameters of the channel to obtain the echo signal of the channel.

[0114] The audio output signal of each channel contains the media audio signal and the call audio signal sent by the target terminal. The echo signal corresponding to each channel can be estimated based on the audio output signal of each channel and the filter parameters.

[0115] For example, taking a first audio signal consisting of two channels (left and right), the echo signals corresponding to the two channels can be estimated based on the left and right channel output signals and filter parameters. Let x1 represent the left channel output signal, w1(n) represent the corresponding filter parameters for the left channel, and yw1 represent the left channel echo signal. The formula for calculating the left channel echo signal is as follows:

[0116] yw1(n+m)=x1 T (n+m)w1(n)m=0,…,L-1

[0117] Where n represents the filter length, yw1(n+m) represents the echo signal estimation of the left channel, L represents the frame length for framing the left channel signal, L is an integer greater than or equal to 1 and less than or equal to n, and m is an integer greater than or equal to 0 and less than or equal to L-1.

[0118] For example, x2 can be used to represent the sound output signal of the right channel, w2(n) can be used to represent the filter parameters corresponding to the right channel, and yw2 can be used to represent the echo signal of the right channel. The formula for calculating the echo signal of the right channel is as follows:

[0119] yw2(n+m)=x2 T (n+m)w2(n)m=0,…,L-1

[0120] Where n represents the filter length, yw2(n+m) represents the echo signal estimation of the right channel, L represents the frame length for framing the right channel signal, L is an integer greater than or equal to 1 and less than or equal to n, and m is an integer greater than or equal to 0 and less than or equal to L-1.

[0121] In step 3042, the echo signal characteristics of each channel are analyzed to obtain the echo signal characteristics of each channel.

[0122] In some embodiments, the following processing is performed on each channel: the echo signal of the channel is sampled at a rate conversion to obtain a target echo signal with a target sampling rate of the second audio signal; the target echo signal of the channel is subjected to characteristic analysis to obtain the echo signal characteristics of the channel. Here, the echo signal of each channel is sampled at a rate conversion to obtain a target echo signal with the same sampling rate as the second audio signal, and the target echo signal of each channel is subjected to characteristic analysis.

[0123] For example, characteristic analysis can be performed on the echo signals of each channel in the time domain or frequency domain. When performing time-domain analysis on the echo signal, it is necessary to preprocess the echo signal, such as removing noise and smoothing it, to facilitate subsequent analysis. Time-domain characteristics such as peak value, zero-crossing rate, and impulse response are calculated on the preprocessed echo signal to obtain characteristic information such as the delay time and amplitude of the echo signal.

[0124] For example, when performing frequency domain analysis on echo signals, the Fourier transform is a commonly used mathematical method that converts the signal from the time domain to the frequency domain to facilitate the analysis of the echo signal's frequency characteristics. By calculating the frequency spectrum and power spectrum of the echo signal, we can understand the intensity and distribution of the echo signal at different frequencies. Frequency domain features of the echo signal, such as frequency components, amplitude distribution, and phase characteristics, can be extracted from the frequency domain. These features can provide information about the modulation and interference of the echo signal. In echo signal analysis, various signal processing tools and techniques, such as digital filtering, spectrum analysis, and autocorrelation functions, can also be used to gain a more comprehensive understanding of the echo signal's characteristics.

[0125] In step 3043, sound data that matches the echo signal characteristics of each channel is removed from the second sound signal to obtain the fourth sound signal.

[0126] Based on the echo signal characteristics of each channel, sound data that matches the echo signal characteristics of each channel is removed from the second sound signal, thereby obtaining a fourth sound signal that only contains the voice of the near-end user corresponding to the local terminal.

[0127] For example, taking the first audio signal as an output signal from two channels, the second audio signal is processed by dividing it into frames, and echo cancellation is performed on multiple parts in each frame. The second audio signal can be represented by 'd', and the fourth audio signal by 'e'. The formula for calculating the fourth audio signal is as follows:

[0128] e(n+m)=d(n+m)-yw2(n+m)-yw1(n+m)m=0,…,L-1

[0129] Where n represents the filter length, yw1(n+m) represents the echo signal estimation of the left channel, yw2(n+m) represents the echo signal estimation of the right channel, L represents the frame length for framing the left and right channel signals, L is an integer greater than or equal to 1 and less than or equal to n, and m is an integer greater than or equal to 0 and less than or equal to L-1.

[0130] In some embodiments, for each channel corresponding to the first audio signal, the following processing is performed: based on the fourth audio signal and the audio output signal of the channel, the filter parameters of the channel are updated to obtain the target filter parameters.

[0131] For example, to improve echo cancellation, the filter parameters for each channel can be updated. The target filter parameters are used to cancel the echo of the target second sound signal input at a subsequent target time, referencing the target first sound signal output at a later target time. Based on the fourth sound signal and the sound output signals of each channel, the current filter parameters for each channel can be adjusted. After adjusting the current filter parameters for each channel, the acquisition of the first and second sound signals, as well as the echo cancellation process, can continue, and the filter parameters for each channel can be readjusted to achieve a loop of echo cancellation and filter parameter adjustment.

[0132] For example, taking the first audio signal as an example that includes two audio channels, we can use w1(n) to represent the current filter parameters corresponding to the left channel, w1u(n) to represent the adjusted filter parameters corresponding to the left channel, x1 to represent the audio output signal of the left channel, and e to represent the fourth audio signal. The formula for adjusting the left channel filter parameters is as follows:

[0133]

[0134] Where β represents the preset convergence weight to control the convergence speed of the filter. β can be a preset constant or a dynamically calculated real number. n is the filter length. L is an integer greater than or equal to 1 and less than or equal to n. L represents the frame length for dividing x1 into frames. m is an integer greater than or equal to 0 and less than or equal to L-1.

[0135] For example, based on the updated filter parameters w1u(n) of the left channel, the echo signal yw1u of the left channel is estimated using the following formula:

[0136] yw1u(n+m)=x1 T (n+m)w1u(n)m=0,…,L-1

[0137] The echo signals yw1u from the left channel and yw2 from the right channel are removed from the second audio signal to generate the residual signal es. The formula for calculating the residual signal is as follows:

[0138] es(n+m)=d(n+m)-yw2(n+m)-yw1u(n+m)m=0,…,L-1

[0139] For example, w2(n) can represent the current filter parameters for the right channel, w2u(n) can represent the adjusted filter parameters for the right channel, and x2 can represent the sound output signal for the right channel. The formula for adjusting the right channel filter parameters is as follows:

[0140]

[0141] The adjusted filter parameters for each channel can be used as the current filter parameters for each channel in the subsequent echo cancellation process, and the above iterative calculation can be performed again.

[0142] In this embodiment of the application, by using the first audio signal as a reference, echo cancellation is performed on the audio signals of each channel in the second audio signal, avoiding the omission of the echo signal of one channel, thus improving the echo cancellation effect of the near-end call voice signal.

[0143] Continue to refer to Figure 3A In step 305, the third and fourth audio signals are mixed to obtain a mixed signal, and the mixed signal is sent to the target terminal to share the media audio signal during the call.

[0144] For example, a third audio signal that approximates the media audio signal and a fourth audio signal that approximates the voice of the near-end user are mixed and processed. The resulting mixed signal is then sent to the remote user corresponding to the target terminal, thereby sharing the media audio signal of the local terminal during an audio call.

[0145] In some embodiments, see Figure 3F, Figure 3F This is a schematic diagram of the sixth process of the sound signal processing method provided in the embodiments of this application. Figure 3A Step 305 shown can be achieved through Figure 3F Steps 3051 to 3052 are implemented, and will be explained in detail below.

[0146] In step 3051, the first weight of the third sound signal is obtained, and the second weight of the fourth sound signal is obtained.

[0147] For example, before mixing the third and fourth audio signals, mixing weights can be applied to the third and fourth audio signals respectively. The first weight refers to the mixing weight corresponding to the third audio signal, and the second weight refers to the mixing weight corresponding to the fourth audio signal.

[0148] In step 3052, the third and fourth audio signals are weighted and summed based on the first and second weights to obtain the mixed signal.

[0149] For example, the third and fourth audio signals, each with its own mixing weight, are summed to obtain the mixed signal. We can use w1 to represent the first weight, w2 to represent the second weight, l to represent the third audio signal, e to represent the fourth audio signal, and g to represent the mixed signal. The mixing calculation formula is as follows:

[0150] g = w1 × l + w2 × e

[0151] In some embodiments, the first weight and the second weight are the same; based on this... Figure 3F Step 3051 can be achieved by performing the following steps: obtaining the sound signal threshold; adding the third sound signal and the fourth sound signal to obtain the addition result; when the addition result is less than or equal to the sound signal threshold, determining the first weight of the third sound signal as the first target value; when the addition result is greater than the sound signal threshold, determining the first weight of the third sound signal as the second target value.

[0152] For example, the audio signal threshold is preset based on the value corresponding to the optimal mixing effect. The waveforms of the third and fourth audio signals are sampled. When the sum of the amplitudes of the third and fourth audio signals is less than or equal to the audio signal threshold, the first weight is set to the first target value; when the sum of the amplitudes of the third and fourth audio signals is greater than the audio signal threshold, the first weight is set to the second target value.

[0153] For example, W1 can represent the first target value, W2 can represent the second target value, l can represent the third sound signal, and e can represent the fourth sound signal. The formula for calculating the first weight is as follows:

[0154]

[0155] The threshold THRD1 can be set to 28672 empirically. When the sum of the amplitudes of the third and fourth sound signals is less than or equal to 28672, the first target value W1 can be set to 0.875. When the sum of the amplitudes of the third and fourth sound signals is greater than 28672, the second target value W2 is expressed as follows:

[0156] W2 = (l + e - 28672) * 0.11

[0157] In this embodiment of the application, by applying mixing weights to the third and fourth audio signals and performing a weighted summation to obtain the mixed signal, it is ensured that the volume and quality of the shared audio are not affected during audio calls.

[0158] In some embodiments, see Figure 3G , Figure 3G This is a schematic diagram of the seventh process of the sound signal processing method provided in the embodiments of this application. Figure 3A After step 303 shown, the following steps can also be performed: Figure 3G Steps 3034 to 3036 are explained in detail below.

[0159] In step 3034, the gain control parameters are obtained.

[0160] For example, gain control parameters are used to adjust the volume level or gain level of an audio signal, which can amplify or reduce the audio signal.

[0161] In some embodiments, the third audio signal is analyzed to obtain the signal amplitude of the third audio signal; when the signal amplitude is lower than the target amplitude, a gain control parameter for amplifying the third audio signal is obtained; when the signal amplitude is higher than the target amplitude, a gain control parameter for reducing the third audio signal is obtained.

[0162] For example, the gain control parameter can be a multiplier factor applied to the third audio signal to amplify or reduce it. The target amplitude can be set based on the system's output capability (or maximum output capability) and the desired audio quality; it can be a single value or a range. When the amplitude of the third audio signal is lower than the target amplitude, a multiplier factor is obtained to amplify the third audio signal; when the amplitude of the third audio signal is higher than the target amplitude, a multiplier factor is obtained to reduce the third audio signal.

[0163] In step 3035, the gain of the third sound signal is controlled based on the gain control parameters to obtain a gain sound signal.

[0164] Gain audio signal is the audio signal obtained by amplifying or attenuating a third audio signal using acquired gain control parameters. For example, m can represent the multiplier factor corresponding to the gain control parameter, and p can represent the gain audio signal. Taking a two-channel audio output signal as an example, the calculation formula for gain control is as follows:

[0165]

[0166] Specifically, when the multiplier factor is set to 1.8, it can compensate for the amplitude attenuation of the third audio signal caused by channel merging. The multiplier factor m can also be set according to actual needs to control the volume of the media audio signal in the third audio signal. When the multiplier factor is set to greater than 1, the volume of the media audio signal in the third audio signal can be amplified; when the multiplier factor is set to less than 1, the volume of the media audio signal in the third audio signal can be reduced.

[0167] In step 3036, the gain sound signal is amplitude limited to obtain a limited sound signal.

[0168] For example, in order to prevent the gain audio signal from exceeding the range of values ​​that the quantization precision can represent, it is necessary to limit the gain audio signal by comparing the amplitude value of the gain audio signal with the preset amplitude limit range to obtain the limited gain audio signal.

[0169] In some embodiments, an amplitude limiting range is obtained, and the right and left endpoint values ​​of the amplitude limiting range are determined; when the amplitude of the gain sound signal is greater than the right endpoint value, the amplitude of the gain sound signal is controlled to be the right endpoint value, resulting in a limited sound signal; when the amplitude of the gain sound signal is within the amplitude limiting range, the amplitude of the gain sound signal is controlled to remain unchanged, resulting in a limited sound signal; when the amplitude of the gain sound signal is less than the left endpoint value, the amplitude of the gain sound signal is controlled to be the left endpoint value, resulting in a limited sound signal.

[0170] For example, the amplitude limiting interval is determined based on the number of sampling points of the audio signal. The right endpoint of the amplitude limiting interval is the maximum amplitude of the audio signal controlling the gain, and the left endpoint is the minimum amplitude of the audio signal controlling the gain. MAX can be used to represent the right endpoint of the amplitude limiting interval, MIN to represent the left endpoint, and l1 to represent the amplitude-limited audio signal. The formula for calculating the amplitude limit is as follows:

[0171]

[0172] When using 16-bit integers to represent the sampling points of the audio signal, the amplitude limitation range that the quantization precision can represent is -32768 to 32767. The right endpoint (MAX) of the amplitude limitation range is 32767, and the left endpoint (MIN) is -32768. When the amplitude of the gain audio signal is greater than 32767, the amplitude of the limiter audio signal is 32767. When the amplitude of the gain audio signal is between -32768 and 32767, the amplitude of the limiter audio signal is the same as the amplitude of the gain audio signal; when the amplitude of the gain audio signal is less than -32768, the amplitude of the limiter audio signal is -32768.

[0173] In some embodiments, Figure 3A Step 305 can be achieved by performing the following steps: mixing the limited-amplitude audio signal and the fourth audio signal to obtain a mixed audio signal; encoding and packetizing the mixed audio signal and sending it to the target terminal, thereby sharing the media audio signal played on the local terminal with the remote user.

[0174] In this embodiment of the application, the shared audio quality of multi-channel audio is improved by performing gain control and amplitude limiting on the third audio signal and mixing and encoding it with the fourth audio signal.

[0175] In some embodiments, the audio signal processing method provided in this application can be applied to online conferencing scenarios. In a video conferencing scenario, a first terminal of a first user participating in the video conference and a second terminal of a second user (a user different from the first user in the online conference) establish a communication connection based on an online conferencing app, and share media audio (such as conference-related audio) during the conference call based on the communication connection. During the conference call, the first terminal acquires the currently output first audio signal and the currently input second audio signal. The first audio signal includes multiple audio output signals, including the audio signal of one channel from the shared media audio (i.e., media audio signal) and the call audio signal sent by the second terminal. The second audio signal includes the human voice signal emitted by the first user and the echo signal formed when the first audio signal is played. Then, the multiple audio output signals are combined to obtain a mono output signal. On the one hand, using the call audio signal sent by the second user's terminal as a reference, the mono output signal is subjected to echo cancellation to obtain a third audio signal. On the other hand, using the first audio signal as a reference, the second audio signal is subjected to echo cancellation to obtain a fourth audio signal. The third and fourth audio signals are then mixed to obtain a mixed signal, which is sent to the second user's terminal, enabling the sharing of local conference audio during the video conference. Thus, when sharing conference audio in a video conference, multi-channel voice echo cancellation is achieved, avoiding the loss of echo signals in some channels, thereby improving the echo cancellation effect in video conference calls and the effect of sharing conference audio.

[0176] In some embodiments, the audio signal processing method provided in this application can be applied to an instant messaging scenario, where multiple users can watch a video or listen to audio together based on an established audio / video call. A first terminal of a first user participating in the audio / video call and a second terminal of a second user (a user different from the first user) establish a communication connection based on an instant messaging app, and share media audio (such as watching a movie or listening to music) during the audio / video call based on this communication connection. During an audio / video call, the first terminal acquires the currently output first audio signal and the currently input second audio signal. The first audio signal includes multiple audio output signals, including the audio signal of one channel from the media audio (i.e., media audio signal) shared during the live broadcast and the call audio signal sent by the second terminal. The second audio signal includes the human voice signal emitted by the first user and the echo signal formed when the first audio signal is played. Then, the multiple audio output signals are combined to obtain a mono output signal. On one hand, using the call audio signal sent by the second user's terminal as a reference, the mono output signal is subjected to echo cancellation to obtain a third audio signal. On the other hand, using the first audio signal as a reference, the second audio signal is subjected to echo cancellation to obtain a fourth audio signal. The third and fourth audio signals are then mixed to obtain a mixed signal, which is sent to the second user's terminal, enabling the sharing of media audio during instant communication. Thus, when sharing audio in instant messaging scenarios, multi-channel call voice echo cancellation is achieved, avoiding the omission of echo signals in some channels, thereby improving the echo cancellation effect and audio sharing effect in this instant messaging scenario.

[0177] Applying the above embodiments of this application, a first audio signal currently being output is acquired, and a second audio signal currently being input is acquired. The first audio signal includes audio output signals from multiple channels, including a sub-media audio signal from one channel of a media audio signal with multiple channels and a call audio signal sent by the target terminal. Then, the audio output signals from multiple channels are combined to obtain a mono output signal. On one hand, using the call audio signal as a reference, echo cancellation is performed on the mono output signal to obtain a third audio signal. On the other hand, using the audio output signals from each channel as references, echo cancellation is performed on the second audio signal to obtain a fourth audio signal. The third and fourth audio signals are then mixed to obtain a mixed signal, which is sent to the target terminal. This allows sharing of local media audio (i.e., media audio signals) during a call. Thus, when sharing media audio during a call, multi-channel call voice echo cancellation is achieved, avoiding the omission of echo signals in some channels and improving the echo cancellation effect and audio sharing effect during the call.

[0178] The following will describe an exemplary application of the audio signal processing method provided in this application embodiment in an actual audio and video call scenario using a Voice over Internet Protocol (VOIP) application.

[0179] During calls on digital devices such as computers or mobile phones, both the remote and near-end users have VoIP applications installed on their devices' operating systems. Two or more people can use VoIP applications for audio and video calls. Users often need to share local audio, for example, wanting the other party to see their local computer or mobile phone desktop (i.e., screen sharing) while simultaneously hearing media sounds played by other applications on their local computer or mobile phone. VoIP applications often use mono for calls; however, devices like computers or mobile phones typically have two speakers. The local audio played through these speakers contains multi-channel signals, and the sound travels through the air, creating echoes. This results in echo signals mixed in with the voice sent to the remote user, necessitating echo cancellation processing.

[0180] In related technologies, call applications are based on mono audio calls, so echo cancellation for call voice is also based on mono echo cancellation. However, local audio typically uses a two-channel playback format. When the audio signal content of the two channels is unrelated, mono echo cancellation can only eliminate the echo in one channel, missing the echo signal in the other channel. This results in poor echo cancellation and poor audio sharing during the call.

[0181] Based on this, this application proposes a sound signal processing method to address the problems existing in related technologies, which includes improvements in the following aspects compared to related technologies:

[0182] (1) The near-end application acquires the stereo signal after mixing by the operating system through the loopback acquisition interface provided by the operating system. It can simultaneously acquire the audio signals of two channels in the local media audio. The acquired loopback stereo signal is used as a reference signal. The stereo echo canceller is used to perform echo cancellation processing on the audio signals of the two channels in the acquired near-end call voice, so as to avoid missing the echo signal of one channel and improve the echo cancellation effect of the near-end call voice.

[0183] (2) The voice signal sent by the remote application is used as a reference signal. The echo cancellation of the loopback stereo signal after channel merging is performed by a mono echo canceller. The echo-cancelled near-end voice and loopback stereo signal are mixed and encoded and sent to the remote user to improve the echo cancellation effect of multi-channel shared audio.

[0184] For example, the operating system's Application Programming Interface (API) can be used to call a hardware microphone to capture local audio and send the remote application's voice call to a speaker for playback. For instance, after initiating an audio / video call, the remote user's voice data packets are sent to the near-end application. After unpacking and decoding the voice data packets, a digital audio signal is obtained and sent to the speaker for playback. In hands-free mode, this is represented by speaker output. The digital audio signal before being played to the speaker is the remote user's voice signal. The sound played from the speaker travels through the air and enters the near-end microphone, creating an acoustic echo. In addition to the acoustic echo, the near-end user's voice also enters the microphone. To prevent the echo signal entering the near-end microphone from being transmitted back to the remote application, allowing the remote user to hear their own voice (i.e., echo), echo cancellation, noise suppression, gain adjustment, and other processing are required. Then, the near-end application's voice call is encoded, packetized, and sent to the remote application. Media audio played by other applications, such as audio from live video streams played by web browsers and audio played by music players, will also be transmitted to remote users in real time.

[0185] The stereo echo cancellation process provided in the embodiments of this application will be described in detail below. (Reference) Figure 4 , Figure 4This is a block diagram of a shared audio stereo echo cancellation embodiment provided in this application. The block diagram includes a near-end application 501, an operating system audio engine 502, other audio modules 504, a stereo speaker 505, an echo path 506, and a microphone 507. The near-end application 501 includes a packet decoding module 503, a stereo echo cancellation module 508, a channel merging module 509, a mono echo cancellation module 510, a gain control module 511, a mixing module 512, and an encoding and sending module 513.

[0186] For example, the voice data packets of the remote application corresponding to the target terminal are transmitted to the near-end application 501. The voice data packets are input to the receiving and decoding module 503. After unpacking and decoding the voice data packets, the remote voice signal i is output.

[0187] Near-end application 501 acquires the loopback stereo signal x in real time from the operating system audio engine 502 via the operating system audio loopback acquisition interface. The loopback stereo signal is the multi-channel audio signal sent by the operating system to the stereo speaker 505 before playback. The loopback stereo signal x has a left channel and a right channel. Both the left and right channels contain the far-end speech signal i, and the left channel also includes the left channel signal of media audio played by other applications, while the right channel also includes the right channel signal of media audio.

[0188] For example, the formulas for calculating the left and right channels of a loopback stereo signal are as follows:

[0189] x1=i+j1 (1)

[0190] x2=i+j2 (2)

[0191] Where x1 represents the left channel of the loopback stereo signal x, x2 represents the right channel of the loopback stereo signal x, j1 represents the left channel signal of the media audio, and j2 represents the right channel signal of the media audio.

[0192] For example, the loopback stereo signal x is input to the channel merging module 509, which outputs a mono audio signal k. During the channel merging process, if the mono audio signal obtained by linearly adding the two channels exceeds the upper and lower limits of the values ​​that the quantization precision can represent, nonlinear distortion will be introduced. However, the echo canceller cannot eliminate nonlinear distortion. To ensure that the linear addition of the two channels does not exceed the upper and lower limits of the values ​​that the quantization precision can represent, the audio signals of the left and right channels of the loopback stereo signal x can be attenuated before being added.

[0193] For example, the formula for calculating channel merging is as follows:

[0194]

[0195] in, This indicates the left channel of the surround stereo signal with its amplitude attenuated by half. This indicates the right channel of the surround stereo signal, where the x-amplitude is attenuated by half. This indicates that the audio amplitude of the media is attenuated by half for both channels.

[0196] For example, the far-end speech signal i is used as a reference signal and input together with the audio signal k obtained from channel merging processing into the mono echo cancellation module 510. The far-end speech signal i contained in the audio signal k is eliminated, and the output is a media audio signal h containing only the audio played by other applications. The audio signal h is approximately...

[0197] For example, in the mono echo cancellation module 510, the audio signal k can be decomposed into a first-band audio signal kh and a second-band audio signal kl with the same sampling rate as the far-end speech signal i. The audio signal k has a high sampling rate, such as 48kHz, i.e., a wideband signal with a 24kHz bandwidth. The far-end speech signal i generally has a low sampling rate, such as 16kHz, i.e., an 8kHz bandwidth. The echo signal matching the far-end speech signal i is removed from the second-band audio signal kl to obtain the echo-cancelled audio signal ke. The echo-cancelled audio signal ke and the first-band audio signal kh are then combined into a single band to obtain the media audio signal h containing only the audio played by other applications.

[0198] For example, the audio signal h is input to the gain control module 511, which amplifies or reduces the audio signal h. A multiplier factor m can be applied to the audio signal h, and the calculation formula for amplifying or reducing the audio signal h is as follows:

[0199]

[0200] For example, if the multiplier factor m equals 1.8, the amplitude of the audio signal h is amplified by a factor of 1.8, which can compensate for the amplitude attenuation caused by dividing the left channel signal j1 and the right channel signal j2 of the other media audio by 2. The multiplier factor m can also be set according to actual needs to control the volume of the media audio in the audio signal h. When the multiplier factor is set to greater than 1, the volume of the media audio can be amplified; when the multiplier factor is set to less than 1, the volume of the media audio can be reduced.

[0201] For example, to prevent the multiplied audio signal from exceeding the range of values ​​that the quantization precision can represent, the audio signal p can be clipped by comparing the amplitude value of the signal with the quantization precision value. The clipping calculation formula is as follows:

[0202]

[0203] Where l represents the clipped audio signal, MAX represents the maximum value that the preset quantization precision can represent, and MIN represents the minimum value that the preset quantization precision can represent. When the quantization precision of audio signal p exceeds the maximum value MAX, the quantization precision of audio signal l is set to the maximum value MAX; when the quantization precision of audio signal p is within the specified range, clipping of audio signal p is not required; when the quantization precision of audio signal p is lower than the minimum value MIN, the quantization precision of audio signal l is set to the minimum value MIN.

[0204] For example, when using 16-bit integers to represent the sampling points of an audio signal, the quantization precision can represent values ​​between -32768 and 32767, that is, the maximum value MAX is set to 32767 and the minimum value MIN is set to -32768.

[0205] For example, the sound played through the stereo speaker 505 forms an echo signal after passing through the air echo path 506. The loopback stereo signal x is used as a reference signal and is input together with the near-end call voice signal d collected by the microphone 507 into the stereo echo cancellation module 508. The output is an echo-cancelled audio signal e. The purpose is to eliminate the far-end voice signal i that may be contained in the call voice signal d, as well as other media audio signals j1 and j2, to obtain a mono audio signal e containing only the near-end speaker's voice.

[0206] For example, the acquired near-end call audio signal d contains echo signals from both the left and right channels. The near-end call audio signal d can be segmented, for instance, by dividing it into frames and canceling echoes from multiple parts within each frame. In the stereo echo cancellation module 508, the echo signal yw1 of the left channel is estimated based on the left channel signal x1 and the corresponding filter parameter w1(n). The echo signal yw2 of the right channel is estimated based on the right channel signal x2 and the corresponding filter parameter w2(n). The calculation formulas for yw1 and yw2 are as follows:

[0207] yw1(n+m)=x1 T (n+m)w1(n)m=0,…,L-1 (6)

[0208] yw2(n+m)=x2 T (n+m)w2(n)m=0,…,l-1 (7)

[0209] Where n represents the filter length, yw1(n+m) represents the echo signal estimation of the left channel, yw2(n+m) represents the echo signal estimation of the right channel, L represents the frame length for framing the left and right channel signals, L is an integer greater than or equal to 1 and less than or equal to n, and m is an integer greater than or equal to 0 and less than or equal to L-1.

[0210] For example, echo signals yw1 and yw2 from both channels can be removed from the near-end call speech signal d to generate a first residual signal ef. This first residual signal can be used to adjust the filter parameters of the left channel. The formula for calculating the first residual signal is as follows:

[0211] ef(n+m)=d(n+m)-yw2(n+m)-yw1(n+m)m=0,…,L-1 (8)

[0212] Based on the obtained first residual signal ef and left channel signal x1, the current filter parameters of the left channel are adjusted. The calculation formula for adjusting the left channel filter parameters is as follows:

[0213]

[0214] Where w1u(n) represents the adjusted left channel filter parameters, and β is a preset constant or a dynamically calculated real number used to control the convergence speed of the filter.

[0215] Based on the left channel signal x1 and the adjusted left channel filter parameters w1u(n), the echo signal yw1u of the left channel is estimated using the following formula:

[0216] yw1u(b+m)=x1 T (n+m)w1u(n)m=0,…,L-1 (10)

[0217] For example, the echo signals yw1u and yw2 of the two channels are removed from the near-end call speech signal d to generate a second residual signal es. The second residual signal can be used to adjust the filter parameters of the right channel. The formula for calculating the second residual signal is as follows:

[0218] es(n+m)=d(n+m)-yw2(n+m)-yw1u(n+m)m=0,…,L-1 (11)

[0219] Based on the obtained first residual signal es and right channel signal x2, the current filter parameters of the right channel are adjusted. The calculation formula for adjusting the right channel filter parameters is as follows:

[0220]

[0221] Where w2u(n) represents the adjusted right channel filter parameters.

[0222] Based on the right channel signal x2 and the adjusted right channel filter parameters w2u(n), the right channel echo signal yw2u is estimated using the following formula:

[0223] yw2u(n+m)=x2 T (n+m)w2u(n)m=0,…,L-1 (13)

[0224] The echo signals yw1u and yw2u of the two channels are removed from the near-end call speech signal d to obtain the echo-cancelled audio signal e. The calculation formula is as follows:

[0225] e(n+m)=d(n+m)-yw2u(n+m)-yw1u(n+m)m=0,…,L-1 (14)

[0226] After adjusting the filter parameters of both the left and right channels once, the resulting echo-cancelled audio signal e can be used as the initial filter parameters in subsequent echo cancellation processing, and the above iterative calculation can be performed again.

[0227] For example, in order to reduce the amount of computation, the echo cancellation method in the stereo echo cancellation module is to adjust the filter parameters in the time domain and use the adjusted filter to cancel the echo. Considering the real-time and high-precision requirements of the call voice signal, an adaptive filter based on the combination of frequency and time domains based on fast Fourier transform can also be used for echo cancellation to balance the echo cancellation effect of the call voice and the low latency requirement.

[0228] For example, audio signal l and audio signal e are input into the mixing module 512, and the output is the mixed audio signal. To avoid the sum of audio signal l and audio signal e exceeding the numerical range, and to maintain the volume and sound quality of the call voice signal as much as possible, the mixing calculation formula is as follows:

[0229] g = w × l + w × e (15)

[0230] Where g represents the audio signal after mixing, and w represents the weights applied to audio signal l and audio signal e respectively. The formula for calculating the weights is as follows:

[0231]

[0232] Where W1 represents a pre-set first weighting coefficient, W2 represents a pre-set second weighting coefficient, and THRD1 represents a pre-set threshold. The waveforms of audio signals l and e are sampled. When the sum of the amplitudes of audio signals l and e is less than or equal to the pre-set threshold THRD1, the weights applied to audio signals l and e are set to the first weighting coefficient W1; otherwise, the weights applied to audio signals l and e are set to the second weighting coefficient W2.

[0233] For example, based on the optimal value corresponding to the mixing test, the threshold THRD1 can be preset to 28672. When the sum of audio signal l and audio signal e is less than or equal to 28672, the first weighting coefficient W1 can be set to 0.875; when the sum of audio signal l and audio signal e is greater than 28672, the calculation formula for the second weighting coefficient W2 is as follows:

[0234] W2 = (l + e - 28672) * 0.11 (17)

[0235] For example, the audio signal g is input to the encoding and packet sending module 513, which encodes and packages the audio signal g and sends it to the remote application, thereby sharing the media audio played locally to the remote user.

[0236] In the aforementioned shared audio multichannel echo cancellation application scenario, by using the acquired loopback stereo signal as a reference signal, a stereo echo canceller is used to perform echo cancellation processing on the audio signals of two channels in the acquired near-end call speech, thereby improving the echo cancellation effect of the near-end call speech. The loopback stereo signal is then processed by channel merging to obtain a mono audio signal. Using the call speech signal sent by the remote application as a reference signal, a mono audio signal is processed by a mono echo canceller to perform echo cancellation processing on the mono audio signal. The echo-cancelled near-end call speech and the gain-controlled mono audio signal are then mixed and encoded before being sent to the remote user, further improving the shared audio echo cancellation effect of the multichannel audio.

[0237] The following continues to describe the exemplary structure of the sound signal processing device 455 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2As shown, the software modules stored in the sound signal processing device 455 in the memory 450 may include: an acquisition module 4551, used to acquire a first sound signal to be output and to acquire a currently input second sound signal; wherein the first sound signal includes multiple channel sound output signals, the sound output signals including: a sub-media sound signal of one channel of a media sound signal with multiple channels, and a call sound signal sent by the target terminal; a merging module 4552, used to merge the multiple channel sound output signals to obtain a mono output signal; a first cancellation module 4553, used to perform echo cancellation of the mono output signal with reference to the call sound signal to obtain a third sound signal; a second cancellation module 4554, used to perform echo cancellation of the first sound signal with reference to the sound output signals of each channel to obtain a fourth sound signal; and a mixing module 4555, used to mix the third sound signal and the fourth sound signal to obtain a mixed signal, and send the mixed signal to the target terminal to share the media sound signal during the call.

[0238] In some embodiments, the acquisition module 4551 is further configured to: mix the call audio signal and the sub-media audio signal of each channel to obtain the audio output signal of the channel; call the loopback acquisition interface to acquire the audio output signal of each channel to obtain the first audio signal currently output; and call the audio acquisition unit to acquire the currently input near-end call audio signal and the echo signal formed when the first audio signal is played to obtain the second audio signal.

[0239] In some embodiments, the merging module 4552 is further configured to attenuate the amplitude of the sound output signal of each channel to obtain the attenuated sound signal of the channel; and to add the attenuated sound signals of multiple channels to obtain a mono output signal.

[0240] In some embodiments, the first elimination module 4553 is further configured to perform characteristic analysis on the call audio signal to obtain audio signal characteristics; and eliminate audio data that matches the audio signal characteristics from the mono output signal to obtain a third audio signal.

[0241] In some embodiments, the second cancellation module 4554 is further configured to perform echo signal estimation on the sound output signal of each channel to obtain the echo signal of each channel; perform characteristic analysis on the echo signal of each channel to obtain the echo signal characteristics of each channel; and eliminate sound data that matches the echo signal characteristics of each channel from the second sound signal to obtain the fourth sound signal.

[0242] In some embodiments, the second cancellation module 4554 is further configured to perform the following processing for each channel: obtain the filter parameters of the channel; and estimate the echo signal of the sound output signal of the channel based on the filter parameters of the channel to obtain the echo signal of the channel.

[0243] In some embodiments, the second elimination module 4554 is further configured to eliminate sound data matching the echo signal characteristics of each channel from the second sound signal to obtain the fourth sound signal, and then perform the following processing for each channel: based on the fourth sound signal and the sound output signal of the channel, update the filter parameters of the channel to obtain the target filter parameters.

[0244] In some embodiments, the second cancellation module 4554 is further configured to perform the following processing for each channel: perform sampling rate conversion on the echo signal of the channel to obtain a target echo signal with a target sampling rate of the second sound signal; and perform characteristic analysis on the target echo signal of the channel to obtain the echo signal characteristics of the channel.

[0245] In some embodiments, the first cancellation module 4553 is further configured to, with reference to the call audio signal, perform echo cancellation on the mono output signal to obtain a third audio signal, then obtain gain control parameters; based on the gain control parameters, perform gain control on the third audio signal to obtain a gain audio signal; and limit the amplitude of the gain audio signal to obtain a limited audio signal.

[0246] In some embodiments, the mixing module 4555 is further configured to mix the limited audio signal and the fourth audio signal to obtain a mixed signal.

[0247] In some embodiments, the first cancellation module 4553 is further configured to analyze the third sound signal to obtain the signal amplitude of the third sound signal; when the signal amplitude is lower than the target amplitude, to obtain a gain control parameter for amplifying the third sound signal; and when the signal amplitude is higher than the target amplitude, to obtain a gain control parameter for reducing the third sound signal.

[0248] In some embodiments, the first cancellation module 4553 is further configured to acquire an amplitude limiting range and determine the right endpoint value and left endpoint value of the amplitude limiting range; when the amplitude of the gain sound signal is greater than the right endpoint value, the amplitude of the gain sound signal is controlled to be the right endpoint value to obtain a limited sound signal; when the amplitude of the gain sound signal is within the amplitude limiting range, the amplitude of the gain sound signal is controlled to remain unchanged to obtain a limited sound signal; when the amplitude of the gain sound signal is less than the left endpoint value, the amplitude of the gain sound signal is controlled to be the left endpoint value to obtain a limited sound signal.

[0249] In some embodiments, the mixing module 4555 is further configured to obtain a first weight of the third audio signal and a second weight of the fourth audio signal; and to perform a weighted summation of the third audio signal and the fourth audio signal based on the first weight and the second weight to obtain a mixed signal.

[0250] In some embodiments, the first weight and the second weight are the same. The mixing module 4555 is further configured to obtain a sound signal threshold; add the third sound signal and the fourth sound signal to obtain an addition result; when the addition result is less than or equal to the sound signal threshold, determine the first weight of the third sound signal as a first target value; when the addition result is greater than the sound signal threshold, determine the first weight of the third sound signal as a second target value.

[0251] This application provides a computer program product, which includes computer-executable instructions or a computer program stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions or computer program from the computer-readable storage medium and executes the computer-executable instructions or computer program, causing the electronic device to perform the sound signal processing method described in this application.

[0252] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the sound signal processing method provided in this application. For example, ... Figure 3A The audio signal processing method is shown.

[0253] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0254] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0255] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0256] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located in one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0257] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A sound signal processing method, characterized in that, The method includes: Acquire the currently output first sound signal and the currently input second sound signal; The first audio signal includes multiple audio output signals, and the audio output signals include: a sub-media audio signal of one channel of the media audio signal having multiple audio channels, and a call audio signal sent by the target terminal. The audio output signals from the multiple channels are combined to obtain a mono output signal. Using the call audio signal as a reference, echo cancellation of the call audio signal is performed on the mono output signal to obtain a third audio signal; Using the sound output signal of each of the aforementioned channels as a reference, the echo of the first sound signal is canceled on the second sound signal to obtain the fourth sound signal; The third and fourth audio signals are mixed to obtain a mixed signal, and the mixed signal is sent to the target terminal to share the media audio signal during the call.

2. The method according to claim 1, characterized in that, The acquisition of the currently output first sound signal includes: For each of the sub-media audio signals of the audio channel, the call audio signal and the sub-media audio signal of the audio channel are mixed to obtain the audio output signal of the audio channel; Call the loopback acquisition interface to acquire the sound output signals of each of the aforementioned channels, and obtain the currently output first sound signal; The acquisition of the currently input second sound signal includes: The sound acquisition unit is invoked to acquire the currently input near-end call sound signal and the echo signal formed when the first sound signal is played, thereby obtaining the second sound signal.

3. The method according to claim 1, characterized in that, The step of combining the audio output signals from the multiple channels to obtain a mono output signal includes: For each of the aforementioned audio channels, the amplitude of the audio output signal of that channel is attenuated to obtain the attenuated audio signal of that channel; The attenuated sound signals from the multiple channels are added together to obtain the mono output signal.

4. The method according to claim 1, characterized in that, The step of using the call audio signal as a reference to perform echo cancellation on the mono output signal to obtain a third audio signal includes: The characteristics of the call audio signal are obtained by performing characteristic analysis. The third sound signal is obtained by eliminating sound data that matches the characteristics of the sound signal from the mono output signal.

5. The method according to claim 1, characterized in that, The step of using the sound output signal of each of the aforementioned channels as a reference to perform echo cancellation on the second sound signal to obtain the fourth sound signal includes: Echo signal estimation is performed on the sound output signal of each channel to obtain the echo signal of each channel; The echo signals of each of the aforementioned channels are analyzed to obtain the echo signal characteristics of each of the aforementioned channels; The fourth sound signal is obtained by removing sound data from the second sound signal that matches the echo signal characteristics of each of the channels.

6. The method according to claim 5, characterized in that, The step of estimating the echo signal of the sound output signal of each of the aforementioned channels to obtain the echo signal of each of the aforementioned channels includes: For each of the aforementioned audio channels, the following processing is performed: Obtain the filter parameters of the audio channel; Based on the filter parameters of the audio channel, the echo signal of the audio channel is estimated to obtain the echo signal of the audio channel. After removing sound data matching the echo signal characteristics of each of the channels from the second sound signal to obtain the fourth sound signal, the method further includes: For each of the aforementioned audio channels, the following processing is performed: Based on the fourth sound signal and the sound output signal of the channel, the filter parameters of the channel are updated to obtain the target filter parameters.

7. The method according to claim 5, characterized in that, The process of performing characteristic analysis on the echo signals of each of the aforementioned channels to obtain the echo signal characteristics of each of the aforementioned channels includes: The following processing is performed on each of the aforementioned audio channels: The echo signal of the sound channel is sampled and converted to obtain a target echo signal with the target sampling rate of the second sound signal; The characteristics of the echo signal of the target channel are obtained by performing characteristic analysis on the target echo signal of the channel.

8. The method according to claim 1, characterized in that, After using the call audio signal as a reference to perform echo cancellation on the mono output signal to obtain a third audio signal, the method further includes: Obtain the gain control parameters; Based on the gain control parameters, the gain of the third sound signal is controlled to obtain a gained sound signal; The amplitude of the gain sound signal is limited to obtain a limited-amplitude sound signal; The step of mixing the third audio signal and the fourth audio signal to obtain a mixed signal includes: mixing the limited-amplitude audio signal and the fourth audio signal to obtain a mixed signal.

9. The method according to claim 8, characterized in that, The acquisition of gain control parameters includes: The signal amplitude of the third sound signal is obtained by analyzing the third sound signal. When the signal amplitude is lower than the target amplitude, obtain the gain control parameters for amplifying the third sound signal; When the signal amplitude is higher than the target amplitude, a gain control parameter for reducing the third sound signal is obtained.

10. The method according to claim 8, characterized in that, The amplitude limiting of the gain audio signal to obtain an amplitude-limited audio signal includes: Obtain the amplitude limit interval, and determine the right and left endpoint values ​​of the amplitude limit interval; When the amplitude of the gain sound signal is greater than the right endpoint value, the amplitude of the gain sound signal is controlled to be the right endpoint value to obtain the amplitude-limited sound signal; When the amplitude of the gain sound signal is within the amplitude limiting range, the amplitude of the gain sound signal is kept constant to obtain the amplitude-limited sound signal; When the amplitude of the gain sound signal is less than the left endpoint value, the amplitude of the gain sound signal is controlled to the left endpoint value to obtain the amplitude-limited sound signal.

11. The method according to claim 1, characterized in that, The step of mixing the third audio signal and the fourth audio signal to obtain a mixed signal includes: Obtain the first weight of the third sound signal and the second weight of the fourth sound signal; Based on the first weight and the second weight, the third audio signal and the fourth audio signal are weighted and summed to obtain the mixed signal.

12. The method according to claim 11, characterized in that, The first weight and the second weight are the same. Obtaining the first weight of the third sound signal includes: Obtain the sound signal threshold; The third sound signal and the fourth sound signal are added together to obtain the sum; When the summation result is less than or equal to the sound signal threshold, the first weight of the third sound signal is determined as the first target value; When the summation result is greater than the sound signal threshold, the first weight of the third sound signal is determined as the second target value.

13. A sound signal processing device, characterized in that, The device includes: The acquisition module is used to acquire the currently output first sound signal and the currently input second sound signal; wherein, the first sound signal includes sound output signals of multiple channels, and the sound output signals include: a sub-media sound signal of one channel of the media sound signal having multiple channels, and a call sound signal sent by the target terminal; The merging module is used to merge the audio output signals of the multiple channels to obtain a mono output signal. The first cancellation module is used to perform echo cancellation on the mono output signal of the call audio signal with reference to obtain a third audio signal; The second cancellation module is used to perform echo cancellation on the second sound signal based on the sound output signal of each of the channels, thereby obtaining the fourth sound signal. The mixing module is used to mix the third audio signal and the fourth audio signal to obtain a mixed audio signal, and send the mixed audio signal to the target terminal to share the media audio signal during the call.

14. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the sound signal processing method according to any one of claims 1 to 12.

15. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, the sound signal processing method according to any one of claims 1 to 12 is implemented.

Images