Audio separation method and device, electronic equipment and computer readable storage medium

Abstract

The application provides an audio separation method and device, electronic equipment and computer readable storage medium. The method separates the remote audio from the mixed audio to obtain the target audio by the mixed audio of the target device collected, the remote audio sent by the remote device, the weight of the remote audio in the mixed audio, the target playing delay of the remote audio calculated according to the preset delay calculation condition, and the first target audio calculation condition. The method separates the remote audio from the mixed audio according to the first target audio calculation condition, the playing of the remote audio and the weight of participating in the mixed audio by the calculation mode, the whole audio separation process does not need to increase the additional virtual sound card, and the integrity of the target audio after separation is better guaranteed, and the audio separation effect is improved.

Description

Technical Field

[0001] This application relates to the field of audio processing technology, and in particular to an audio separation method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] In real-time conferencing scenarios, because the audio transmitted from the conferencing equipment to the remote device is a mixed audio sampled from the sound card, the remote device is prone to echo during audio interaction with the conferencing equipment.

[0003] To address the aforementioned echo phenomenon, current technologies primarily employ two methods for echo cancellation: First, a virtual sound card is used in the conferencing equipment. Upon entering the meeting, the virtual sound card is set as the default sound card, causing the conferencing equipment's system audio output to be sent to the virtual sound card. The virtual sound card then forwards this audio to the real sound card for playback through the conferencing equipment, simultaneously sending it to the remote device so that the remote device only receives the system audio from the conferencing equipment. However, this method requires adding a virtual sound card and is prone to playback abnormalities when switching sound cards. Second, the conferencing equipment uses the remote audio as the reference audio for echo cancellation. An echo cancellation algorithm is applied to the acquired mixed audio to obtain the conferencing equipment's system audio. However, the echo cancellation algorithm is less effective in two-way scenarios (e.g., the conferencing equipment is speaking while the remote device is playing its own system audio), resulting in the conferencing equipment being unable to clearly hear the remote system audio sent from the remote device.

[0004] Therefore, the current method has technical problems such as requiring an additional virtual sound card and poor audio separation effect, and needs to be improved. Summary of the Invention

[0005] This application provides an audio separation method, apparatus, electronic device, and computer-readable storage medium to alleviate the technical problems of current methods, which require additional virtual sound cards and have poor audio separation effects.

[0006] To address the aforementioned technical problems, this application provides the following technical solution:

[0007] This application provides an audio separation method, including:

[0008] The system collects mixed audio from the target device and remote audio transmitted from the remote device. The mixed audio includes the system audio of the target device and the remote audio received from the remote device.

[0009] Obtain the weight of the far-end audio in the mixed audio;

[0010] The target playback delay of the remote audio is calculated based on the preset delay calculation conditions.

[0011] Based on the first target audio calculation conditions, the mixed audio, the far-end audio, the target playback delay, and the weight, the far-end audio is separated from the mixed audio to obtain the target audio.

[0012] Accordingly, this application also provides an audio separation device, comprising:

[0013] An audio acquisition module is used to acquire mixed audio from a target device and remote audio sent by a remote device. The mixed audio includes the system audio of the target device and the remote audio received from the remote device.

[0014] The weight acquisition module is used to acquire the weight of the far-end audio in the mixed audio;

[0015] The delay calculation module is used to calculate the target playback delay of the far-end audio according to preset delay calculation conditions.

[0016] The separation processing module is used to separate the far-end audio from the mixed audio to obtain the target audio based on the first target audio calculation conditions, the mixed audio, the far-end audio, the target playback delay, and the weight.

[0017] Meanwhile, this application provides an electronic device including a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to run the computer program in the memory to perform the steps in the above-described audio separation method.

[0018] In addition, this application also provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to perform the steps in the above-described audio separation method.

[0019] Beneficial Effects: This application provides an audio separation method, apparatus, electronic device, and computer-readable storage medium. Specifically, the method first acquires mixed audio from a target device and remote audio transmitted from a remote device. The mixed audio includes the target device's system audio and the received remote audio from the remote device. Then, it obtains the weight of the remote audio in the mixed audio. Next, it calculates the target playback delay of the remote audio based on preset delay calculation conditions. Finally, based on the first target audio calculation conditions, the mixed audio, the remote audio, the target playback delay, and the weights, it separates the remote audio from the mixed audio to obtain the target audio. This method separates the remote audio from the mixed audio through calculation based on the first target audio calculation conditions, the playback of the remote audio, and the weights of the participating audio. The entire audio separation process only requires calculation to separate the remote audio, without the need for an additional virtual sound card. Furthermore, the first target audio calculation conditions effectively ensure the integrity of the separated target audio, improving the audio separation effect and avoiding the problem of poor audio separation in dual-talk scenarios caused by echo cancellation algorithms in current technologies. Attached Figure Description

[0020] The technical solution and other beneficial effects of this application will become apparent from the following detailed description of specific embodiments in conjunction with the accompanying drawings.

[0021] Figure 1 This is a system architecture diagram of the audio separation system provided in the embodiments of this application.

[0022] Figure 2 This is a flowchart illustrating the audio separation method provided in the embodiments of this application.

[0023] Figure 3 This is a schematic diagram of an audio transmission scenario without audio separation, provided in an embodiment of this application.

[0024] Figure 4 This is a schematic diagram of an audio transmission scenario when using a virtual sound card for audio separation, as provided in an embodiment of this application.

[0025] Figure 5 This is a schematic diagram of an audio transmission scenario when using an algorithm for audio separation, as provided in an embodiment of this application.

[0026] Figure 6 This is a schematic diagram of the audio separation device provided in the embodiments of this application.

[0027] Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0029] The terms "comprising" and "having" and any variations thereof in the specification and claims of this application are intended to cover non-exclusive inclusion; the division of modules appearing in this application is merely a logical division, and in actual application, there may be other division methods, such as multiple modules being combined into or integrated into another system, or some features being ignored or not executed.

[0030] In this application, the target device refers to a device terminal that can be used for real-time conferencing.

[0031] In this application, a remote device refers to a device terminal that can be used to conduct audio and video calls with a target device. It should be noted that the remote device can be on a different network than the target device, communicating via a remote network connection.

[0032] In this application, system audio refers to local / network audio (e.g., audio played by a local music player) that is not acquired by the device's built-in audio acquisition unit (e.g., a microphone).

[0033] In this application, remote audio refers to audio emitted by a remote device (e.g., external audio acquired through the audio acquisition unit of the remote device; or, for example, local / network audio acquired through the remote device).

[0034] In this application, the preset delay calculation conditions refer to the pre-defined calculation method used to calculate the target playback delay of the far-end audio. Specifically, the preset delay calculation conditions include preset time intervals, relevant feature value calculation conditions, methods for determining the target mixed audio frame based on the magnitude of relevant feature values, methods for calculating the initial playback delay, and methods for determining the target playback delay, etc.

[0035] In this application, the first target audio calculation condition refers to the method of calculating the target audio based on the mixed audio, the far-end audio, the target playback delay, and the weights.

[0036] This application provides an audio separation method, apparatus, electronic device, and computer-readable storage medium.

[0037] Please see Figure 1 , Figure 1 This is a schematic diagram of the system architecture of the audio separation system provided in this application, as shown below. Figure 1 As shown, the audio separation system includes at least a target device 101 and a remote device 102, wherein:

[0038] A communication link is provided between the target device 101 and the remote device 102 to enable information exchange. The type of communication link may include a wireless communication link or a fiber optic cable, etc., and this application does not impose any restrictions.

[0039] The target device 101 can be a device terminal with real-time conferencing capabilities; for example, a conference board, a laptop, a tablet, and a smartphone.

[0040] The remote device 102 can be a device terminal with real-time conferencing capabilities, such as a laptop, tablet, or smartphone, used for audio and video calls with the target device. It should be noted that the remote device and the target device can be on different networks, communicating via a remote network connection.

[0041] This application proposes an audio separation system, which includes a target device and a remote device. Specifically, the remote device sends remote audio to the target device. The remote audio can be speech audio collected by the acquisition unit or local / network audio played by the remote device's local player. Then, the target device collects the system audio of the target device and combines it with the remote audio sent by the remote device to form a mixed audio. Next, the target device obtains the weight of the remote audio in the mixed audio and calculates the target playback delay of the remote audio according to preset delay calculation conditions. Finally, based on the first target audio calculation conditions, the mixed audio, the remote audio, the target playback delay, and the weights, the remote audio is separated from the mixed audio to obtain the target audio.

[0042] In the aforementioned audio separation process, the target device separates the far-end audio from the mixed audio by calculating based on the first target audio calculation conditions, the playback of the far-end audio, and the weights of the audio involved in the mixing. The entire separation process relies solely on background calculation and corresponding processing, without the need for additional virtual sound cards. Furthermore, by using the first target audio calculation conditions, the integrity of the separated target audio is well guaranteed, improving the audio separation effect and avoiding the problem of poor audio separation effect in dual-talk scenarios caused by the use of echo cancellation algorithms in current technologies.

[0043] It should be noted that, Figure 1The system architecture diagram shown is merely an example. The servers, terminals, devices, and scenarios described in this application are for the purpose of more clearly illustrating the technical solutions of this application and do not constitute a limitation on the technical solutions provided in this application. As those skilled in the art will know, with the evolution of systems and the emergence of new business scenarios, the technical solutions provided in this application are also applicable to similar technical problems. Detailed descriptions are provided below. It should be noted that the order of description of the following embodiments is not intended to limit the preferred order of the embodiments.

[0044] Based on the system architecture of the audio separation system described above, the audio separation method in this application will be described in detail below. Please refer to [link / reference]. Figure 2 As shown, Figure 2 This is a schematic flowchart of an audio separation method provided in an embodiment of this application, which is applied to a target device. The method includes at least the following steps:

[0045] S201: Acquire the mixed audio from the target device and the remote audio sent by the remote device. The mixed audio includes the system audio from the target device and the remote audio received from the remote device.

[0046] like Figure 3 As shown, Figure 3 This is a schematic diagram illustrating an audio transmission scenario without audio separation, as provided in an embodiment of this application. Assume that target device B is playing system audio (e.g., music played by a local music player) and remote audio from remote device A. In this case, the audio transmitted from target device B to remote device A is a mixture of audio captured from the actual sound card (including system audio and remote audio from the remote device). If target device B directly sends the mixture to remote device A, remote device A will hear the audio it sent to target device B being transmitted back, resulting in an echo.

[0047] To resolve this echo phenomenon, such as Figure 4 As shown, Figure 4This is a schematic diagram illustrating the audio transmission scenario using a virtual sound card for audio separation, as provided in this application. Specifically, a virtual sound card is added to the target device B. The virtual sound card is not connected to a speaker; if the program outputs audio to the virtual sound card, no sound will be emitted. The real sound card, however, is connected to a speaker, meaning the audio emitted by the speaker is the audio output from the real sound card. In practice, when entering a meeting, the user sets the virtual sound card as the default sound card. Assuming the target device B outputs system audio (e.g., music played from a local music player) to the virtual sound card and outputs remote audio from the remote device A to the real sound card, the software captures the system audio from the virtual sound card and forwards it to the real sound card. At this point, the audio on the real sound card is a mixed audio (including system audio and remote audio from the remote device). This mixed audio is then played through the real sound card. Simultaneously, the target device B forwards its system audio to the remote device A via software. In this way, the audio sent from the target device B to the remote device A will not contain the remote audio sent from the remote device A to the target device B. However, this solution is prone to playback abnormalities when switching sound cards. For example, if the user manually switches the default sound card to the virtual sound card in a non-interactive state, the speakers will not play system audio.

[0048] Therefore, as Figure 5 , Figure 5 This is a schematic diagram illustrating an audio transmission scenario using an algorithm for audio separation, as provided in an embodiment of this application. In current technology, echo cancellation (AEC) is generally used for audio separation. Specifically, assuming target device B is playing system audio (e.g., a local music player playing music) and remote audio from remote device A, the system audio from B and the remote audio from remote device A are forwarded to the physical sound card. At this point, the audio on the physical sound card is mixed audio. The software then uses the received remote audio from remote device A as a reference audio and performs the echo cancellation (AEC) algorithm on the mixed audio to obtain the system audio from B after removing the remote audio. Finally, the processed system audio from B is sent to remote device A. However, the echo cancellation (AEC) algorithm is less effective in two-way communication scenarios (e.g., remote device A is speaking while target device B is playing music), making it difficult for remote device A to hear the system audio sent by target device B.

[0049] This application provides a novel audio separation method, such as Figure 5As shown, the target device B first acquires mixed audio from its physical sound card and remote audio sent by the remote device A. This mixed audio includes the target device B's system audio and the received remote audio from the remote device A. The acquired audio is then processed by software to separate the audio, resulting in the target device B's system audio, which is then sent to the remote device A. The target device B's system audio refers to local / network audio not acquired through its built-in audio acquisition unit (e.g., a microphone) (e.g., audio played by a local music player); the remote audio refers to audio emitted by a remote device (e.g., external audio acquired through the remote device's audio acquisition unit; or, for example, local / network audio acquired through the remote device).

[0050] S202: Obtain the weight of the remote audio in the mixed audio.

[0051] In one embodiment, to accurately separate the far-end audio from the mixed audio, it is necessary to obtain the weight of the far-end audio in the mixed audio. The specific steps include: obtaining the audio parameters of the target device's volume synthesizer interface; and determining the weight of the far-end audio in the mixed audio based on the audio parameters. The volume synthesizer interface is a built-in interface of the device system (e.g., Windows system), through which at least one audio parameter of the device can be obtained, including the volume and weight of different audio components. This method of obtaining weights through the system's volume synthesizer interface is simple and direct, reducing complex calculation processes.

[0052] S203: Calculate the target playback delay of the far-end audio based on the preset delay calculation conditions.

[0053] In one embodiment, to accurately separate the far-end audio from the mixed audio, in addition to the weight of the far-end audio in the mixed audio, it is also necessary to obtain the target playback delay of the far-end audio. The specific steps include: determining the target mixed audio frame and the target far-end audio frame according to preset delay calculation conditions; and determining the target playback delay of the far-end audio according to the preset delay calculation conditions, the time interval between the target mixed audio frame and the target far-end audio frame. The preset delay calculation conditions refer to a pre-set calculation method for obtaining the target playback delay of the far-end audio. This pre-set method can be set by the manufacturer during production according to requirements, or by the user during use according to actual usage needs. The preset delay calculation conditions specifically include preset time intervals, relevant feature value calculation conditions, a method for determining the target mixed audio frame based on the magnitude of relevant feature values, a method for calculating the initial playback delay, and a method for determining the target playback delay, etc.

[0054] The process involves obtaining a frame of distant audio, calculating the correlation feature values ​​between it and the mixed audio within a recent time period, and selecting the frame with the highest correlation from these feature values ​​as the target mixed audio frame. The specific steps include: selecting any frame from the distant audio as the target distant audio frame and determining the time corresponding to the target distant audio frame as the target time; determining the target time period containing the target time based on a preset time interval and the target time; determining each correlation feature value based on the correlation feature value calculation conditions, the target distant audio frame, and each mixed audio frame within the target time period; and determining the target mixed audio frame from each mixed audio frame based on preset delay calculation conditions and the magnitude of each correlation feature value. The preset time interval refers to the pre-set time interval used to determine the target time period, which can be set by the manufacturer during production or by the user during use, and can be 4s, 6s, etc. The correlation feature value calculation conditions refer to the calculation method used to calculate the correlation feature values, which include cross-correlation coefficients, variance, and other values ​​used to describe the degree of correlation between two signals.

[0055] Since audio is composed of a series of consecutive audio frames, any frame selected from the distant audio can be considered the target distant audio frame. Using this target distant audio frame as a reference, the target mixed audio frame is further determined. Specifically, the time corresponding to the target distant audio frame is first taken as the target time (e.g., 00:02:10). Then, based on the preset time interval (e.g., 4s) in the preset delay calculation conditions, the target time is used as the midpoint to determine the target time period (e.g., 00:02:08 to 00:02:12). Each time within the target time period corresponds to a mixed audio frame. Then, based on the relevant feature value calculation conditions (e.g., cross-correlation coefficient calculation conditions, as shown in Formula 1) in the preset delay calculation conditions, the relevant feature values ​​of each mixed audio frame and the target distant audio frame within the target time period are calculated respectively.

[0056]

[0057] In the formula, x represents the mixed audio frame, y represents the target far-end audio frame, and C xy (t) represents the covariance of the two audio frames, C yy (0) represents the variance of the target remote audio frame at t=0.

[0058] As illustrated in the example above, when the target time is 00:02:10 and the preset time interval is 4, there are a total of 4 mixed audio frames. Therefore, it is necessary to calculate the correlation feature values ​​between the 4 mixed audio frames and the target far-end audio frame in turn. Taking the cross-correlation coefficient as an example, the 4 cross-correlation coefficients ρ are obtained respectively. xy1 ρ xy2 ρxy3 and ρ xy4 Then, the highest value among these four cross-correlation coefficients is selected as the target correlation feature value, and the mixed audio frame corresponding to the target correlation feature value is determined as the target mixed audio frame.

[0059] In one embodiment, after determining the target mixed audio frame and the target far-end audio frame, the target playback delay is calculated according to preset delay calculation conditions. The specific steps include: determining the initial playback delay of the target far-end audio frame based on the preset delay calculation conditions and the time interval between the target mixed audio frame and the target far-end audio frame; determining a corresponding number of consecutive comparison far-end audio frames based on the target far-end audio frame and a preset number of comparison frames; calculating the initial playback delay of the comparison far-end audio frames according to the preset delay calculation conditions; and, when the initial playback delay of the comparison far-end audio frames matches the initial playback delay of the target far-end audio frame, using the initial playback delay as the target playback delay of the far-end audio. The preset number of comparison frames refers to the number of audio frames pre-set in the target playback delay calculation to determine the number of consecutive audio frames with the target far-end audio frame. This pre-setting method can be a system default or a user-defined setting.

[0060] Specifically, firstly, based on the time interval between the target mixed audio frame and the target far-end audio frame determined in the aforementioned steps, the initial playback delay of the target far-end audio frame is calculated. Then, based on the preset number of comparison frames (e.g., 5), audio frames that are continuous with the target far-end audio frame are selected as comparison far-end audio frames. The calculation steps for the initial playback delay in the aforementioned steps are repeated to calculate the initial playback delay corresponding to these 5 comparison far-end audio frames. According to the method for determining the target playback delay in the preset delay calculation conditions, when the initial playback delay of the comparison far-end audio frame is consistent with the initial playback delay of the target far-end audio frame, it is considered that the correct playback delay has been obtained, and this initial playback delay is taken as the target playback delay of the far-end audio.

[0061] It should be noted that the target playback delay obtained through the above method is relatively accurate, which improves the audio separation effect.

[0062] In one embodiment, since finding and calculating the playback delay takes time, in order to output audio without the far-end audio to the far-end device before determining the target playback delay, simple processing of the mixed audio is required. The specific steps include: obtaining the volume ratio of the far-end audio played by the far-end device to the left and right channels; and determining the target audio based on the second target audio calculation conditions and the volume ratio. Here, the second target audio calculation conditions refer to the method of calculating the target audio based on the volume ratio of the far-end audio to the left and right channels.

[0063] First, it is necessary to obtain the volume ratio of the remote device playing remote audio to the left and right channels, for example, the volume ratio of the left channel to the right channel is 6:10. Then, the target audio is calculated according to the second target audio calculation conditions (e.g., Formula 2).

[0064] Output(t)=Left-0.6*Right (Formula 2).

[0065] In the formula, Output is the target audio, Left represents the left channel, and Right represents the right channel.

[0066] Based on the second target audio calculation conditions (e.g., Formula 2), it is possible to ensure that the audio received by the target device is output to the speaker by the remote device without the remote audio.

[0067] However, the difference between the left and right channels of the target audio obtained in this way will be amplified, which is not suitable for long-term use. Therefore, after the correct target playback delay is calculated, the target audio after separating the far-end audio can be obtained by using the first target audio calculation conditions.

[0068] S204: Based on the first target audio calculation conditions, mixed audio, far-end audio, target playback delay, and weights, separate the far-end audio from the mixed audio to obtain the target audio.

[0069] The first target audio calculation condition is the calculation formula for the target audio, as shown in Formula 3.

[0070] Output(t)=SystemAudio(t)-volume*FarendAudio[t+delay] (Formula 3).

[0071] Here, Output is the target audio, SystemAudio is the mixed audio captured from the actual sound card of the target device, volume is the weight, FarendAudio is the far-end audio, and delay is the target playback delay.

[0072] In one embodiment, the specific steps for calculating the target audio based on the first target audio calculation conditions include: determining the far-end delayed audio based on the far-end audio and the target playback delay; determining the target far-end delayed audio based on the far-end delayed audio and a weight; and calculating the difference between the mixed audio and the target far-end delayed audio based on the first target audio calculation conditions to obtain the target audio.

[0073] Specifically, based on the far-end audio FarendAudio and the target playback delay delay, the far-end delayed audio FarendAudio[t+delay] is determined. Then, based on the far-end delayed audio FarendAudio[t+delay] and the weight volume, the target far-end delayed audio volume*FarendAudio[t+delay] is determined. Finally, based on the first target audio calculation condition (e.g., Formula 3), the difference between the mixed audio SystemAudio(t) and the target far-end delayed audio volume*FarendAudio[t+delay] is calculated to obtain the target audio Output(t).

[0074] Therefore, this application separates the far-end audio from the mixed audio by calculating the first target audio calculation conditions, the playback of the far-end audio, and the weight of the audio participating in the mixing. The entire audio separation process does not require the addition of an extra virtual sound card, and the first target audio calculation conditions effectively ensure the integrity of the separated target audio, thus improving the audio separation effect and avoiding the problem of poor audio separation effect in dual-talk scenarios caused by the use of echo cancellation algorithms in the current technology.

[0075] Based on the above embodiments, this application provides an audio separation device that can be installed in a target device. This audio separation device is used to perform the audio separation method provided in the above method embodiments. Specifically, please refer to... Figure 6 The device includes:

[0076] The audio acquisition module 601 is used to acquire mixed audio from the target device and remote audio sent by the remote device, wherein the mixed audio includes the system audio of the target device and the remote audio received from the remote device;

[0077] The weight acquisition module 602 is used to acquire the weight of the far-end audio in the mixed audio;

[0078] The delay calculation module 603 is used to calculate the target playback delay of the far-end audio according to preset delay calculation conditions.

[0079] The separation processing module 604 is used to separate the far-end audio from the mixed audio to obtain the target audio based on the first target audio calculation conditions, the mixed audio, the far-end audio, the target playback delay, and the weight.

[0080] In one embodiment, the weight acquisition module 602 includes:

[0081] The parameter acquisition module is used to acquire the audio parameters of the volume synthesizer interface of the target device;

[0082] The weight determination module is used to determine the weight of the far-end audio in the mixed audio based on the audio parameters.

[0083] In one embodiment, the delay calculation module 603 includes:

[0084] The audio frame determination module is used to determine the target mixed audio frame and the target far-end audio frame based on preset delay calculation conditions.

[0085] The delay determination module is used to determine the target playback delay of the far-end audio based on the preset delay calculation conditions, the time interval between the target mixed audio frame and the target far-end audio frame.

[0086] In one embodiment, the preset delay calculation conditions include a preset time interval and related feature value calculation conditions, and the audio frame determination module includes:

[0087] The first selection module is used to select any frame from the far-end audio as the target far-end audio frame, and determine the time corresponding to the target far-end audio frame as the target time.

[0088] The first determining module is used to determine a target time period including the target time based on the preset time interval and the target time.

[0089] The second determining module is used to determine each relevant feature value based on the relevant feature value calculation conditions, the target far-end audio frame, and each mixed audio frame within the target time period;

[0090] The third determining module is used to determine the target mixed audio frame from each mixed audio frame based on the preset delay calculation conditions and the magnitude of each relevant feature value.

[0091] In one embodiment, the delay determination module includes:

[0092] The fourth determining module is used to determine the initial playback delay of the target far-end audio frame based on the preset delay calculation conditions, the time interval between the target mixed audio frame and the target far-end audio frame;

[0093] The fifth determining module is used to determine a corresponding number of consecutive comparison far-end audio frames based on the target far-end audio frame and a preset number of comparison frames.

[0094] The calculation module is used to calculate the initial playback delay of the comparison far-end audio frame according to the preset delay calculation conditions.

[0095] The comparison module is used to take the initial playback delay as the target playback delay of the far-end audio when the initial playback delay of the comparison far-end audio frame is consistent with the initial playback delay of the target far-end audio frame, based on the preset delay calculation conditions.

[0096] In one embodiment, the audio separation device further includes:

[0097] The first acquisition module is used to acquire the volume ratio of the remote device playing the remote audio to the left and right channels;

[0098] The sixth determining module is used to determine the target audio based on the second target audio calculation conditions and the volume ratio.

[0099] In one embodiment, the separation processing module 604 includes:

[0100] The seventh determining module is used to determine the remote delayed audio based on the remote audio and the target playback delay;

[0101] The eighth determining module is used to determine the target far-end delayed audio based on the far-end delayed audio and the weight;

[0102] The difference calculation module is used to calculate the difference between the mixed audio and the target far-end delayed audio according to the first target audio calculation conditions, so as to obtain the target audio.

[0103] The audio separation device of this application embodiment can be used to execute the technical solution of the aforementioned method embodiment. Its implementation principle and technical effect are similar, and will not be repeated here.

[0104] Unlike current technologies, the audio separation device provided in this application includes a separation processing module. This module calculates and separates the far-end audio from the mixed audio based on the first target audio calculation conditions, the playback of the far-end audio, and the weights of the audio components. The entire audio separation process does not require an additional virtual sound card, and the first target audio calculation conditions effectively ensure the integrity of the separated target audio, thus improving the audio separation effect. It also avoids the problem of poor audio separation in dual-talk scenarios caused by echo cancellation algorithms in current technologies.

[0105] Accordingly, embodiments of this application also provide an electronic device, such as... Figure 7As shown, the electronic device may include a processor 701 with one or more processing cores, a wireless (WiFi, Wireless Fidelity) module 702, a memory 703 with one or more computer-readable storage media, an audio circuit 704, a display unit 705, an input unit 706, a sensor 707, a power supply 708, and a radio frequency (RF) circuit 709, among other components. Those skilled in the art will understand that... Figure 7 The structure of the electronic device shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0106] The processor 701 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 703, and by calling data stored in the memory 703, it performs various functions and processes data, thereby providing overall monitoring of the electronic device. In one embodiment, the processor 701 may include one or more processing cores; preferably, the processor 701 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 701.

[0107] WiFi is a short-range wireless transmission technology. Electronic devices, through the wireless module 702, can help users send and receive emails, browse web pages, and access streaming media, providing users with wireless broadband internet access. Although Figure 7 The wireless module 702 is shown, but it is understood that it is not an essential component of the terminal and can be omitted as needed without changing the nature of the invention.

[0108] The memory 703 can be used to store software programs and modules. The processor 701 executes various functional applications and data processing by running the computer programs and modules stored in the memory 703. The memory 703 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the terminal (such as audio data, phone book, etc.). In addition, the memory 703 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 703 may also include a memory controller to provide access to the memory 703 for the processor 701 and the input unit 706.

[0109] Audio circuit 704 includes a speaker, which provides an audio interface between the user and the electronic device. Audio circuit 704 can convert received audio data into electrical signals and transmit them to the speaker, where the speaker converts them into sound signals for output. Conversely, the speaker can convert collected sound signals into electrical signals, which are then received by audio circuit 704, converted back into audio data, and then processed by processor 701 before being transmitted via radio frequency circuit 709 to, for example, another electronic device, or output to memory 703 for further processing. Audio circuit 704 may also include an earphone jack to facilitate communication between peripheral headphones and the electronic device.

[0110] Display unit 705 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of the terminal. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. Display unit 705 may include a display panel, which in one embodiment can be configured as a liquid crystal display (LCD), organic light-emitting diode (OLED), or similar form. Furthermore, a touch-sensitive surface can cover the display panel. When the touch-sensitive surface detects a touch operation on or near it, it transmits the information to processor 701 to determine the type of touch event. Subsequently, processor 701 provides corresponding visual output on the display panel according to the type of touch event. Although in Figure 7 In this context, the touch-sensitive surface and the display panel are two separate components for implementing input and output functions. However, in some embodiments, the touch-sensitive surface and the display panel can be integrated to achieve input and output functions.

[0111] The input unit 706 can be used to receive input numerical or character information, and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control. Specifically, in one embodiment, the input unit 706 may include a touch-sensitive surface and other input devices. A touch-sensitive surface, also known as a touch display or touchpad, can collect user touch operations on or near it (e.g., user operations using fingers, styluses, or any suitable object or accessory on or near the touch-sensitive surface) and drive corresponding connection devices according to a pre-set program. In one embodiment, the touch-sensitive surface may include a touch detection device and a touch controller. The touch detection device detects the user's touch orientation and the signal generated by the touch operation, transmitting the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, sends it to the processor 701, and can receive and execute commands from the processor 701. Furthermore, various types of touch-sensitive surfaces, such as resistive, capacitive, infrared, and surface acoustic wave, can be used. In addition to the touch-sensitive surface, the input unit 706 may also include other input devices. Specifically, other input devices may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0112] The electronic device may also include at least one sensor 707, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor can adjust the brightness of the display panel according to the ambient light level (the proximity sensor can turn off the display panel and / or backlight when the terminal is moved to the ear). As a type of motion sensor, a gravity acceleration sensor can detect the magnitude of acceleration in various directions (generally three axes), and can detect the magnitude and direction of gravity when stationary, which can be used for applications that recognize the phone's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping, etc.); as for other sensors that may be configured in the electronic device, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be described in detail here.

[0113] The electronic device also includes a power supply 708 (such as a battery) that supplies power to various components. Preferably, the power supply can be logically connected to the processor 701 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 708 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0114] The radio frequency (RF) circuit 709 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and hands it over to one or more processors 701 for processing; additionally, it transmits uplink data to the base station. Typically, the RF circuit 709 includes, but is not limited to, an antenna, at least one amplifier, a tuner, one or more oscillators, a Subscriber Identity Module (SIM) card, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, etc. Furthermore, the RF circuit 709 can also communicate wirelessly with networks and other devices. Wireless communication can use any communication standard or protocol, including but not limited to GSM, GPRS, CDMA, WCDMA, LTE, email, and SMS.

[0115] Although not shown, the electronic device may also include a camera, Bluetooth module, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 701 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 703 according to the following instructions, and the processor 701 runs the applications stored in the memory 703 to achieve the following functions:

[0116] The system collects mixed audio from the target device and remote audio transmitted from the remote device. The mixed audio includes the system audio of the target device and the remote audio received from the remote device.

[0117] Obtain the weight of the far-end audio in the mixed audio;

[0118] The target playback delay of the remote audio is calculated based on the preset delay calculation conditions.

[0119] Based on the first target audio calculation conditions, the mixed audio, the far-end audio, the target playback delay, and the weight, the far-end audio is separated from the mixed audio to obtain the target audio.

[0120] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0121] Therefore, embodiments of this application provide a computer-readable storage medium storing multiple instructions that can be loaded by a processor to implement the functions of the aforementioned audio separation method.

[0122] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0123] The audio separation method, apparatus, electronic device, and computer-readable storage medium provided in the embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An audio separation method, characterized in that, include: The system collects mixed audio from the target device and remote audio transmitted from the remote device. The mixed audio includes the system audio of the target device and the remote audio received from the remote device. Obtain the weight of the far-end audio in the mixed audio; The target playback delay of the far-end audio is calculated according to the preset delay calculation conditions. The preset delay calculation conditions include at least one of the following: preset time interval, relevant feature value calculation conditions, method of determining the target mixed audio frame based on the magnitude of the relevant feature value, calculation method of initial playback delay, and method of determining target playback delay. The relevant feature value is a numerical value used to describe the degree of correlation between two signals. Determine the remote delayed audio based on the remote audio and the target playback delay; The target far-end delayed audio is determined based on the far-end delayed audio and the weight. The difference between the mixed audio and the target far-end delayed audio is calculated to obtain the target audio.

2. The audio separation method according to claim 1, characterized in that, The step of obtaining the weight of the far-end audio in the mixed audio includes: Obtain the audio parameters of the volume synthesizer interface of the target device; Based on the audio parameters, the weight of the far-end audio in the mixed audio is determined.

3. The audio separation method according to claim 1, characterized in that, The step of calculating the target playback delay of the far-end audio according to preset delay calculation conditions includes: Based on the preset delay calculation conditions, the target mixed audio frame and the target far-end audio frame are determined; The target playback delay of the far-end audio is determined based on the preset delay calculation conditions, the time interval between the target mixed audio frame and the target far-end audio frame.

4. The audio separation method according to claim 3, characterized in that, The preset delay calculation conditions include preset time intervals and related feature value calculation conditions. The step of determining the target mixed audio frame and the target far-end audio frame according to the preset delay calculation conditions includes: Select any frame from the far-end audio as the target far-end audio frame, and determine the time corresponding to the target far-end audio frame as the target time; Based on the preset time interval and the target time, a target time period including the target time is determined; Based on the relevant feature value calculation conditions, the target far-end audio frame, and each mixed audio frame within the target time period, determine each relevant feature value; Based on the preset delay calculation conditions and the magnitude of each relevant feature value, the target mixed audio frame is determined from each mixed audio frame.

5. The audio separation method according to claim 3, characterized in that, The step of determining the target playback delay of the far-end audio based on the preset delay calculation conditions, the time interval between the target mixed audio frame and the target far-end audio frame includes: Based on the preset delay calculation conditions, the time interval between the target mixed audio frame and the target far-end audio frame, the initial playback delay of the target far-end audio frame is determined; Based on the target far-end audio frame and the preset number of comparison frames, determine a corresponding number of consecutive far-end audio frames for comparison. Calculate the initial playback delay of the comparison distant audio frame according to the preset delay calculation conditions; According to the preset delay calculation conditions, when the initial playback delay of the comparison far-end audio frame is consistent with the initial playback delay of the target far-end audio frame, the initial playback delay is taken as the target playback delay of the far-end audio.

6. The audio separation method according to claim 5, characterized in that, Before the step of using the initial playback delay as the target playback delay of the far-end audio frame when the initial playback delay of the far-end audio frame matches the initial playback delay of the target far-end audio frame, the method further includes: Obtain the volume ratio of the remote device playing the remote audio to the left and right channels; The target audio is determined based on the second target audio calculation conditions and the volume ratio.

7. An audio separation device, characterized in that, include: An audio acquisition module is used to acquire mixed audio from a target device and remote audio sent by a remote device. The mixed audio includes the system audio of the target device and the remote audio received from the remote device. The weight acquisition module is used to acquire the weight of the far-end audio in the mixed audio; The delay calculation module is used to calculate the target playback delay of the far-end audio according to preset delay calculation conditions. The preset delay calculation conditions include at least one of the following: preset time interval, relevant feature value calculation conditions, method of determining the target mixed audio frame according to the magnitude of the relevant feature value, calculation method of initial playback delay, and method of determining target playback delay. The relevant feature value is a numerical value used to describe the degree of correlation between two signals. The separation processing module is used to determine the far-end delayed audio based on the far-end audio and the target playback delay, determine the target far-end delayed audio based on the far-end delayed audio and the weight, calculate the difference between the mixed audio and the target far-end delayed audio, and obtain the target audio.

8. An electronic device, characterized in that, It includes a processor and a memory, the memory being used to store a computer program, and the processor being used to run the computer program in the memory to perform the steps of the audio separation method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the steps of the audio separation method according to any one of claims 1 to 6.

Images