Audio processing method, electronic device, computer storage medium, and computer program product

Abstract

Provided in the embodiments of the present application are an audio processing method, an electronic device, a computer storage medium, and a computer program product. The audio processing method comprises: acquiring shared audio data from shared audio and video, and remote-end audio data received from a remote end; acquiring first audio data from an audio conversion apparatus, and collecting second audio data by means of an audio collection apparatus, wherein the first audio data is obtained by inputting the shared audio data and the remote-end audio data into the audio conversion apparatus, and the second audio data includes the first audio data, which is played by an audio playback apparatus; using the first audio data as reference data to perform echo cancellation processing on the second audio data, so as to obtain primary echo-canceled data; and on the basis of the primary echo-canceled data and the shared audio data in the first audio data, performing an audio mixing operation, so as to obtain near-end audio data. By means of the embodiments of the present application, users can be prevented from hearing the superimposed sound of shared audio.

Description

Audio processing method, electronic device, computer storage medium and computer program product

[0001] The present application claims priority to the Chinese patent application No. 202411264397.5, filed on September 10, 2024, and entitled "Audio processing method, electronic device, computer storage medium and computer program product", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] Embodiments of the present application relate to the field of audio technology, and in particular to an audio processing method, an electronic device, a computer storage medium and a computer program product. BACKGROUND

[0003] In actual work or life, there are often scenarios that need to share audio or video through a network. For example, in a remote meeting, audio or video sharing is often performed by sharing audio or video. Specifically, a user in a meeting connection shares audio or video by playing the audio or video on a local meeting device to share to a remote meeting device.

[0004] In the process of audio or video sharing, for a sharing party, the sharing content can be output by a playing program such as an audio or video playing program installed on the local meeting device. After the shared audio enters an audio conversion device such as a sound card of the device, the local meeting device collects the audio output by the sound card, and also plays the shared audio through an audio playing device such as a loudspeaker. The shared audio played out is collected by an audio collecting device such as a microphone of the local meeting device.

[0005] Because the time delay between the shared audio collected from the audio conversion device and the shared audio collected through the audio collecting device is different, the result heard by a remote user will be the superposition of the shared audio twice, causing overlapping sound. Moreover, the greater the time delay interval between the two, the more serious the overlapping sound, resulting in poor meeting experience of the user. SUMMARY

[0006] Therefore, embodiments of the present application provide an audio processing method, an electronic device, a computer storage medium and a computer program product to at least partially solve the above problems.

[0007] According to a first aspect of embodiments of the present application, an audio processing method is provided, comprising: obtaining shared audio data in shared audio-video and remote audio data received from a remote end; obtaining first audio data from an audio conversion device and collecting second audio data through an audio collection device; the first audio data is obtained by inputting the shared audio data and the remote audio data into the audio conversion device; the second audio data contains the first audio data played by an audio playing device; performing echo cancellation processing on the second audio data by taking the first audio data as reference data to obtain main echo cancellation data; and performing mixing operation based on the main echo cancellation data and the shared audio data in the first audio data to obtain near-end audio data.

[0008] According to a second aspect of embodiments of the present application, an audio processing device is provided, comprising an obtaining module, an audio collection module, an echo cancellation module and a mixing module. The obtaining module is configured to obtain shared audio data in shared audio-video and remote audio data received from a remote end; the audio collection module is configured to obtain first audio data from an audio conversion device and collect second audio data through an audio collection device; the first audio data is obtained by inputting the shared audio data and the remote audio data into the audio conversion device; the second audio data contains the first audio data played by an audio playing device; the echo cancellation module is configured to perform echo cancellation processing on the second audio data by taking the first audio data as reference data to obtain main echo cancellation data; and the mixing module is configured to perform mixing operation based on the main echo cancellation data and the shared audio data in the first audio data to obtain near-end audio data.

[0009] According to a third aspect of embodiments of the present application, an electronic device is provided, comprising a processor, a memory, a communication interface and a communication bus, the processor, the memory and the communication interface complete communication with each other through the communication bus; the memory is configured to store at least one executable instruction, the executable instruction causes the processor to perform operations corresponding to the method of the first aspect.

[0010] According to a fourth aspect of embodiments of the present application, a computer storage medium is provided, which stores a computer program, the program is executed by a processor to implement the method of the first aspect.

[0011] According to a fifth aspect of embodiments of the present application, a computer program product is provided, comprising computer instructions, the computer instructions instruct a computing device to perform operations corresponding to the method of the first aspect.

[0012] In the embodiment of the present application, the second audio data obtained by the audio collection device is subjected to echo cancellation processing with reference to the first audio data obtained by the audio conversion device as reference data, so that the audio data other than the first audio data, i.e., the main echo cancellation data, is filtered out from the collected second audio data. Then, the mixing operation is performed based on the main echo cancellation data and the shared audio data in the first audio data, to obtain the near-end audio data.

[0013] The second audio data is collected by the audio collection device, and thus the first audio data contained therein has noise interference and is not good in data quality. The first audio data contains the shared audio data of good audio quality collected from the audio conversion device. Based on this, in the embodiment of the present application, the audio data with noise interference contained in the second audio data is eliminated to obtain the main echo cancellation data. Then, the mixing operation is performed to add the shared audio data of good audio quality to the main echo cancellation data, so that the near-end audio data obtained finally contains the shared audio data of high quality and only one shared audio data, thereby avoiding audio overlap and improving the quality of the near-end audio data containing the shared audio data shared. BRIEF DESCRIPTION OF DRAWINGS

[0014] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments described in the embodiments of the present application, and other drawings can also be obtained by those skilled in the art based on these drawings.

[0015] FIG. 1 is a schematic diagram of an application scenario of an exemplary audio processing method;

[0016] FIG. 2 is a step flowchart of an audio processing method according to an exemplary embodiment of the present application;

[0017] FIG. 3 is a step flowchart of another audio processing method according to an exemplary embodiment of the present application;

[0018] FIG. 4 is a processing process diagram of an audio processing method according to an exemplary embodiment of the present application;

[0019] FIG. 5 is a structural block diagram of an audio processing device according to another embodiment of the present application;

[0020] FIG. 6 is a structural schematic diagram of an electronic device according to another embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the personnel in the art better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all embodiments. Based on the embodiments in the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art shall fall within the scope of protection of the embodiments of the present application.

[0022] Before the audio processing method of the embodiments of the present application is described, the application scenario of the audio processing method is briefly described to facilitate understanding.

[0023] Referring to FIG. 1, FIG. 1 is a schematic diagram of an exemplary application scenario of an audio processing method. In this example, a remote conference scenario is taken as an example, but those skilled in the art should understand that other scenarios that can share audio or video over a network are also applicable to the solutions of the embodiments of the present application.

[0024] In this example, in the remote conference scenario, the local user and the remote user are connected for a conference through a conference device, and the conference device is provided with an audio conversion device, an audio playback device and an audio acquisition device. In FIG. 1, the audio conversion device uses a sound card, the audio playback device uses a loudspeaker, and the audio acquisition device uses a microphone. The conference device can be a mobile phone, a tablet, a notebook computer, a desktop computer or other computing terminal device, and the embodiments of the present application do not limit this.

[0025] During the conference, the local user can share audio or video with the remote conference device of the remote user through the local conference device, or the remote user can share audio or video with the local conference device of the local user through the remote conference device. In the embodiments of the present application, the solutions of the embodiments of the present application are mainly described from the perspective of the party sharing audio or video. Moreover, for the sake of simplicity, the shared audio or video is collectively referred to as shared audio video, and the audio data corresponding to the shared audio and the audio data in the shared video are collectively referred to as shared audio data. Accordingly, since the processing of audio data is mainly described, the program for playing audio or the program for playing video is also simply referred to as an audio playback program.

[0026] In addition, in the embodiments of the present application, the local user sharing audio video with the remote conference device of the remote user through the local conference device is taken as an example for description, and in the audio video sharing process, the local conference device usually processes the shared audio data as follows:

[0027] The local user can output the shared audio data through an audio playing program installed in the local conference device, for example, the audio playing program can be a third-party application different from the conference application corresponding to the current conference for playing audio or video. When one of the users shares a playing video, the remote user can synchronously watch and listen to the related audio in real time, and the embodiments of the present application do not limit this. In this way, the local user can further demonstrate and share the content that needs to be explained in the conference through the third-party application, which is convenient for information sharing, understanding, discussion and the like of the local and remote multi-site users participating in the conference. For example, the local user A introduces a product as the conference lecturer, and the product has a corresponding video demonstration (denoted as video X). Therefore, the local user A can play the video X using a third-party video playing program different from the conference application while using the conference application for the multi-site conference, and other users participating in the conference can share the video X with the local user A, realize synchronous real-time watching of the multi-site, and can listen to the audio therein.

[0028] In the conference process, when the local conference device plays the shared audio and video through the third-party application, the receiving unit of the local conference device also receives the remote audio data (such as voice data and / or environmental sound data of the remote participants and the like) sent by the remote conference device, and simultaneously sends the received remote audio data to the main echo cancellation unit, the secondary echo cancellation unit and the audio conversion device such as a sound card of the local conference device. The main echo cancellation unit and the secondary echo cancellation unit can both be an acoustic echo canceller AEC, which is used to filter the data of the reference audio contained in the target audio based on the data corresponding to the reference audio (reference data). Generally, when implementing the filtering operation, the acoustic echo canceller AEC needs to receive the reference audio earlier than the target audio.

[0029] The audio playing program also outputs the shared audio data to the audio conversion device such as a sound card, and then processes the remote audio data and the shared audio data through the audio conversion device such as a sound card to form mixed audio mixed with the two kinds of audio data, and transmits the mixed audio to the audio playing device such as a loudspeaker for playing, so that the local user can hear the mixed audio which contains the remote audio data and the shared audio data.

[0030] In the secondary echo cancellation unit, the mixed audio data of the mixed audio is obtained from the audio conversion device such as a sound card, and the mixed audio data containing the shared audio data and the remote audio data is processed for echo cancellation with the remote audio data sent by the receiving unit as reference data, the remote audio data is filtered out, the shared audio data is obtained, and the shared audio data is sent to the mixing unit. Here, because the delay of obtaining the remote audio data from the receiving unit is very small and can be ignored, it is approximately considered that there is no delay, and the mixed audio data obtained from the audio conversion device such as a sound card has a system delay of a fixed time length caused by the processing of the audio conversion device such as a sound card. Therefore, the time when the secondary echo cancellation unit receives the remote audio data is earlier than the time when the mixed audio data is received.

[0031] In the primary echo cancellation unit, the shared audio data and the remote audio data played by the audio playback device such as a loudspeaker are obtained from the audio acquisition device such as a microphone, and the local audio data (such as voice data of the local user and / or environmental sound data) can also be included; and the audio data acquired by the audio acquisition device such as a microphone is processed for echo cancellation with the remote audio data sent by the receiving unit as reference data, the shared audio data and the local audio data are obtained, and the shared audio data and the local audio data are sent to the mixing unit. Here, the reason for taking the remote audio data sent by the receiving unit as reference data is as follows: there is no delay in obtaining the remote audio data from the receiving unit, but there is a delay in acquiring the audio data by the audio acquisition device such as a microphone. Therefore, the time when the primary echo cancellation unit receives the remote audio data is earlier than the time when the audio data acquired by the audio acquisition device such as a microphone is received. In contrast, there is an unfixed system delay in obtaining the audio data from the audio conversion device such as a sound card, and there is also a delay in acquiring the audio data by the audio acquisition device such as a microphone, so it cannot be determined whether the delay of the two meets the delay requirement of the echo canceller. Therefore, the remote audio data sent by the receiving unit is usually taken as reference data to ensure that the echo canceller can work normally.

[0032] In the mixing unit, the shared audio data sent by the secondary echo cancellation unit, and the shared audio data and the local audio data sent by the primary echo cancellation unit are mixed to obtain target audio data, and the target audio data is sent to the remote conference device through the sending unit to realize the sharing of the shared audio data by the local user and the remote user. In addition, the local audio data can be description or explanation data of the shared audio data, such as explanation data of the shared audio and video in a conference scenario, so the local audio data is also sent to the remote conference device, so that the remote user can hear the audio corresponding to the shared audio data and the voice of the local user corresponding to the local audio data at the same time.

[0033] But in the above audio processing process, since the target audio data contains two shared audio data, and the delay of the two shared audio data is different, when the local conference equipment sends the target audio data to the remote conference equipment, the audio heard by the remote user will produce the phenomenon of overlapping sound. To this end, the embodiments of the present application provide further solutions. It should be noted that in the following embodiments of the present application, the audio conversion device is taken as a sound card, the audio playback device is taken as a loudspeaker, and the audio acquisition device is taken as a microphone. However, it should be understood by those skilled in the art that this is only an exemplary description, and is not a limitation on the embodiments of the present application.

[0034] The specific implementation of the solution provided by the embodiments of the present application will be further described below in combination with the drawings of the embodiments of the present application.

[0035] Referring to FIG. 2, a flowchart of steps of an audio processing method according to an exemplary embodiment of the present application is shown. The audio processing method of the present embodiment is applicable to an audio-video sharing initiator, which includes the following steps:

[0036] S202, obtaining shared audio data in shared audio-video and remote audio data received from a remote end.

[0037] In an audio-video sharing scenario, there is usually interaction between a local user and a remote user. In the embodiments of the present application, the user corresponding to the audio-video sharing initiator is the local user. Therefore, in this step, the shared audio data can be obtained from the audio-video shared by the local user, and the remote audio data sent by the remote user can be received.

[0038] S204, obtaining first audio data from the audio conversion device, and collecting second audio data through the audio acquisition device.

[0039] The first audio data can be obtained based on the remote audio data and the shared audio data. For example, the shared audio data and the remote audio data can be input into an audio conversion device such as a sound card, and the first audio data can be obtained after processing by the sound card. The sound card usually has a sound processing program, and the first audio data can be obtained by processing the shared audio data and the remote audio data through the sound processing program. The specific processing process of the sound processing program for audio can be set and implemented by those skilled in the art according to actual needs, or can be implemented through the program burned in the sound card. The embodiments of the present application do not limit the implementation of the specific processing of the sound.

[0040] In order to accurately determine the time when the relevant audio data is obtained in the subsequent processing process, in one feasible manner, a mixing operation can be performed based on a preset ultrasonic signal and the remote audio data to obtain reference audio data, and the first audio data can be obtained based on the reference audio data and the shared audio data.

[0041] On this basis, after obtaining the reference audio data, the time when the reference audio data is obtained can be determined as the reference time in the process of playing the shared audio data.

[0042] Exemplarily, the reference time of the reference audio data can be taken as a time reference to determine the acquisition delay of other audio data. As described above, the reference audio data can be remote audio data received from the remote conference device, or can be audio data obtained by adding an ultrasonic signal to the remote audio data, so as to detect the delay by using the ultrasonic signal, and the embodiments of the present application do not limit this.

[0043] Exemplarily, the first audio data can be obtained from the audio conversion device, wherein the first audio data is obtained by inputting the shared audio data and the reference audio data into the audio conversion device (such as a sound card). It should be noted that the reference audio data in the first audio data can be remote audio data, or can be remote audio data + ultrasonic signal. In order to distinguish from the mixed audio data in the mode shown in FIG. 1, the data output by the sound card in the embodiment is referred to as the first audio data.

[0044] And the second audio data can be collected by the audio collection device (such as a microphone), wherein the second audio data includes the first audio data played by the audio playing device such as a loudspeaker, that is, the second audio data includes the shared audio data and the reference audio data collected by the microphone. The first audio data processed by the audio conversion device is transmitted to the secondary echo cancellation unit on the one hand, and is transmitted to the audio playing device such as a loudspeaker for external playing, and is collected by the audio collection device such as a microphone. But not limited to this, further, the second audio data can also include local audio data, such as voice data and / or environmental sound data of the local user.

[0045] S206, taking the first audio data as reference data, performing echo cancellation processing on the second audio data to obtain primary echo cancellation data.

[0046] In order to make the audio shared subsequently without echo and with high audio data quality, therefore, it is necessary to take the first audio data as reference data, to perform echo cancellation processing on the second audio data with noise and slightly poor quality, to filter out the first audio part in the second audio data, and to retain other data parts, such as local audio data parts.

[0047] In an implementation, to make the echo cancellation processing more efficient and accurate, in an implementation, the first audio data is taken as the reference data to perform echo cancellation processing on the second audio data to obtain the primary echo cancellation data, in a case that a first delay between the reference time and an acquisition time of the first audio data acquired from an audio conversion device (such as a sound card) is less than a second delay between the reference time and an acquisition time of the second audio data acquired by an audio acquisition device (such as a microphone). Illustratively, the acquisition time of the first audio data can be determined by the secondary echo cancellation unit when the first audio data output by the audio conversion device is acquired, and the acquisition time of the second audio data can be determined by the primary echo cancellation unit when the second audio data is acquired by the audio acquisition device. Then, the secondary echo cancellation unit determines the first delay according to the reference time and the acquisition time of the first audio data and sends the first delay to the primary echo cancellation unit. The primary echo cancellation unit determines the second delay according to the reference time and the acquisition time of the second audio data. After receiving the first delay and determining the second delay, the primary echo cancellation unit compares the first delay and the second delay. If the first delay is less than the second delay, it means that the first audio data can be acquired earlier than the second audio data, and thus the first audio data can be taken as the reference data to perform echo cancellation processing on the second audio data to obtain the primary echo cancellation data. Here, taking the first audio data as the reference data to perform echo cancellation processing on the second audio data can eliminate the shared audio data and the reference audio data in the second audio data, and thus the primary echo cancellation data in this embodiment does not include the shared audio data, which is different from the implementation shown in FIG. 1.

[0048] In S208, a mixing operation is performed based on the primary echo cancellation data and the shared audio data in the first audio data to obtain the near-end audio data.

[0049] Illustratively, the first audio data can be first taken as the reference data to perform echo cancellation processing (filtering out the reference audio data) to obtain the secondary echo cancellation data including the shared audio data, and then a mixing operation is performed on the primary echo cancellation data and the secondary echo cancellation data to obtain the near-end audio data.

[0050] In this embodiment, during the playing of the shared audio data, the first delay of acquiring the first audio data from the audio conversion device and the second delay of acquiring the second audio data by the audio acquisition device are determined based on the reference time of acquiring the reference audio data, and the first delay and the second delay are compared. If the first delay is less than the second delay, the first audio data is taken as the reference data to perform echo cancellation processing on the second audio data to obtain the primary echo cancellation data, and then a mixing operation is performed based on the primary echo cancellation data and the shared audio data in the first audio data to obtain the near-end audio data.

[0051] In an example, the second audio data includes not only the first audio data collected by the audio collection device (such as a microphone), but also local audio data (including but not limited to voice data of the local user and / or environmental sound data). The second audio data can be processed by echo cancellation with the first audio data as the reference data, and the local audio data obtained after the processing is taken as the main echo cancellation data. The data obtained from the main echo cancellation unit after filtering out the first audio data is mixed with the shared audio data obtained from the secondary echo cancellation unit after the filtering out of the first audio data from the reference audio data, to obtain the near-end audio data containing the shared audio data and the local audio data. This is because the quality of the shared audio data in the first audio data obtained from the sound card is much better than that of the shared audio data collected by the microphone, and selecting the shared audio data with good quality and filtering out the shared audio data with poor quality in the second audio data collected by the microphone can effectively avoid the generation of overlapping sound, and the quality of the generated near-end audio data is high.

[0052] That is, the second audio data contains shared audio data with noise interference collected by the audio collection device, and the first audio data contains shared audio data with good sound quality collected from the audio conversion device. Based on this, in the embodiment of the application, when the time delay of the first audio data is less than that of the second audio data, the shared audio data with noise interference contained in the second audio data can be eliminated with the first audio data as the reference data, to obtain the main echo cancellation data. Then, the shared audio data with good sound quality is re-added in the main echo cancellation data through mixing operation, so that the final obtained near-end audio data contains only one shared audio, thereby avoiding the situation that the remote user hears the overlapping sound of the shared audio due to the fact that the near-end audio data contains two shared audios, and the audio data quality of the obtained near-end audio data is also high.

[0053] Referring to FIG. 3, a flowchart of steps of an audio processing method according to another example embodiment of the application is shown. As shown in FIG. 3, the embodiment is further improved on the basis of the above-mentioned embodiment, and the specific implementation of the determination of the reference time, the acquisition time and the collection time is further improved.

[0054] The embodiment includes the following steps:

[0055] S302, shared audio data in the shared audio and video is obtained, and during the playing of the shared audio data, remote audio data is received, and mixing operation is performed on the preset ultrasonic signal and the remote audio data to obtain reference audio data, and the time at which the reference audio data is obtained is determined as the reference time.

[0056] Exemplarily, the preset ultrasonic signal can be an ultrasonic signal pre-stored in the conference device, and the ultrasonic signal can be mixed with the remote audio data by the mixing module to obtain the reference audio data after the remote audio data is received.

[0057] S304, obtaining the first audio data from the audio conversion device, and collecting the second audio data by the audio collection device.

[0058] The first audio data is obtained by inputting the shared audio data and the reference audio data into the audio conversion device and processing the shared audio data and the reference audio data by the audio conversion device; and the second audio data contains the first audio data played by the audio playing device.

[0059] S306, performing the matched filtering processing on the ultrasonic signal in the first audio data to determine the acquisition time of the first audio data.

[0060] Exemplarily, the matched filtering signal can be obtained by performing the time domain reverse sequence processing on the ultrasonic signal, and the matched filtering processing can be performed on the ultrasonic signal in the first audio data by the matched filter according to the matched filtering signal when the audio conversion device such as the sound card outputs the first audio data, so as to determine the time when the signal amplitude of the ultrasonic signal in the first audio data is most matched with the signal amplitude of the matched filtering signal, that is, the acquisition time of the first audio data. The time domain reverse sequence processing on the ultrasonic signal can be understood as reversing the ultrasonic signal on the time axis.

[0061] S308, performing the matched filtering processing on the ultrasonic signal in the second audio data to determine the collection time of the second audio data.

[0062] Here, the process of performing the matched filtering processing on the ultrasonic signal in the second audio data is similar to the process in step S306, and will not be described herein. In actual application, step S308 can be executed before step S306 or in parallel with step S306.

[0063] S310, if the first delay between the reference time and the acquisition time of the first audio data is less than the second delay between the reference time and the collection time of the second audio data, taking the first audio data as the reference data, performing the echo cancellation processing on the second audio data to obtain the main echo cancellation data.

[0064] It should be noted that step S310 can be implemented by referring to the specific embodiments of step S206 described above, and will not be described herein.

[0065] S312, perform a mix operation based on the main echo cancellation data and the shared audio data in the first audio data to obtain the near-end audio data.

[0066] In this embodiment, the reference audio data is obtained by performing a mix operation on the preset ultrasonic signal and the far-end audio data. The reference time can be obtained by performing a matched filtering process on the ultrasonic signal. Since the reference data contains the ultrasonic signal, the first audio signal and the second audio signal obtained subsequently both contain the ultrasonic signal. The characteristics of the ultrasonic signal are used to obtain the acquisition time of the first audio signal and the acquisition time of the second audio signal by performing a matched filtering process on the ultrasonic signal, and then the first delay and the second delay are determined, which can improve the accuracy of the delay detection result. The ultrasonic signal cannot be heard by human ears, so the whole process is not noticeable to the user and does not interfere with the user. In addition, the data calculation amount of the matched filtering process on the ultrasonic signal is small, and the memory occupation of the conference equipment is less.

[0067] In an optional implementation, performing a mix operation based on the main echo cancellation data and the shared audio data in the first audio data to obtain the near-end audio data includes: taking the reference audio data as reference data, performing echo cancellation processing on the first audio data to obtain secondary echo cancellation data (the reference audio data is filtered out and the shared audio data is retained); and performing a mix operation on the main echo cancellation data (the reference audio data and the shared audio data are filtered out and the local audio data is retained) and the secondary echo cancellation data to obtain the near-end audio data.

[0068] For example, the reference audio data can be taken as reference data, and the first audio data can be subjected to echo cancellation processing to obtain secondary echo cancellation data. Here, the reference audio data in the first audio data is eliminated, so that the secondary echo cancellation data only includes the shared audio data. Then, a mix operation is performed on the main echo cancellation data and the secondary echo cancellation data to obtain the near-end audio data. Here, the reference audio data is not included in the near-end audio data.

[0069] In this implementation, the reference audio data is taken as reference data, and the first audio data is subjected to echo cancellation processing to obtain secondary echo cancellation data. Then, a mix operation is performed on the main echo cancellation data and the secondary echo cancellation data to obtain the near-end audio data, which eliminates the reference audio data in the near-end audio data, i.e., eliminates the far-end audio data sent by the far-end conference equipment. After the conference equipment transmits the near-end audio data to the far-end user, the far-end user will not hear the overlap of the shared audio data, and will not hear the audio sent by himself again, which improves the conference experience of the user.

[0070] In an optional implementation, the method further includes: if the first delay between the reference time and the acquisition time of the first audio data is greater than or equal to the second delay between the reference time and the collection time of the second audio data, taking the reference audio data as the reference data, performing echo cancellation processing on the second audio data to obtain primary echo cancellation data; and performing mixing operation based on the primary echo cancellation data and the first audio data to obtain the near-end audio data.

[0071] For example, if the first delay between the reference time and the acquisition time of the first audio data is greater than or equal to the second delay between the reference time and the collection time of the second audio data, it means that the acquisition of the first audio data cannot be earlier than the acquisition of the second audio data, and thus the requirement of echo cancellation processing cannot be met. Therefore, the reference audio data is taken as the reference data, the echo cancellation processing is performed on the second audio data to obtain the primary echo cancellation data, and the mixing operation is performed based on the primary echo cancellation data and the first audio data to obtain the near-end audio data.

[0072] In the implementation, when the acquisition of the first audio data cannot be earlier than the acquisition of the second audio data, the reference audio data is taken as the reference data, the echo cancellation processing is performed on the second audio data to eliminate the reference audio data in the second audio signal, that is, to eliminate the far-end audio data, so that the near-end audio data obtained at least does not contain the far-end audio data transmitted by the far-end conference device, and the echo of the far-end user is avoided.

[0073] In an optional implementation, the receiving of the far-end audio data includes: receiving the far-end code stream data transmitted by the far-end conference terminal, and performing decoding operation on the far-end code stream data to obtain the far-end audio data. After the near-end audio data is obtained, the method further includes: performing encoding processing on the near-end audio data to obtain near-end code stream data; and transmitting the near-end code stream data to the far-end conference terminal.

[0074] For example, the conference device further includes an encoding unit and a decoding unit. The far-end audio data is carried in the far-end code stream data, and thus after the far-end code stream data transmitted by the far-end conference terminal is received, the decoding operation is performed on the far-end code stream data by the decoding unit to obtain the far-end audio data. After the near-end audio data is obtained, the encoding processing is performed on the near-end audio data by the encoding unit to obtain the near-end code stream data, and the near-end code stream data is transmitted to the far-end conference terminal. The audio data is transmitted after the encoding operation, and the audio data is obtained after the decoding operation, which can improve the security of the audio data in the transmission process.

[0075] Referring to FIG. 4, FIG. 4 is a schematic diagram of a processing procedure of an audio processing method according to an example embodiment of the present application. As shown in FIG. 4, the example embodiment is described by taking an example of a local user sharing audio and video with a remote conference device of a remote user through a local conference device. During the audio and video sharing process, the audio processing method according to the example embodiment of the present application processes the audio data as follows.

[0076] During the audio and video sharing process, the decoding unit of the local conference device receives the remote code stream data sent by the remote conference device, decodes the remote code stream data to obtain remote audio data, and sends the obtained remote audio data to the first mixing unit.

[0077] The first mixing unit mixes the preset ultrasonic signal and the remote audio data to obtain reference audio data, and sends the reference audio data to the main echo cancellation unit, the secondary echo cancellation unit and the sound card of the local conference device.

[0078] At the sound card, the audio playing program also outputs the shared audio data to the sound card, and then processes the reference audio data and the shared audio data to obtain first audio data and transmits the first audio data to the loudspeaker for playing, so that the local user can hear the remote audio and the shared audio.

[0079] At the secondary echo cancellation unit, the first audio data is obtained from the sound card, and the reference audio data sent by the first mixing unit is used as reference data to perform echo cancellation processing on the first audio data to obtain shared audio data, and the shared audio data is sent to the second mixing unit. In addition, the secondary echo cancellation unit can also determine a reference time of obtaining the reference audio data according to the ultrasonic signal in the reference audio data, and determine an obtaining time of the first audio data according to the ultrasonic signal in the first audio data, and then determine a first delay according to the reference time and the obtaining time, and send the first delay to the main echo cancellation unit.

[0080] In the primary echo cancellation unit, the shared audio data and the reference audio data played by the loudspeaker and collected by the microphone, and the local audio data (e.g. the voice data of the local user and / or the environmental sound data) are obtained to obtain second audio data; and the reference time point at which the reference audio data is obtained is determined according to the ultrasonic signal in the reference audio data, and the collection time point of the second audio data is determined according to the ultrasonic signal in the second audio data; then the second delay is determined according to the reference time point and the collection time point; it is judged whether the first delay is less than the second delay; if the first delay is less than the second delay, the first audio data is obtained from the sound card, the second audio data collected by the microphone is processed for echo cancellation with the first audio data as the reference data to obtain the local audio data, and the local audio data is sent to the second mixing unit; in the second mixing unit, the shared audio data sent by the secondary echo cancellation unit and the local audio data sent by the primary echo cancellation unit are mixed to obtain the near-end audio data. In this case, the shared near-end audio data has no echo and has high audio quality.

[0081] If the first delay is greater than or equal to the second delay, the second audio data collected by the microphone is processed for echo cancellation with the reference audio data as the reference data to obtain the shared audio data and the local audio data, and the shared audio data and the local audio data are sent to the second mixing unit; in the second mixing unit, the shared audio data sent by the secondary echo cancellation unit and the shared audio data and the local audio data sent by the primary echo cancellation unit are mixed to obtain the near-end audio data. In this case, the scheme reverts to the mode shown in FIG. 1, but the normal sharing of the audio data can still be ensured.

[0082] The near-end audio data is encoded by the encoding unit to obtain near-end code stream data, and the near-end code stream data is sent to the far-end conference terminal.

[0083] Referring to FIG. 5, a structural block diagram of an audio processing apparatus according to an example embodiment of the present application is shown.

[0084] The audio processing apparatus of the present embodiment includes an obtaining module 502, an audio collecting module 504, an echo cancellation module 506 and a mixing module 508.

[0085] The acquisition module 502 is configured to acquire shared audio data in the shared audio and video and remote audio data received from a remote end; the audio acquisition module 504 is configured to acquire first audio data from an audio conversion device and collect second audio data through an audio collection device; the first audio data is obtained by inputting the shared audio data and the remote audio data into the audio conversion device; the second audio data includes the first audio data played through an audio playing device; the echo cancellation module 506 is configured to perform echo cancellation processing on the second audio data by taking the first audio data as reference data to obtain main echo cancellation data; and the mix module 508 is configured to perform mix operation based on the main echo cancellation data and the shared audio data in the first audio data to obtain near-end audio data.

[0086] In an optional implementation, the audio acquisition module 504 obtains the first audio data by performing mix operation based on a preset ultrasonic signal and the remote audio data to obtain reference audio data and obtaining the first audio data based on the reference audio data and the shared audio data.

[0087] In an optional implementation, the audio acquisition module 504 is further configured to determine a time when the reference audio data is obtained as a reference time after obtaining the first audio data. In this case, the echo cancellation module 506 is configured to perform echo cancellation processing on the second audio data by taking the first audio data as reference data to obtain main echo cancellation data if the first delay is less than the second delay, wherein the first delay is a delay between the reference time and a time when the audio conversion device outputs the first audio data, and the second delay is a delay between the reference time and a time when the audio collection device collects the second audio data.

[0088] In an optional implementation, the audio acquisition module 504 is further configured to perform matched filtering processing on the ultrasonic signal in the first audio data to determine a time when the first audio data is obtained and perform matched filtering processing on the ultrasonic signal in the second audio data to determine a time when the second audio data is collected.

[0089] In an optional implementation, the mix module 508 is further configured to perform echo cancellation processing on the first audio data by taking the reference audio data as reference data to obtain secondary echo cancellation data including the shared audio data, and perform mix operation on the main echo cancellation data and the secondary echo cancellation data to obtain the near-end audio data.

[0090] In an optional implementation, the second audio data further includes local audio data; and the echo cancellation module 506 is further configured to perform echo cancellation processing on the second audio data by taking the first audio data as reference data, and take the local audio data obtained after the processing as the main echo cancellation data.

[0091] In an optional implementation, the echo cancellation module 506 is further configured to: if the first delay between the reference time and the acquisition time of the first audio data is greater than or equal to the second delay between the reference time and the acquisition time of the second audio data, perform echo cancellation processing on the second audio data by taking the reference audio data as reference data to obtain main echo cancellation data; and the mixing module 508 is further configured to: perform mixing operation based on the main echo cancellation data and the first audio data to obtain the near-end audio data.

[0092] In an optional implementation, the acquisition module 502 is further configured to: receive remote code stream data sent by the remote conference terminal, and perform decoding operation on the remote code stream data to obtain remote audio data.

[0093] The audio processing apparatus of the embodiment further includes a sending module configured to: perform encoding processing on the near-end audio data to obtain near-end code stream data; and send the near-end code stream data to the remote conference terminal.

[0094] The audio processing apparatus of the embodiment is configured to implement the corresponding audio processing method in the foregoing method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described here again. In addition, the functions of each module in the audio processing apparatus of the embodiment can be implemented with reference to the descriptions of the corresponding parts in the foregoing method embodiments, which will not be described here again.

[0095] Referring to FIG. 6, a structural schematic diagram of an electronic device according to another embodiment of the present application is shown, and the specific implementation of the electronic device is not limited in the embodiments of the present application.

[0096] As shown in FIG. 6, the electronic device can include a processor 602 configured to execute a program 610, a communications interface 604, a memory 606, and a communications bus 608.

[0097] Among them:

[0098] The processor, the communications interface, and the memory complete the communication among each other through the communications bus.

[0099] The communications interface is configured to communicate with other electronic devices or servers.

[0100] The processor is configured to execute the program, and specifically can execute the related steps in the method embodiments.

[0101] Specifically, the program can include program code, and the program code includes computer operation instructions.

[0102] The processor can be a CPU, or a GPU (Graphics Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement one or more embodiments of the present application. The one or more processors included in the electronic device can be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0103] The memory is configured to store programs. The memory can include a high-speed RAM memory, and can further include a non-volatile memory, such as at least one disk memory.

[0104] The program can include a plurality of computer instructions, and the program can specifically cause the processor to perform operations corresponding to the audio processing method described in any one of the foregoing method embodiments.

[0105] The specific implementation of each step in the program can refer to the corresponding description in the corresponding steps and units in the foregoing method embodiments, and has corresponding beneficial effects, which will not be described here. Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device and the module described above can refer to the corresponding process description in the foregoing method embodiments, which will not be described here.

[0106] The embodiments of the present application also provide a computer storage medium, which stores a computer program, and the program is executed by a processor to implement the method described in any one of the foregoing method embodiments. The computer storage medium includes but is not limited to: a CD-ROM (Compact Disc Read-Only Memory), a RAM (Random Access Memory), a floppy disk, a hard disk, or a magneto-optical disk, etc.

[0107] The embodiments of the present application also provide a computer program product, which includes computer instructions, and the computer instructions instruct a computing device to perform operations corresponding to the method described in any one of the foregoing method embodiments.

[0108] In addition, it should be noted that the information related to the user (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to sample data for training the model, data for analysis, stored data, displayed data, etc.) involved in the embodiments of the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations and standards, and provide corresponding operation portal for user to choose authorization or rejection.

[0109] It should be noted that, according to the needs of implementation, each component / step described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or part of the operation of the components / steps can be combined into a new component / step, to achieve the purpose of the embodiments of the present application.

[0110] The above method according to the embodiments of the present application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium such as a CD-ROM, RAM, floppy disk, hard disk or magneto-optical disk, or be downloaded through a network and originally stored in a remote recording medium or non-transitory machine readable medium and then stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general computer, a special processor or programmable or special hardware such as an Application Specific Integrated Circuit (ASIC) or a Field Programmable Gate Array (FPGA). It can be understood that the computer, processor, microprocessor controller or programmable hardware includes a storage component (for example, Random Access Memory (RAM), Read-Only Memory (ROM), flash memory, etc.) that can store or receive software or computer code, when the software or computer code is accessed and executed by the computer, processor or hardware, the method described herein is implemented. In addition, when the general computer accesses the code for implementing the method shown herein, the execution of the code will convert the general computer into a special computer for executing the method shown herein.

[0111] Those of ordinary skill in the art can realize that the units and method steps of the examples described in combination with the embodiments disclosed herein can be realized in electronic hardware, or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of the present application.​​​

[0112] The above embodiments are only used for illustrating the present application, but not for limiting the present application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present application. Therefore, all equivalent technical solutions belong to the scope of the present application, and the patent protection scope of the present application should be defined by the claims.

Claims

1. An audio processing method comprising: obtaining shared audio data in a shared audio-video and remote audio data received from a remote end; obtaining first audio data from an audio conversion device and second audio data collected by an audio collection device; the first audio data is obtained by inputting the shared audio data and the remote audio data into the audio conversion device; the second audio data contains the first audio data played by an audio playing device; performing echo cancellation on the second audio data with the first audio data as reference data to obtain main echo cancellation data; performing mixing operation based on the main echo cancellation data and the shared audio data in the first audio data to obtain near-end audio data.

2. The method of claim 1, wherein, The first audio data is obtained by: performing mixing operation based on a preset ultrasonic signal and the remote audio data to obtain reference audio data; obtaining the first audio data based on the reference audio data and the shared audio data.

3. The method of claim 2, wherein, after obtaining the reference audio data, the method further comprises: determining a time when the reference audio data is obtained as a reference time; the echo cancellation on the second audio data with the first audio data as reference data to obtain main echo cancellation data comprises: if a first delay is less than a second delay determined by a main echo cancellation unit, performing echo cancellation on the second audio data with the first audio data as reference data to obtain main echo cancellation data; wherein the first delay is a delay between the reference time and a time when the audio conversion device outputs the first audio data obtained by a secondary echo cancellation unit, and the second delay is a delay between the reference time and a time when the second audio data is collected by the audio collection device.

4. The method of claim 3, wherein, The method further comprises: performing matched filtering processing on the ultrasonic signal in the first audio data to determine a time when the first audio data is obtained; performing matched filtering processing on the ultrasonic signal in the second audio data to determine a time when the second audio data is collected.

5. The method according to any one of claims 2-4, wherein, The mixing operation based on the main echo cancellation data and the shared audio data in the first audio data to obtain near-end audio data comprises: performing echo cancellation on the first audio data with the reference audio data as reference data to obtain secondary echo cancellation data containing the shared audio data; performing mixing operation on the main echo cancellation data and the secondary echo cancellation data to obtain near-end audio data.

6. The method of claim 5, wherein, The second audio data further contains local audio data; The echo cancellation on the second audio data with the first audio data as reference data to obtain main echo cancellation data comprises: performing echo cancellation on the second audio data with the first audio data as reference data, and taking the local audio data obtained after processing as main echo cancellation data.

7. The method according to any one of claims 3-6, wherein, The method further comprises: If a first delay between the reference time and the acquisition time of the first audio data is greater than or equal to a second delay between the reference time and the collection time of the second audio data, the second audio data is processed by echo cancellation with the reference audio data as reference data to obtain main echo cancellation data.

8. An electronic device comprising: The processor, the memory, the communication interface and the communication bus complete communication among each other through the communication bus; The memory is used for storing at least one executable instruction, and the executable instruction causes the processor to execute the corresponding operation of the method according to any one of claims 1-7.

9. A computer storage medium, having stored thereon a computer program, which is executed by a processor to implement the method according to any one of claims 1-7.

10. A computer program product, comprising computer instructions, which instruct a computing device to execute the operation corresponding to the method according to any one of claims 1-7.

Images