Audio resource echo cancellation method and device, storage medium and electronic device

Abstract

The present application discloses an echo elimination method and device for audio resources, a storage medium and an electronic device, and relates to the field of smart home technology. The method comprises: obtaining a first audio resource and a second audio resource, wherein the first audio resource is an audio resource collected by a target device during the process of playing the second audio resource; inputting the first audio resource and the second audio resource into a target parameter generation network to obtain a target filter parameter output by the target parameter generation network; filtering the first audio resource using the target filter parameter and the target filter to obtain a target audio resource. The above technical solution is adopted to solve the problems of low echo elimination efficiency of audio resources in related technologies.

Description

Technical Field

[0001] The present application relates to the field of smart home technology, and more specifically, to an echo elimination method and device for audio resources, a storage medium, and an electronic device. Background Art

[0002] The development of science and technology has promoted the widespread application of audio acquisition technology, such as in Internet telephony, video conferencing, human-computer voice interaction and other fields. In these fields, certain requirements are placed on the quality of the collected audio resources. Interference audio (such as echo, noise, reverberation, etc.) is an important factor affecting the quality of audio acquisition and affects the quality of audio resources. For example, the impact of echo, that is, the audio resources played by the device are collected by itself, thus forming an echo, which seriously affects the quality of audio resource collection. How to eliminate interference audio in the collected audio resources has become a research focus for technicians in this field.

[0003] Currently, the common way to eliminate interfering audio is to build an audio elimination network model, that is, by training the initial audio elimination network model, the trained audio elimination network model is then input into the trained audio elimination network model, and then the audio resources collected by the device and the audio resources played by the device are input into the trained audio elimination network model, thereby obtaining clean audio resources after eliminating the interfering audio. However, although this method eliminates interfering audio, the problem is that the model is highly complex, the number of parameters is huge, and it consumes a lot of computing resources, so these methods cannot be run on platforms with limited computing resources.

[0004] With regard to the problems in related technologies such as low echo cancellation efficiency for audio resources, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide a method and device for echo cancellation of audio resources, a storage medium, and an electronic device, so as to at least solve the problem of low efficiency of echo cancellation of audio resources in the related art.

[0006] According to an embodiment of the embodiments of the present application, a method for echo cancellation of an audio resource is provided, comprising: acquiring a first audio resource and a second audio resource, wherein the first audio resource is an audio resource collected by a target device during playback of the second audio resource; inputting the first audio resource and the second audio resource into a target parameter generation network to obtain target filtering parameters output by the target parameter generation network, wherein the target parameter generation network is obtained by training an echo cancellation model using training samples labeled with echo cancellation audio, the echo cancellation model comprises an initial parameter generation network and a target filter connected in sequence, the training samples comprise a first audio sample and a second audio sample, the first audio sample is an audio resource collected by the target device during playback of the second audio sample, and the echo cancellation audio is an audio resource in which the second audio sample is eliminated from the first audio sample; filtering the first audio resource using the target filtering parameters and the target filter to obtain the target audio resource.

[0007] Optionally, the step of inputting the first audio resource and the second audio resource into a target parameter generation network to obtain target filter parameters output by the target parameter generation network includes: inputting the first audio resource and the second audio resource into the target parameter generation network; obtaining a first filter parameter output by a first branch of the target parameter generation network, and a second filter parameter output by a second branch of the target parameter generation network, wherein the first filter parameter is an operating parameter of the target filter, and the second filter parameter is used to characterize characteristics of an echo component carried in the first audio resource and caused by the second audio resource.

[0008] Optionally, before inputting the first audio resource and the second audio resource into the target parameter generation network to obtain the target filtering parameters output by the target parameter generation network, the method also includes: inputting the training sample into the initial parameter generation network to obtain the filtering parameter results output by the initial parameter generation network; using the filtering parameter results and the target filter to filter the first audio sample to obtain an audio resource result; adjusting the network parameters of the initial parameter generation network according to the loss value between the audio resource result and the echo cancellation audio until the network converges to obtain the target parameter generation network.

[0009] Optionally, the step of inputting the training sample into the initial parameter generation network to obtain a filtering parameter result output by the initial parameter generation network includes: inputting the training sample into the initial parameter generation network; obtaining a first result output by a first branch of the initial parameter generation network and a second result output by a second branch of the initial parameter generation network as the filtering parameter result, wherein the first branch of the initial parameter generation network is used to estimate the noise variance of the target filter, and the second branch of the initial parameter generation network is used to estimate the echo component carried in the first audio sample and caused by the second audio sample.

[0010] Optionally, the using the filtering parameter result and the target filter to filter the first audio sample to obtain an audio resource result includes: using the second result to extract a reference echo component carried in the first audio sample, wherein the reference echo component is caused by the second audio sample; and using the target filter with the first result as an operating parameter to filter the reference echo component to obtain the audio resource result.

[0011] Optionally, the use of the target filtering parameters and the target filter to filter the first audio resource to obtain the target audio resource includes: using a second filtering parameter included in the target filtering parameters to extract target audio features in the first audio resource, wherein the second filtering parameters are used to characterize features of an echo component carried in the first audio resource and caused by the second audio resource; and inputting the target audio features into the target filter with the first filtering parameters as operating parameters to obtain the target audio.

[0012] Optionally, the use of the second filtering parameters included in the target filtering parameters to extract the target audio features in the first audio resource includes: calculating the product of an image mask and the spectrum graph of the first audio resource to obtain a feature spectrum graph, wherein the second filtering parameters include the image mask, and the image mask is used to characterize the area in the spectrum graph that includes the target audio features; determining the spectrum features recorded in the feature spectrum graph as the target audio features.

[0013] According to another embodiment of the embodiment of the present application, an echo cancellation device for an audio resource is also provided, including: an acquisition module, used to acquire a first audio resource and a second audio resource, wherein the first audio resource is an audio resource collected by a target device during the process of playing the second audio resource; a first input module, used to input the first audio resource and the second audio resource into a target parameter generation network to obtain a target filtering parameter output by the target parameter generation network, wherein the target parameter generation network is obtained by training an echo cancellation model using training samples labeled with echo cancellation audio, the echo cancellation model includes an initial parameter generation network and a target filter connected in sequence, the training samples include a first audio sample and a second audio sample, the first audio sample is an audio resource collected by the target device during the process of playing the second audio sample, and the echo cancellation audio is an audio resource in which the second audio sample is eliminated from the first audio sample; a first filtering module, used to filter the first audio resource using the target filtering parameters and the target filter to obtain the target audio resource.

[0014] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned echo elimination method of the audio resource when running.

[0015] According to another aspect of an embodiment of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the echo elimination method of the audio resource through the computer program.

[0016] In an embodiment of the present application, a first audio resource and a second audio resource are obtained, wherein the first audio resource is an audio resource collected by a target device during the process of playing the second audio resource; the first audio resource and the second audio resource are input into a target parameter generation network to obtain target filter parameters output by the target parameter generation network, wherein the target parameter generation network is obtained by training an echo cancellation model using training samples labeled with echo cancellation audio, the echo cancellation model includes an initial parameter generation network and a target filter connected in sequence, the training samples include a first audio sample and a second audio sample, the first audio sample is an audio resource collected by a target device during the process of playing the second audio sample, and the echo cancellation audio is an audio resource in which the second audio sample is eliminated from the first audio sample; the first audio resource is filtered using the target filter parameters and the target filter to obtain the target audio resource, that is, the echo cancellation model includes an initial parameter generation network and a target filter connected in sequence, By using training samples with echo cancellation audio labeled to train the echo cancellation model, a trained target parameter generation network is obtained. Then, when performing echo cancellation, the collected first audio resource and the played second audio resource are input into the target parameter generation network. The target parameter output network outputs the target filter parameters used by the target filter. Then, the target filter parameters and the target filter can be used to filter the first audio resource to obtain the target audio resource after echo cancellation. That is to say, by using training samples to train the initial parameter generation network in the echo cancellation model, a target parameter generation network is obtained, so that the network can output the target filter parameters used by the target filter, instead of enabling the trained network model to directly output the filtered target audio resource, thereby reducing the complexity of the model and the amount of data processing in the model construction process, and achieving the operation of completing the echo cancellation of the first audio resource with limited computing resources. The above technical solution solves the problems of low echo cancellation efficiency of audio resources in the related technology, and achieves the technical effect of improving the echo cancellation efficiency of audio resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0019] Figure 1is a hardware environment diagram of an echo cancellation method for audio resources according to an embodiment of the present application;

[0020] Figure 2 is a flow chart of a method for echo cancellation of an audio resource according to an embodiment of the present application;

[0021] Figure 3 is an optional target echo cancellation flow chart according to an embodiment of the present application;

[0022] Figure 4 is a schematic diagram of a network structure generated by an optional initial parameter according to an embodiment of the present application;

[0023] Figure 5 It is a structural block diagram of an echo elimination device for audio resources according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] According to one aspect of an embodiment of the present application, a method for echo cancellation of audio resources is provided. The method is widely used in smart home (Smart Home), smart home, smart home equipment ecology, smart residential (IntelligenceHouse) ecology and other whole-house intelligent digital control application scenarios. Optionally, in this embodiment, Figure 1 is a hardware environment diagram of an echo cancellation method for audio resources according to an embodiment of the present application. The above method can be applied to Figure 1In the hardware environment composed of the terminal device 102 and the server 104 shown in FIG. Figure 1 As shown, the server 104 is connected to the terminal device 102 via a network, and can be used to provide services (such as application services, etc.) for the terminal or a client installed on the terminal. A database can be set on the server or independently of the server to provide data storage services for the server 104. Cloud computing and / or edge computing services can be configured on the server or independently of the server to provide data computing services for the server 104.

[0027] The network may include but is not limited to at least one of the following: wired network, wireless network. The wired network may include but is not limited to at least one of the following: wide area network, metropolitan area network, local area network, and the wireless network may include but is not limited to at least one of the following: WIFI (Wireless Fidelity), Bluetooth. The terminal device 102 may be but is not limited to a PC, a mobile phone, a tablet computer, a smart air conditioner, a smart range hood, a smart refrigerator, a smart oven, a smart stove, a smart washing machine, a smart water heater, a smart washing device, a smart dishwasher, a smart projection device, a smart TV, a smart clothes drying rack, a smart curtain, a smart audio and video, a smart socket, a smart speaker, a smart fresh air device, a smart kitchen and bathroom device, a smart bathroom device, a smart sweeping robot, a smart window cleaning robot, a smart mopping robot, a smart air purification device, a smart steamer, a smart microwave oven, a smart kitchen treasure, a smart purifier, a smart water dispenser, a smart door lock, etc.

[0028] In this embodiment, a method for canceling echo of an audio resource is provided, which is applied to the above-mentioned device terminal. Figure 2 is a flowchart of an echo cancellation method for an audio resource according to an embodiment of the present application, such as Figure 2 As shown, the process includes the following steps:

[0029] Step S202, obtaining a first audio resource and a second audio resource, wherein the first audio resource is an audio resource collected by the target device during the process of playing the second audio resource;

[0030] Step S204: input the first audio resource and the second audio resource into a target parameter generation network to obtain a target filter parameter output by the target parameter generation network, wherein the target parameter generation network is obtained by training an echo cancellation model using a training sample labeled with echo cancellation audio, the echo cancellation model includes an initial parameter generation network and a target filter connected in sequence, the training sample includes a first audio sample and a second audio sample, the first audio sample is an audio resource collected by the target device during the process of playing the second audio sample, and the echo cancellation audio is an audio resource in which the second audio sample is eliminated from the first audio sample;

[0031] Step S206: Filter the first audio resource using the target filtering parameter and the target filter to obtain a target audio resource.

[0032] Through the above steps, the echo cancellation model includes an initial parameter generation network and a target filter connected in sequence. The echo cancellation model is trained by using training samples marked with echo cancellation audio, thereby obtaining a trained target parameter generation network. Then, when performing echo cancellation, the collected first audio resource and the played second audio resource are input into the target parameter generation network. The target parameter output network outputs the target filter parameters used by the target filter. Then, the first audio resource can be filtered using the target filter parameters and the target filter to obtain the target audio resource after echo cancellation. That is to say, the initial parameter generation network in the echo cancellation model is trained by using training samples to obtain a target parameter generation network, so that the network can output the target filter parameters used by the target filter, instead of enabling the trained network model to directly output the filtered target audio resource, thereby reducing the complexity of the model and the amount of data processing in the model construction process, and realizing the operation of completing the echo cancellation of the first audio resource through limited computing resources. The above technical solution solves the problems of low echo cancellation efficiency of audio resources in the related technology, and achieves the technical effect of improving the echo cancellation efficiency of audio resources.

[0033] In the technical solution provided in the above step S202, the second audio resource is the audio played on the target device.

[0034] Optionally, in this embodiment, the audio resource is data representing audio content, and the audio resource may include but is not limited to the audio itself, a spectrogram of the audio, and the like.

[0035] Optionally, in this embodiment, the target device is a device with audio playback function and audio collection function, for example, the target device may be a phone, a smart TV, a smart speaker, etc.

[0036] In the technical solution provided in the above step S204, the target filtering parameters are parameters used by the target filter in the process of using the target filter to filter out the echo in the first audio resource, and may include but are not limited to parameters used to indicate the operating state of the filter and parameters used to characterize the characteristics of the echo component in the first audio.

[0037] Optionally, in this embodiment, using training samples to train the echo cancellation model may be, but is not limited to, inputting the first audio sample and the second audio sample into an initial parameter generation network in the echo cancellation model to obtain a filter parameter result output by the initial parameter generation network, the filter parameter result being used to filter out the audio of the echo component caused by the second audio sample, and then inputting the filter parameter result into a target filter, using the target filter and the filter parameter result to filter the first audio sample to obtain an audio resource result after eliminating the audio sample caused by the second audio sample in the first audio sample, and then adjusting the network parameters of the initial parameter generation network according to the audio resource result output by the target filter and the echo cancellation audio, for example, calculating the loss value of the initial parameter generation network according to the audio resource result and the echo cancellation audio, performing gradient update on the parameters of the initial parameter generation network according to the loss value, and stopping updating the network parameters of the initial parameter generation network when the loss value is less than a target threshold value to obtain a target parameter generation network.

[0038] In the technical solution provided in the above step S206, the target filter can, but is not limited to, simulate the echo path of the real scene according to the target filtering parameters, and then estimate the audio of the echo component caused by the second audio resource in combination with the second audio resource, and then subtract the audio of the echo component from the first audio resource to achieve the purpose of filtering the first audio resource.

[0039] Optionally, in this embodiment, the target filter may be, but is not limited to, an adaptive filter such as a Kalman filter or a Wiener filter.

[0040] As an optional embodiment, inputting the first audio resource and the second audio resource into a target parameter generation network to obtain a target filtering parameter output by the target parameter generation network includes:

[0041] Inputting the first audio resource and the second audio resource into the target parameter generation network;

[0042] Obtain a first filter parameter output by a first branch of the target parameter generation network, and a second filter parameter output by a second branch of the target parameter generation network, wherein the first filter parameter is an operating parameter of the target filter, and the second filter parameter is used to characterize the characteristics of an echo component carried in the first audio resource and caused by the second audio resource.

[0043] Optionally, in this embodiment, the echo component caused by the second audio resource is audio formed after the second audio resource is played and propagated in space, and may include but is not limited to echo, reverberation, noise, etc. formed by audio propagation.

[0044] Optionally, in this embodiment, the characteristics of the echo component may be spectral characteristics of the audio corresponding to the echo component, or may also be characteristics such as the position of the audio corresponding to the echo component in the first audio resource, the position of the spectrum of the audio corresponding to the echo component in the spectrum of the first audio resource, and thus the echo component in the first audio resource can be determined according to the second filtering parameters.

[0045] Optionally, in this embodiment, the target filter can be but is not limited to being used to simulate the echo path of a real scene using the first filtering parameter and the echo component, and then combined with the second audio resource to estimate the echo audio, so that the echo audio in the first audio resource can be filtered out. Figure 3 is an optional target echo cancellation flow chart according to an embodiment of the present application, such as Figure 3 As shown, the steps may include but are not limited to:

[0046] S301, obtaining a second audio resource played by a target device;

[0047] S302, when the target device plays the second audio resource, obtaining a first audio resource in the environment collected by the target device;

[0048] S303, performing frame division, windowing, and short-time Fourier transformation on the first audio resource and the second audio resource, and inputting the transformed results into a target parameter generation network in an echo cancellation model;

[0049] S304, obtaining a first filter parameter output by the first branch of the target parameter generation network for indicating the operation of the target filter, where the first filter parameter may be, but is not limited to, a noise variance (covariance matrix) estimated based on the first audio resource and the second audio resource. If the estimated first filter parameters are different, the filtering function of the target filter after using the first filter parameter is also different;

[0050] S305, obtaining a second filtering parameter output by the second branch of the target parameter generation network for characterizing a feature of an echo component (which may include but is not limited to an echo, noise, reverberation, etc. caused by transmission of the second audio resource) carried in the first audio resource, where the second filtering parameter may be but is not limited to an image mask for indicating the echo component;

[0051] S306, calculating the product of the image mask and the first audio resource to obtain an echo component, which can be subsequently used by a target filter to estimate a linear transfer function between clean audio without an echo component and the echo audio;

[0052] S307, inputting the echo component and the first filter parameter into a target filter, where the target filter may be, but is not limited to, used to simulate an echo path of a real scene using the first filter parameter and the echo component, and then estimate the echo audio in combination with the second audio resource, so as to filter out the echo audio in the first audio resource;

[0053] S308: Acquire the target audio resource output by the target filter.

[0054] Through the above embodiment, through the combination of the target parameter generation network and the target filter, the target parameter generation network outputs the target filter parameters used by the target filter, so that the target filter uses the target filter parameters to filter the collected audio resources, instead of directly outputting the filtered audio through the trained network model in the related art. The above embodiment simplifies the estimation task of the target parameter generation network to achieve the purpose of reducing the number of parameters while maintaining good noise reduction performance, reducing the amount of data computing resources occupied during model construction and use, and the method only needs to use the audio of the past frame, so that the time delay in the system is low.

[0055] In the above embodiment, the target parameter generation network is obtained by training the echo cancellation model using training samples labeled with echo cancellation audio. The echo cancellation model includes an initial parameter generation network and a target filter connected in sequence. The training sample input value is input into the initial parameter generation network, and the training sample is filtered according to the filter parameter result and the filter output by the initial parameter generation network to obtain the audio resource result output by the target filter. Then, the network parameters of the initial parameter generation network are adjusted according to the audio resource result and the echo cancellation audio. That is, the network parameters of the initial parameter generation network are adjusted during the entire training process, so that the initial parameter generation network can output the filter parameters used by the target filter, thereby reducing the problems of large parameters and high data calculation amount in the model training process. Figure 4 is a schematic diagram of a network structure generated by an optional initial parameter according to an embodiment of the present application, such as Figure 4As shown, the initial parameter generation model includes a recurrent neural network with three gated recurrent units as hidden layers, and two parameter output branches (each branch includes a fully connected layer and a sigma activation function), and the training sample of the echo-cancelled audio is labeled to include a first audio sample and a second audio sample (the first audio sample is an audio resource collected by the target device during the process of playing the second audio sample, and the echo-cancelled audio is an audio resource in which the second audio sample is eliminated from the first audio sample). The training sample is input into the recurrent neural network with three gated recurrent units as hidden layers, and after passing through the recurrent neural network, it is input into two different branches, both of which are composed of a The difference is that the first branch of the initial parameter generation network is used to estimate the noise variance of the target filter, and the second branch of the initial parameter generation network is used to estimate the echo component caused by the second audio sample carried in the first audio sample. The estimated echo component and the noise variance pass through the target filter to obtain the estimated clean speech. The estimated clean speech audio and the echo-cancelled audio used as a label are used to calculate the network loss. The loss function adopts the minimum mean square error method. The estimated loss is fed back to the initial parameter production network along with the differentiable Kalman filter for gradient update, thereby completing the training of the initial parameter generation network.

[0056] As an optional embodiment, before inputting the first audio resource and the second audio resource into a target parameter generation network to obtain a target filtering parameter output by the target parameter generation network, the method further includes:

[0057] Inputting the training sample into the initial parameter generation network to obtain the filtering parameter result output by the initial parameter generation network;

[0058] Filter the first audio sample using the filtering parameter result and the target filter to obtain an audio resource result;

[0059] The network parameters of the initial parameter generation network are adjusted according to the loss value between the audio resource result and the echo cancellation audio until the network converges to obtain the target parameter generation network.

[0060] Optionally, in this embodiment, the audio resource result may be, but is not limited to, audio with echo components filtered out, or a corresponding frequency spectrum graph of the audio with echo components filtered out.

[0061] Optionally, in this embodiment, the loss value between the audio resource result and the echo-cancelled audio may be, but is not limited to, calculated using a loss function on the audio resource result and the echo-cancelled audio, and the loss function may be, but is not limited to, a mean square error loss function.

[0062] Optionally, in this embodiment, when the loss value between the audio resource result and the echo-cancelled audio is less than or equal to a target value, the network converges.

[0063] As an optional embodiment, inputting the training sample into the initial parameter generation network to obtain the filtering parameter result output by the initial parameter generation network includes:

[0064] Inputting the training samples into the initial parameter generation network;

[0065] A first result output by a first branch of the initial parameter generation network and a second result output by a second branch of the initial parameter generation network are obtained as the filtering parameter result, wherein the first branch of the initial parameter generation network is used to estimate the noise variance of the target filter, and the second branch of the initial parameter generation network is used to estimate the echo component caused by the second audio sample and carried in the first audio sample.

[0066] Optionally, in this embodiment, the second branch of the initial parameter generation network can directly estimate the echo component caused by the second audio sample, or estimate the parameters used to extract the echo component caused by the second audio in the first audio sample. The parameters can be but are not limited to being used to indicate the characteristics of the echo component in the first audio sample, such as the position of the echo component in the first audio sample, the spectral characteristics of the echo component, etc., and then the echo component can be determined in the first audio sample based on the parameters.

[0067] Optionally, in this embodiment, the first branch of the initial parameter generation network is used to estimate the noise variance that matches the target filter. Different types of target filters are used, and the estimated noise variance is different. For example, the noise variance can be a covariance matrix estimated based on audio features of the first audio resource and the second audio resource. When different filters are used, the matrix parameters in the estimated covariance matrix are different.

[0068] As an optional embodiment, filtering the first audio sample using the filtering parameter result and the target filter to obtain an audio resource result includes:

[0069] extracting a reference echo component carried in the first audio sample using the second result, wherein the reference echo component is caused by the second audio sample;

[0070] The reference echo component is filtered using the target filter with the first result as an operating parameter to obtain the audio resource result.

[0071] Optionally, in this embodiment, the reference echo component may include, but is not limited to, echo, reverberation, noise, etc. caused by the propagation of the second audio sample.

[0072] Optionally, in this embodiment, the second result may be an image mask estimated by using the initial parameter generation network, and the image mask may be used for but is not limited to indicating the characteristics of the echo component caused by the second audio sample. Extracting the reference echo component carried in the first audio sample using the second result may be by calculating the product of the second result and the first audio sample. For example, the second result may be a spectral feature indicating the echo component, position information of the echo component in the first audio, etc., and then calculating the product of the second result and the first audio sample to obtain the echo component.

[0073] As an optional embodiment, filtering the first audio resource using the target filtering parameter and the target filter to obtain the target audio resource includes:

[0074] Extracting a target audio feature in the first audio resource using a second filter parameter included in the target filter parameter, wherein the second filter parameter is used to characterize a feature of an echo component carried in the first audio resource and caused by the second audio resource;

[0075] The target audio feature is input into the target filter with the first filtering parameter as the operating parameter to obtain the target audio.

[0076] Optionally, in this embodiment, the target audio feature is a feature of an echo component caused by the second audio resource.

[0077] Optionally, in this embodiment, after the target audio feature is input into a target filter with the first filtering parameter as the operating parameter, the target filter can filter out the audio with the target audio feature from the first audio resource, thereby obtaining the target audio.

[0078] As an optional embodiment, the extracting the target audio feature in the first audio resource using the second filter parameter included in the target filter parameter includes:

[0079] Calculating the product of the image mask and the spectrogram of the first audio resource to obtain a feature spectrogram, wherein the second filtering parameter includes the image mask, and the image mask is used to characterize a region in the spectrogram that includes the target audio feature;

[0080] The frequency spectrum features recorded in the characteristic spectrum diagram are determined as the target audio features.

[0081] Optionally, in this embodiment, the image mask may be used to cover an area outside the target audio feature in the spectrum graph of the first audio resource, or may also be used to extract an area described by the target audio feature from the spectrum graph of the first audio resource.

[0082] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0083] Figure 5 is a structural block diagram of an echo cancellation device for audio resources according to an embodiment of the present application; Figure 5 As shown, it includes: an acquisition module 52, used to acquire a first audio resource and a second audio resource, wherein the first audio resource is an audio resource collected by the target device during the process of playing the second audio resource; a first input module 54, used to input the first audio resource and the second audio resource into a target parameter generation network to obtain a target filtering parameter output by the target parameter generation network, wherein the target parameter generation network is obtained by training an echo cancellation model using training samples labeled with echo cancellation audio, the echo cancellation model includes an initial parameter generation network and a target filter connected in sequence, the training samples include a first audio sample and a second audio sample, the first audio sample is an audio resource collected by the target device during the process of playing the second audio sample, and the echo cancellation audio is an audio resource obtained by eliminating the second audio sample from the first audio sample; a first filtering module 56, used to filter the first audio resource using the target filtering parameters and the target filter to obtain a target audio resource.

[0084] Through the above embodiment, the echo cancellation model includes an initial parameter generation network and a target filter connected in sequence. The echo cancellation model is trained by using training samples marked with echo cancellation audio, thereby obtaining a trained target parameter generation network. Then, when performing echo cancellation, the collected first audio resource and the played second audio resource are input into the target parameter generation network. The target parameter output network outputs the target filter parameters used by the target filter. Then, the first audio resource can be filtered using the target filter parameters and the target filter to obtain the target audio resource after echo cancellation. That is to say, the initial parameter generation network in the echo cancellation model is trained by using training samples to obtain a target parameter generation network, so that the network can output the target filter parameters used by the target filter, instead of enabling the trained network model to directly output the filtered target audio resource, thereby reducing the complexity of the model and the amount of data processing in the model construction process, and realizing the operation of completing the echo cancellation of the first audio resource through limited computing resources. The above technical solution solves the problems of low echo cancellation efficiency of audio resources in the related technology, and achieves the technical effect of improving the echo cancellation efficiency of audio resources.

[0085] Optionally, the first input module includes: a first input unit, used to input the first audio resource and the second audio resource into the target parameter generation network; a first acquisition unit, used to acquire a first filter parameter output by a first branch of the target parameter generation network, and a second filter parameter output by a second branch of the target parameter generation network, wherein the first filter parameter is an operating parameter of the target filter, and the second filter parameter is used to characterize the characteristics of the echo component carried by the first audio resource and caused by the second audio resource.

[0086] Optionally, the device also includes: a second input module, used to input the training sample into the initial parameter generation network before inputting the first audio resource and the second audio resource into the target parameter generation network to obtain the target filtering parameters output by the target parameter generation network, so as to obtain the filtering parameter results output by the initial parameter generation network; a second filtering module, used to filter the first audio sample using the filtering parameter results and the target filter to obtain an audio resource result; and an adjustment module, used to adjust the network parameters of the initial parameter generation network according to the loss value between the audio resource result and the echo cancellation audio until the network converges to obtain the target parameter generation network.

[0087] Optionally, the second input module includes: a second input unit, used to input the training sample into the initial parameter generation network; a second acquisition unit, used to acquire a first result output by a first branch of the initial parameter generation network and a second result output by a second branch of the initial parameter generation network as the filtering parameter result, wherein the first branch of the initial parameter generation network is used to estimate the noise variance of the target filter, and the second branch of the initial parameter generation network is used to estimate the echo component carried in the first audio sample and caused by the second audio sample.

[0088] Optionally, the second filtering module includes: an operation unit, used to operate the second result and the first audio sample to obtain an operation result; and a filtering unit, used to filter the operation result using the target filter with the first result as an operating parameter to obtain the audio resource result.

[0089] Optionally, the first filtering module includes: an extraction unit, used to extract target audio features in the first audio resource using a second filtering parameter included in the target filtering parameter, wherein the second filtering parameter is used to characterize features of an echo component carried in the first audio resource and caused by the second audio resource; and a third input unit, used to input the target audio features into the target filter with the first filtering parameter as an operating parameter to obtain the target audio.

[0090] Optionally, the extraction unit is used to: calculate the product of an image mask and the spectrum graph of the first audio resource to obtain a feature spectrum graph, wherein the second filtering parameters include the image mask, and the image mask is used to characterize the area in the spectrum graph that includes the target audio feature; and determine the spectrum feature recorded in the feature spectrum graph as the target audio feature.

[0091] An embodiment of the present application further provides a storage medium, which includes a stored program, wherein when the program is run, any of the above-mentioned methods for eliminating echoes of audio resources is executed.

[0092] Optionally, in this embodiment, the storage medium may be configured to store program codes for executing the following steps: obtaining a first audio resource and a second audio resource, wherein the first audio resource is an audio resource collected by the target device during the process of playing the second audio resource; inputting the first audio resource and the second audio resource into a target parameter generation network to obtain target filtering parameters output by the target parameter generation network, wherein the target parameter generation network is obtained by training an echo cancellation model using training samples labeled with echo cancellation audio, the echo cancellation model includes an initial parameter generation network and a target filter connected in sequence, the training samples include a first audio sample and a second audio sample, the first audio sample is an audio resource collected by the target device during the process of playing the second audio sample, and the echo cancellation audio is an audio resource in which the second audio sample is eliminated from the first audio sample; filtering the first audio resource using the target filtering parameters and the target filter to obtain the target audio resource.

[0093] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any of the above-mentioned audio resource echo cancellation method embodiments.

[0094] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0095] Optionally, in this embodiment, the above-mentioned processor can be configured to perform the following steps through a computer program: obtaining a first audio resource and a second audio resource, wherein the first audio resource is an audio resource collected by the target device during the process of playing the second audio resource; inputting the first audio resource and the second audio resource into a target parameter generation network to obtain target filtering parameters output by the target parameter generation network, wherein the target parameter generation network is obtained by training an echo cancellation model using training samples labeled with echo cancellation audio, the echo cancellation model includes an initial parameter generation network and a target filter connected in sequence, the training samples include a first audio sample and a second audio sample, the first audio sample is an audio resource collected by the target device during the process of playing the second audio sample, and the echo cancellation audio is an audio resource in which the second audio sample is eliminated from the first audio sample; filtering the first audio resource using the target filtering parameters and the target filter to obtain the target audio resource.

[0096] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store program codes.

[0097] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0098] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0099] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for echo cancellation of audio resources, characterized in that: include: Acquire a first audio resource and a second audio resource, wherein the first audio resource is an audio resource collected by the target device during the process of playing the second audio resource; Inputting the first audio resource and the second audio resource into a target parameter generation network to obtain target filtering parameters output by the target parameter generation network, wherein the target parameter generation network is obtained by training an echo cancellation model using training samples labeled with echo cancellation audio, the echo cancellation model includes an initial parameter generation network and a target filter connected in sequence, the training samples include a first audio sample and a second audio sample, the first audio sample is an audio resource collected by the target device during the process of playing the second audio sample, and the echo cancellation audio is an audio resource in which the second audio sample is eliminated from the first audio sample, and inputting the first audio resource and the second audio resource into the target parameter generation network to obtain target filtering parameters output by the target parameter generation network includes: inputting the first audio resource and the second audio resource into the A target parameter generation network; obtaining a first filter parameter output by a first branch of the target parameter generation network and a second filter parameter output by a second branch of the target parameter generation network, wherein the first filter parameter is an operating parameter of the target filter, and the second filter parameter is used to characterize the characteristics of an echo component caused by the second audio resource and carried in the first audio resource, wherein the characteristics of the echo component include the characteristics of the audio formed after the second audio resource is played and propagated in space; filtering the first audio resource using the target filter parameter and the target filter to obtain a target audio resource, wherein the target filter is used to simulate an echo path according to the target filter parameter, estimate the audio of the echo component caused by the second audio resource in combination with the second audio resource, and then filter the first audio resource by subtracting the audio of the echo component from the first audio resource.

2. The method according to claim 1, characterized in that Before inputting the first audio resource and the second audio resource into a target parameter generation network to obtain a target filtering parameter output by the target parameter generation network, the method further includes: Inputting the training sample into the initial parameter generation network to obtain the filtering parameter result output by the initial parameter generation network; Filter the first audio sample using the filtering parameter result and the target filter to obtain an audio resource result; The network parameters of the initial parameter generation network are adjusted according to the loss value between the audio resource result and the echo cancellation audio until the network converges to obtain the target parameter generation network.

3. The method according to claim 2, characterized in that The step of inputting the training sample into the initial parameter generation network to obtain the filtering parameter result output by the initial parameter generation network includes: Inputting the training samples into the initial parameter generation network; A first result output by a first branch of the initial parameter generation network and a second result output by a second branch of the initial parameter generation network are obtained as the filtering parameter result, wherein the first branch of the initial parameter generation network is used to estimate the noise variance of the target filter, and the second branch of the initial parameter generation network is used to estimate the echo component caused by the second audio sample and carried in the first audio sample.

4. The method according to claim 3, characterized in that The filtering the first audio sample using the filtering parameter result and the target filter to obtain an audio resource result includes: extracting a reference echo component carried in the first audio sample using the second result, wherein the reference echo component is caused by the second audio sample; The reference echo component is filtered using the target filter with the first result as an operating parameter to obtain the audio resource result.

5. The method according to claim 1, characterized in that The filtering the first audio resource using the target filtering parameter and the target filter to obtain a target audio resource includes: Extracting a target audio feature in the first audio resource using a second filter parameter included in the target filter parameter, wherein the second filter parameter is used to characterize a feature of an echo component carried in the first audio resource and caused by the second audio resource; The target audio feature is input into the target filter with the first filtering parameter as the operating parameter to obtain the target audio.

6. The method according to claim 5, characterized in that The extracting the target audio feature in the first audio resource using the second filter parameter included in the target filter parameter includes: Calculating the product of the image mask and the spectrogram of the first audio resource to obtain a feature spectrogram, wherein the second filtering parameter includes the image mask, and the image mask is used to characterize a region in the spectrogram that includes the target audio feature; The frequency spectrum features recorded in the characteristic spectrum diagram are determined as the target audio features.

7. An echo cancellation device for audio resources, characterized in that: include: An acquisition module, configured to acquire a first audio resource and a second audio resource, wherein the first audio resource is an audio resource collected by the target device during the process of playing the second audio resource; a first input module, configured to input the first audio resource and the second audio resource into a target parameter generation network, and obtain a target filter parameter output by the target parameter generation network, wherein the target parameter generation network is obtained by training an echo cancellation model using training samples labeled with echo cancellation audio, the echo cancellation model comprises an initial parameter generation network and a target filter connected in sequence, the training samples comprise a first audio sample and a second audio sample, the first audio sample is an audio resource collected by the target device during the process of playing the second audio sample, and the echo cancellation audio is an audio resource obtained by eliminating the second audio sample from the first audio sample, the first input module comprises: a first input unit, configured to input the first audio resource and the second audio resource into the target parameter generation network; a first acquisition unit, configured to acquire a first filter parameter output by a first branch of the target parameter generation network, and a second filter parameter output by a second branch of the target parameter generation network, wherein the first filter parameter is an operating parameter of the target filter, and the second filter parameter is used to characterize a feature of an echo component caused by the second audio resource carried in the first audio resource, the feature of the echo component comprises a feature of an audio formed after the second audio resource is played and propagated in space; A first filtering module is used to filter the first audio resource using the target filtering parameters and the target filter to obtain a target audio resource, wherein the target filter is used to simulate an echo path according to the target filtering parameters, estimate the audio of the echo component caused by the second audio resource in combination with the second audio resource, and then subtract the audio of the echo component from the first audio resource to filter the first audio resource.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the method according to any one of claims 1 to 6 when executed.

9. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 6 through the computer program.

Images