Echo cancellation method and device based on deep learning and readable storage medium

Abstract

The invention provides an echo cancellation method and device based on deep learning, and a readable storage medium, and the specific implementation scheme is as follows: obtaining a far-end microphone signal corresponding to a far-end room, and obtaining a near-end microphone signal corresponding to a near-end room; taking the far-end microphone signal as a reference signal, acquiring a first compressed complex spectrum corresponding to the reference signal, and acquiring a second compressed complex spectrum corresponding to the near-end microphone signal; inputting the first compressed complex spectrum and the second compressed complex spectrum into a trained neural network model for echo cancellation processing, and outputting a near-end voice compressed complex spectrum; and performing short-time inverse Fourier transform on the compressed complex spectrum of the near-end voice to obtain a pure near-end voice signal. Through the implementation of the application, the microphone signal of each far-end room is used as the neural network reference information to recover the near-end pure voice signal, and the echo component can be effectively suppressed, so that the echo suppression of the virtual space sound system is realized.

Description

[Technical field]

[0001] The present application relates to the field of artificial intelligence technology, in particular to the field of speech processing technology, and can be applied to echo suppression scenarios of virtual space sound systems. Specifically, the present application discloses an echo cancellation method, device, and readable storage medium based on deep learning. [Background technology]

[0002] With the development of modern communications, virtual reality and augmented reality technologies, users sometimes talk to different speakers in multiple rooms at the same time. In order to achieve the goal of "virtual reality", audio design sometimes requires reconstructing the information of speakers in different remote rooms to different positions in the near room, and the virtual position will also change accordingly with the movement of the actual position of the remote speaker, so as to satisfy the listener's auditory sense of reality. Auditory realism is an even more important factor in the design of the metaverse. It combines visual, tactile and other elements to provide users with a more realistic virtual environment. However, reverberation is inevitable in the actual acoustic environment, so there will usually be echoes during the call between the two ends, affecting the user's auditory experience.

[0003] It is worth noting that the techniques described in this section are not necessarily techniques that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any technique described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. [Summary of the invention]

[0004] The present application provides an echo cancellation method, device and readable storage medium based on deep learning, aiming to solve one of the problems in the related art at least to a certain extent.

[0005] In order to solve the above technical problems, the first aspect of the present application provides an echo cancellation method based on deep learning, comprising:

[0006] Acquire a far-end microphone signal corresponding to the far-end room, and acquire a near-end microphone signal corresponding to the near-end room;

[0007] Taking the far-end microphone signal as a reference signal, obtaining a first compressed complex spectrum corresponding to the reference signal, and obtaining a second compressed complex spectrum corresponding to the near-end microphone signal;

[0008] Inputting the first compressed complex spectrum and the second compressed complex spectrum into the trained neural network model for echo cancellation processing, and outputting the near-end speech compressed complex spectrum;

[0009] The near-end speech compressed complex spectrum is subjected to inverse short-time Fourier transform to obtain a pure near-end speech signal.

[0010] A second aspect of the present application provides an echo cancellation device based on deep learning, comprising:

[0011] A signal acquisition module, used to acquire a far-end microphone signal corresponding to a far-end room, and to acquire a near-end microphone signal corresponding to a near-end room;

[0012] A complex spectrum acquisition module, used to use the far-end microphone signal as a reference signal, acquire a first compressed complex spectrum corresponding to the reference signal, and acquire a second compressed complex spectrum corresponding to the near-end microphone signal;

[0013] A model processing module, used for inputting the first compressed complex spectrum and the second compressed complex spectrum into the trained neural network model for echo cancellation processing, and outputting a near-end speech compressed complex spectrum;

[0014] The signal conversion module is used to perform short-time Fourier inverse transform on the near-end speech compression complex spectrum to obtain a pure near-end speech signal.

[0015] The third aspect of the present application provides an electronic device, comprising: a memory and a processor, wherein the processor is used to execute a computer program stored in the memory; when the processor executes the computer program, it implements each step of the deep learning-based echo cancellation method provided in the first aspect of the present application.

[0016] The fourth aspect of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the deep learning-based echo cancellation method provided in the first aspect of the present application are implemented.

[0017] As can be seen from the above, according to the deep learning-based echo cancellation method, device and readable storage medium provided by the present application, the far-end microphone signal corresponding to the far-end room is obtained, and the near-end microphone signal corresponding to the near-end room is obtained; the far-end microphone signal is used as a reference signal to obtain the first compressed complex spectrum corresponding to the reference signal, and the second compressed complex spectrum corresponding to the near-end microphone signal is obtained; the first compressed complex spectrum and the second compressed complex spectrum are input into the trained neural network model for echo cancellation processing, and the near-end speech compressed complex spectrum is output; the near-end speech compressed complex spectrum is subjected to short-time Fourier inverse transform to obtain a pure near-end speech signal. Through the implementation of the present application, the microphone signals of each far-end room are used as neural network reference information to restore the near-end pure speech signal, which can effectively suppress the echo component, thereby realizing echo suppression of the virtual space sound system.

[0018] It should be understood that the content described in this section is not intended to identify the key or important features of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description.

Brief Description of the Drawings

[0019] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The drawings shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0020] Figure 1 A schematic diagram of a basic flow chart of an echo cancellation method based on deep learning provided in an embodiment of the present application;

[0021] Figure 2 A schematic diagram of an echo cancellation application scenario based on VBAP technology provided in one embodiment of the present application;

[0022] Figure 3 A schematic diagram of the structure of a neural network model provided in one embodiment of the present application;

[0023] Figure 4 A test result representation diagram corresponding to different speaker layouts provided in an embodiment of the present application;

[0024] Figure 5 A schematic diagram of an echo cancellation application scenario based on Ambisonic technology provided in an embodiment of the present application;

[0025] Figure 6 A representation diagram of test results of using different algorithms to process recorded data under different speaker layouts provided in an embodiment of the present application;

[0026] Figure 7 A schematic diagram of functional modules of an echo cancellation device based on deep learning provided in one embodiment of the present application;

[0027] Figure 8 A schematic diagram of the hardware structure of an electronic device provided in one embodiment of the present application. [Specific implementation method]

[0028] In order to make the purpose, features, and advantages of the invention of this application more obvious and easy to understand, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0029] In the description of the embodiments of the present application, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features; the term "multiple" means two or more, unless otherwise clearly and specifically defined; the term "including" indicates the presence of the described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or their collections; the term "and / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may include the existence of A alone, the existence of A and B at the same time, and the existence of B alone. The character " / " generally indicates that the previously associated objects belong to the "or" relationship.

[0030] In order to solve the problem of echo in the virtual space sound system provided by the related art, an embodiment of the present application provides an echo cancellation method based on deep learning, such as Figure 1 A basic flow chart of the deep learning-based echo cancellation method provided in this embodiment, the deep learning-based echo cancellation method includes the following steps:

[0031] Step 101: Acquire a far-end microphone signal corresponding to a far-end room, and acquire a near-end microphone signal corresponding to a near-end room.

[0032] Specifically, in actual application scenarios, it is necessary to reconstruct the voice of the speaker in the far-end room to different positions in the near-end room. It is worth mentioning that the number of far-end rooms can be K, and the value of K is greater than or equal to 1. The near-end room is equipped with a speaker array, and the speaker array can be arranged in a ring. In this embodiment, when the speaker in the far-end room speaks, the speaker's voice can be reconstructed based on the speaker in the near-end room, and a plurality of virtual sound source positions can be set in the near-end room, each virtual sound source position can be the position of a certain speaker, or can be a position between two adjacent speakers. It should also be noted that speakers and microphones can be set in both the near-end room and the far-end room, and the voice signal played by the speaker is recorded by the microphone through the acoustic echo path.

[0033] Vector basis amplitude translation (VBAP) technology is a virtual sound and image synthesis technology. Compared with the sound field synthesis method, it requires fewer speakers and the speaker placement method is more flexible. In addition, VBAP technology can be applied to different audio systems, such as stereo, surround sound, spatial sound systems, etc., and it has been incorporated into the MPEG-H standard. In this embodiment, for the speaker array on the ring, when using VBAP technology to play back the virtual sound source, two speakers on both sides of the virtual sound source are usually selected to achieve it. For scenes where the virtual sound source position coincides with the speaker position, the speaker at the virtual sound source position is directly used for playback.

[0034] like Figure 2 FIG. 1 is a schematic diagram of an echo cancellation application scenario based on the VBAP technology provided by this embodiment. k (n) represents the speech signal of the speaker in the kth remote room, In some implementations of this embodiment, the step of obtaining the near-end microphone signal corresponding to the near-end room includes: obtaining the near-end speaker signal corresponding to the k-th speaker in the near-end room; and obtaining the corresponding near-end microphone signal according to the near-end speaker signal.

[0035] The near-end loudspeaker signal is represented as:

[0036]

[0037] Where K represents the number of remote rooms, δ p,k represents the sound source signal gain of the p-th speaker for the k-th remote room obtained by the VBAP algorithm, represents the far-end loudspeaker signal of one channel in the k-th far-end room;

[0038] The near-end microphone signal is expressed as:

[0039]

[0040] Where P is the number of loudspeakers arranged in the near-end room array, h p (n) represents the echo path, s(n) represents the near-end speech signal, and v(n) represents the additive noise.

[0041] Step 102: Use the far-end microphone signal as a reference signal, obtain a first compressed complex spectrum corresponding to the reference signal, and obtain a second compressed complex spectrum corresponding to the near-end microphone signal.

[0042] Specifically, before the far-end microphone signal and the near-end microphone signal are input into the neural network model, the present embodiment first performs short-time Fourier transform on the two signals, that is, converts the signals from the time domain to the time-frequency domain, and obtains the compressed complex spectra corresponding to the two signals.

[0043] In some implementations of this embodiment, the first compressed complex spectrum is expressed as:

[0044]

[0045] in, represents the first compressed complex spectrum, and θ represent the amplitude information and phase information of the far-end microphone signal corresponding to the k-th far-end room respectively, β is a preset constant, and the preferred value of β may be 1 / 2.

[0046] Step 103: input the first compressed complex spectrum and the second compressed complex spectrum into the trained neural network model for echo cancellation processing, and output the near-end speech compressed complex spectrum.

[0047] like Figure 3 The figure shows a schematic diagram of the structure of a neural network model provided in this embodiment. Specifically, the neural network model of this embodiment includes: a sequentially connected encoding network (Encoder), a timing modeling network, a decoding network (Decoder) and a linear layer. The encoding network is also skip-connected with the decoding network (Skip Connection). In the preferred implementation of this embodiment, the timing modeling network includes two parallel long short-term memory networks (LSTM, Long Short-Term Memory).

[0048] The compressed complex spectrum includes real part and imaginary part information. In practical applications, this embodiment can be used to compress the first complex spectrum. and the second compressed complex spectrum After performing the fusion operation (Concatenate), the input features (Input features) of the neural network model are used. The parameters of the input features are [10, T, 161]. The encoding network of this embodiment can optionally include five cascaded gated convolutional layers (Conv_GLU). The parameters of the five gated convolutional layers are [16, T, 80], [32, T, 39], [64, T, 19], [128, T, 9], and [256, T, 4], respectively. Next, the signal features extracted by the encoding network are used as the input of the timing modeling network including two parallel long short-term memory networks. The timing relationship of the signal is obtained through the timing modeling network. The parameters of the long short-term memory network are [T, 1024]. The output features of the time series modeling network are used as the input of five gated deconvolution layers (Deconv_GLU). The parameters of the five cascaded gated deconvolution layers are [128, T, 9], [64, T, 19], [32, T, 39], [16, T, 80], and [1, T, 161]. The decoded features are processed by two parallel linear layers to obtain the overall output of the neural network model, that is, the near-end speech compressed complex spectrum. The parameters of the linear layer are [T, 161].

[0049] In this embodiment, the initial neural network model is trained based on preset training samples, and whether the training loss value meets the model convergence condition is determined based on a preset loss function. The loss function is used to measure the gap between the predicted output and the sample label in the training phase. If the model convergence condition is not met, the parameters of the neural network model are further adjusted and the iterative training is continued until the model converges, thereby obtaining a trained neural network model.

[0050] Following the above Figure 3 The neural network model, that is, the decoding network includes two decoding network branches, and the two decoding network branches are respectively connected to a linear layer; the loss function of the neural network model is expressed as:

[0051]

[0052] Among them, Loss represents the loss value, and They represent the prediction language output by the neural network model.

[0053] The real and imaginary parts of the sound signal, and They represent the real part and imaginary part indicated by the labels of the speech signal training samples respectively.

[0054] Step 104: Perform inverse short-time Fourier transform on the near-end speech compressed complex spectrum to obtain a pure near-end speech signal.

[0055] Finally, this embodiment performs short-time inverse Fourier transform on the near-end speech compressed complex spectrum output by the model to reconstruct a pure near-end speech signal, which effectively suppresses the echo component.

[0056] In order to better demonstrate the results of the echo cancellation solution based on VBAP and deep learning technology provided in the above-mentioned embodiment of the present application, an embodiment of the present application also provides a comparative verification experiment for illustration, as follows:

[0057] Assume that the number of far-end rooms in actual applications is 3, that is, K=3, which also means that in the same time period, at most three speakers can speak at the same time in multiple far-end rooms. The near-end room has a 5.1 surround sound speaker arrangement, and the experiment does not use its subwoofer channel (subwoofer), so P=5. The five speakers used in the experiment are of the same height and arranged on the same horizontal plane, forming a circular array, that is, the distance from the five speakers to the center of the circle is the same. In addition, the database used is the DNS database, and the room impulse response is generated by the mirror method. The experimental data covers the situations where there are one, two, and three speakers speaking at the same time at the far end. In addition, the experiment also compares the echo noise reduction algorithm based on the Transformer model, and the test indicators are the echo return loss gain (ERLE, Echo Return Loss Enhancement) and the objective speech quality evaluation index (PESQ, Perceptualevaluation of speech quality).

[0058] The model has good performance for the situation of five near-end speakers. In order to test the model for different speaker arrangements in the near-end room, six speakers evenly arranged on a circle are selected here, and the test condition of two speakers in the far-end room is selected, that is, the near-end room needs to replay two virtual sources, and the virtual source positions are set at 45° and 135°. For the arrangement of six speakers in the near-end room, the test results of the model of this embodiment and the Transformer model with the microphone signal as a reference are given below, as shown in FIG. Figure 4 FIG. 4 is a graph showing the test results corresponding to different speaker layouts provided in this embodiment. Figure 4 (a) is the speech spectrogram of the microphone receiving signal (PESQ = 1.95), Figure 4 (b) is the speech spectrum diagram of the pure near-end speech signal. Figure 4 (c) is the speech spectrum after processing based on the Transformer model (ERLE = 64.3dB, PESQ = 2.25). Figure 4(d) is a speech spectrum diagram after processing based on the model of this embodiment (ERLE=63.5dB, PESQ=2.54). Because for the arrangement of six speakers in the near-end room, from the processing results in the figure, the model of this embodiment and the Transformer model both have a good suppression effect on the echo component, but the speech distortion of the Transformer model is more serious. It is worth noting that when the number of far-end speakers is fixed at K=3, the model has no limit on the number of speakers in the near-end room, and it can be applied to the arrangement of speakers of any number and orientation.

[0059] In some implementations of this embodiment, before using the far-end microphone signal as a reference signal, the method further includes: obtaining the number of sound sources in all far-end rooms; if the number of sound sources is greater than or equal to a preset threshold (eg, 3), encoding the far-end microphone signal into a B-format.

[0060] In different actual application scenarios, the number of sound sources in the remote room may vary. For application scenarios with a small number of sound sources in the remote room, the echo cancellation solution based on VBAP and deep learning technology in the aforementioned embodiment may be referred to, and the single-channel microphone signals of multiple remote rooms (i.e., the remote microphone signals) may be used as reference signals. For application scenarios with a large number of sound sources in the remote room, this embodiment adopts an echo cancellation solution based on Ambisonics and deep learning technology. Different from the traditional method that usually uses the D-format signal as the reference signal, this embodiment encodes the microphone signals of multiple channels in the remote room into the B-format format (i.e., the first-order Ambisonics format). At this time, there is no upper limit on the number of remote rooms. Then, the format signal is used as the reference signal and the compressed complex spectrum is extracted and input into the neural network model to achieve the purpose of echo cancellation.

[0061] It is worth mentioning that Ambisonic technology is a classic method for spatial sound field reproduction. It is based on the spherical harmonic analysis theory of plane waves and uses spherical harmonic functions to encode and reconstruct spatial sound fields. Depending on the order of spherical harmonic functions, Ambisonic technology can use different numbers and arrangements of speakers to reproduce spatial sound fields, which has high flexibility and scalability.

[0062] like Figure 5 FIG. 1 is a schematic diagram of an echo cancellation application scenario based on the Ambisonic technology provided by this embodiment, and FIG. Figure 2 The difference between the schematic diagrams of the echo cancellation application scenarios based on VBAP technology shown is that the far-end microphone signal needs to be encoded into a reference signal in B-format format. The B-format reference signal has more channels than the ordinary reference signal. The other implementations of the two solutions can remain consistent.

[0063] In order to better demonstrate the results of the above-mentioned echo cancellation solution based on Ambisonic and deep learning technology provided in the above-mentioned embodiment of the present application, an embodiment of the present application also provides a comparative verification experiment for illustration, as follows:

[0064] Assuming that the number of speakers in the near-end room is four, the following three comparative experiments are set up. Experiment 1: The near-end room uses speakers with a fixed layout, that is, there is only one speaker mode, and this model is marked as D-format; Experiment 2: The speaker layout of the near-end room is not fixed, and the decoding mode is generated according to the actual layout during the generation of the near-end microphone signal, but the reference signal only uses the W channel of the B-format signal, and this model is marked as Singlechn; Experiment 3: The speaker layout of the near-end room is not fixed, and the decoding mode is generated according to the actual layout during the generation of the near-end microphone signal, and the reference signal only uses the B-format signal, and this model is marked as B-format.

[0065] In order to test the generalization ability of the model for flexible speaker layouts, a speaker layout that does not appear in the training sets of the three models is given, and experiments are conducted in real rooms. This experiment has only one far-end room and the number of speakers in the near-end room is 4, such as Figure 6 FIG. 1 is a graph showing the test results of processing recorded data using different algorithms under different speaker layouts provided in this embodiment. Figure 6 (a) is the speech spectrogram of the microphone receiving signal (PESQ = 2.45), Figure 6 (b) is the speech spectrum diagram of the pure near-end speech signal. Figure 6 (c) is a speech spectrum diagram based on the PBFDLMS algorithm (ERLE = 14.1dB, PESQ = 2.59). Figure 6 (d) is a speech spectrogram based on the D-format model of this embodiment (ERLE=17.7dB, PESQ=2.76), Figure 6 (e) is a speech spectrogram based on the Singlechn model (ERLE = 52.6dB, PESQ = 2.85). Figure 6 (f) is the speech spectrum diagram based on the B-format model (ERLE=54.7dB, PESQ=2.97).

[0066] From the above test results, it can be seen that if the speaker signal of a single layout is used as a reference, the model's generalization ability for unknown speaker layouts is insufficient. Using a channel of D-format as a reference can also achieve a certain echo suppression effect, but its performance is not as good as that of the B-format model, and the B-format model also demonstrates good model generalization ability.

[0067] In some implementations of the present embodiment, the echo cancellation method further includes: determining virtual sound source positions in the near-end room corresponding to multiple actual sound sources in the far-end room; generating multiple speech reconstruction instructions based on multiple pure near-end speech signals corresponding to the multiple actual sound sources; and sending speech reconstruction instructions to the speakers or speaker combinations corresponding to each virtual sound source position in the near-end room, respectively, to instruct the speakers in different positions in the near-end room to play the corresponding pure near-end speech signals.

[0068] Specifically, after recovering the pure near-end speech signal from the microphone received signal based on the neural network model, the embodiment determines the corresponding speaker / speaker combination according to the virtual sound source position of the near-end room, and then sends the corresponding pure near-end speech signal to the speakers in different directions, so that the speech signals of multiple speakers in the far-end room are reconstructed in the near-end room respectively, and the echo suppression of the virtual space sound system is also achieved. It is worth mentioning that multiple speakers can be arranged in a ring in the near-end room. If the virtual source is located between two speakers, the two speakers on both sides of the virtual source are controlled to reconstruct the corresponding actual sound source. If the virtual source coincides with the position of a single speaker, the single speaker at the virtual source position is controlled to reconstruct the corresponding actual sound source.

[0069] It should be understood that the size of the serial number of each step in this embodiment does not mean the order of execution of the steps. The execution order of each step should be determined by its function and internal logic, and should not constitute a sole limitation on the implementation process of the embodiment of this application.

[0070] Figure 7 An echo cancellation device based on deep learning is provided in one embodiment of the present application. The echo cancellation device can be used to implement the echo cancellation method in the aforementioned embodiment, and mainly includes:

[0071] The signal acquisition module 701 is used to acquire the far-end microphone signal corresponding to the far-end room and acquire the near-end microphone signal corresponding to the near-end room;

[0072] The complex spectrum acquisition module 702 is used to use the far-end microphone signal as a reference signal, acquire a first compressed complex spectrum corresponding to the reference signal, and acquire a second compressed complex spectrum corresponding to the near-end microphone signal;

[0073] The model processing module 703 is used to input the first compressed complex spectrum and the second compressed complex spectrum into the trained neural network model for echo cancellation processing, and output the near-end speech compressed complex spectrum;

[0074] The signal conversion module 704 is used to perform short-time inverse Fourier transform on the near-end speech compressed complex spectrum to obtain a pure near-end speech signal.

[0075] In an optional implementation of this embodiment, the echo cancellation device also includes: a signal encoding module, which is used to obtain the number of sound sources in all remote rooms before using the remote microphone signal as a reference signal; if the number of sound sources is greater than or equal to a preset number threshold, encode the remote microphone signal into a B-format format.

[0076] In an optional implementation of the present embodiment, the echo elimination device further includes: a speech reconstruction module, which is used to determine virtual sound source positions in the near-end room corresponding to multiple actual sound sources in the far-end room; generate multiple speech reconstruction instructions based on multiple pure near-end speech signals corresponding to the multiple actual sound sources; and send speech reconstruction instructions to the speakers or speaker combinations corresponding to each virtual sound source position in the near-end room, respectively, to instruct the speakers in different positions in the near-end room to play the corresponding pure near-end speech signals.

[0077] It should be noted that the echo cancellation methods in the aforementioned method embodiments can all be implemented based on the echo cancellation device provided in this embodiment. Ordinary technicians in the relevant field can clearly understand that for the convenience and conciseness of description, the specific working process of the echo cancellation device described in this embodiment can be implemented by referring to the corresponding working process in the aforementioned method embodiments, and will not be repeated here.

[0078] Based on the technical solution of the above-mentioned embodiment of the present application, the far-end microphone signal corresponding to the far-end room is obtained, and the near-end microphone signal corresponding to the near-end room is obtained; the far-end microphone signal is used as a reference signal to obtain a first compressed complex spectrum corresponding to the reference signal, and a second compressed complex spectrum corresponding to the near-end microphone signal is obtained; the first compressed complex spectrum and the second compressed complex spectrum are input into the trained neural network model for echo cancellation processing, and the near-end speech compressed complex spectrum is output; the near-end speech compressed complex spectrum is subjected to short-time inverse Fourier transform to obtain a pure near-end speech signal. Through the implementation of the present application, the microphone signals of each far-end room are used as neural network reference information to restore the near-end pure speech signal, which can effectively suppress the echo component, thereby realizing echo suppression of the virtual space sound system.

[0079] See also Figure 8 , Figure 8 An electronic device is provided in one embodiment of the present application. The electronic device can be used to implement the echo cancellation method based on deep learning in the above-mentioned embodiment. Figure 8 As shown, the electronic device mainly includes:

[0080] Memory 801, processor 802 and bus 803, memory 801 and processor 802 are connected via bus 803. Memory 801 stores a computer program that can be run on processor 802, and when processor 802 executes the computer program, the echo cancellation method based on deep learning in the above embodiment is implemented. The number of processors can be one or more.

[0081] The memory 801 may be a high-speed random access memory (RAM) memory, or a non-volatile memory, such as a disk memory. The memory 801 is used to store executable program codes, and the processor 802 is coupled to the memory 801 .

[0082] Furthermore, the present application also provides a computer-readable storage medium, which may be provided in the electronic device in the above embodiments. Figure 8 Memory in the illustrated embodiment.

[0083] The computer readable storage medium stores a computer program, and when the program is executed by the processor, the echo cancellation method based on deep learning in the aforementioned embodiment is implemented. Furthermore, the computer storable medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a RAM, a magnetic disk or an optical disk, and other media that can store program codes.

[0084] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0085] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0086] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0087] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a readable storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned readable storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0088] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0089] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0090] The above is a description of the deep learning-based echo cancellation method, device and readable storage medium provided in the present application. For technicians in this field, according to the ideas of the embodiments of the present application, there may be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A deep learning-based echo cancellation method, characterized in that: include: Acquire a far-end microphone signal corresponding to the far-end room, and acquire a near-end microphone signal corresponding to the near-end room; Taking the far-end microphone signal as a reference signal, obtaining a first compressed complex spectrum corresponding to the reference signal, and obtaining a second compressed complex spectrum corresponding to the near-end microphone signal; Inputting the first compressed complex spectrum and the second compressed complex spectrum into the trained neural network model for echo cancellation processing, and outputting the near-end speech compressed complex spectrum; The near-end speech compressed complex spectrum is subjected to inverse short-time Fourier transform to obtain a pure near-end speech signal.

2. The echo cancellation method according to claim 1, characterized in that: Before using the far-end microphone signal as a reference signal, the method further includes: Obtain the number of sound sources in all the remote rooms; If the number of the sound sources is greater than or equal to a preset number threshold, the far-end microphone signal is encoded into a B-format.

3. The echo cancellation method according to claim 1, characterized in that: The obtaining of the near-end microphone signal corresponding to the near-end room includes: Obtain a near-end speaker signal corresponding to the kth speaker in the near-end room; the near-end speaker signal is expressed as: Where K represents the number of remote rooms, δ p,k represents the sound source signal gain of the p-th loudspeaker for the k-th remote room, represents the far-end loudspeaker signal of one channel in the k-th far-end room; A corresponding near-end microphone signal is obtained according to the near-end loudspeaker signal; the near-end microphone signal is expressed as: Where P is the number of loudspeakers arranged in the near-end room array, h p (n) represents the echo path, s(n) represents the near-end speech signal, and v(n) represents the additive noise.

4. The echo cancellation method according to claim 1, characterized in that: The first compressed complex spectrum is expressed as: in, represents the first compressed complex spectrum, and θ represent the amplitude information and phase information of the far-end microphone signal corresponding to the k-th far-end room respectively, and β is a preset constant.

5. The echo cancellation method according to claim 1, characterized in that: The neural network model includes: a sequentially connected encoding network, a temporal modeling network, a decoding network and a linear layer, and the encoding network is also jump-connected to the decoding network.

6. The echo cancellation method according to claim 5, characterized in that: The decoding network includes two decoding network branches, and the two decoding network branches are respectively connected to a linear layer; the loss function of the neural network model is expressed as: Among them, Loss represents the loss value, and represent the real part and imaginary part of the predicted speech signal output by the neural network model, respectively, and They represent the real part and imaginary part indicated by the labels of the speech signal training samples respectively.

7. The echo cancellation method according to any one of claims 1 to 6, characterized in that: Also includes: Determining virtual sound source positions in the near-end room that respectively correspond to a plurality of actual sound sources in the far-end room; Based on the plurality of the clean near-end speech signals corresponding to the plurality of the actual sound sources, correspondingly generate a plurality of speech reconstruction instructions; The speech reconstruction instruction is sent to the speakers or speaker combinations corresponding to the positions of the virtual sound sources in the near-end room respectively, so as to instruct the speakers at different positions in the near-end room to play the corresponding pure near-end speech signals.

8. An echo cancellation device based on deep learning, characterized in that: include: A signal acquisition module, used to acquire a far-end microphone signal corresponding to a far-end room, and to acquire a near-end microphone signal corresponding to a near-end room; A complex spectrum acquisition module, used to use the far-end microphone signal as a reference signal, acquire a first compressed complex spectrum corresponding to the reference signal, and acquire a second compressed complex spectrum corresponding to the near-end microphone signal; A model processing module, used for inputting the first compressed complex spectrum and the second compressed complex spectrum into the trained neural network model for echo cancellation processing, and outputting a near-end speech compressed complex spectrum; The signal conversion module is used to perform short-time Fourier inverse transform on the near-end speech compression complex spectrum to obtain a pure near-end speech signal.

9. An electronic device, characterized in that: include: Memory and processor; The processor is used to execute the computer program stored in the memory; When the processor executes the computer program, the steps in the deep learning-based echo cancellation method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps in the deep learning-based echo cancellation method as described in any one of claims 1 to 7 are implemented.

Images