A local sound reinforcement method, device, terminal equipment, medium and product

Abstract

The present application provides a local sound reinforcement method, device, terminal equipment, medium and product. According to an example of the present application, the method may include: extracting features from a received first time-frequency domain signal to obtain a first time-frequency domain feature; extracting features from a second time-frequency domain signal to be played to obtain a second time-frequency domain feature; inputting the first time-frequency domain feature and the second time-frequency domain feature into a human voice extraction model to obtain a time-frequency mask output by the human voice extraction model, the time-frequency mask being used to represent the human voice time-frequency domain signal in the first time-frequency domain signal; based on the time-frequency mask, suppressing the acoustic factors in the first time-frequency domain signal to obtain a human voice time-frequency domain signal; converting the human voice time-frequency domain signal into a human voice time-domain signal, and locally amplifying the human voice time-domain signal, which can simultaneously suppress howling, environmental noise, room reverberation and echo residue in the first time-frequency domain signal, thereby improving the clarity and audibility of local sound reinforcement.

Description

Technical Field

[0001] The present application relates to the field of audio technology, and in particular to a local sound reinforcement method, device, terminal equipment, medium and product. Background Art

[0002] As a key technology for achieving sound amplification and transmission, local sound reinforcement is widely used in education, conferences, entertainment and other scenarios, such as large classrooms, auditoriums, karaoke, etc. The terminal device captures the sound source signal through a microphone, and then uses a speaker to amplify and play the sound, so that the sound can cover a wider area and enhance the listening experience.

[0003] However, in the actual local sound reinforcement process, a variety of unfavorable acoustic factors will arise, seriously affecting the clarity and audibility of the sound. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present application provides a local sound reinforcement method, device, terminal equipment, medium and product.

[0005] According to a first aspect of any embodiment of the present application, a local sound reinforcement method is provided, the method comprising:

[0006] Extracting features from the received first time-frequency domain signal to obtain first time-frequency domain features;

[0007] Perform feature extraction on the second time-frequency domain signal to be played to obtain second time-frequency domain features;

[0008] Inputting the first time-frequency domain feature and the second time-frequency domain feature into a human voice extraction model to obtain a time-frequency mask output by the human voice extraction model, wherein the time-frequency mask is used to represent a human voice time-frequency domain signal in the first time-frequency domain signal;

[0009] Based on the time-frequency masking, suppressing acoustic factors in the first time-frequency domain signal to obtain the human voice time-frequency domain signal; the acoustic factors include: howling, environmental noise, room reverberation and echo residue;

[0010] The human voice time-frequency domain signal is converted into a human voice time domain signal, and the human voice time domain signal is locally amplified.

[0011] According to a second aspect of any embodiment of the present application, a local sound reinforcement device is provided, the device comprising:

[0012] A first extraction module, used to extract features from the received first time-frequency domain signal to obtain first time-frequency domain features;

[0013] A second extraction module, used for extracting features of the second time-frequency domain signal to be played, to obtain second time-frequency domain features;

[0014] A human voice extraction module, used for inputting the first time-frequency domain feature and the second time-frequency domain feature into a human voice extraction model, obtaining a time-frequency mask output by the human voice extraction model, wherein the time-frequency mask is used to represent a human voice time-frequency domain signal in the first time-frequency domain signal;

[0015] A factor suppression module, configured to suppress acoustic factors in the first time-frequency domain signal based on the time-frequency masking to obtain the human voice time-frequency domain signal; the acoustic factors include: howling, ambient noise, room reverberation and echo residue;

[0016] The local sound reinforcement module is used to convert the human voice time-frequency domain signal into a human voice time domain signal, and perform local sound reinforcement on the human voice time domain signal.

[0017] According to a third aspect of any embodiment of the present application, a terminal device is provided, including:

[0018] processor;

[0019] a memory for storing processor-executable instructions;

[0020] The processor implements the method described in any embodiment of the present application by running the executable instructions.

[0021] According to a fourth aspect of any embodiment of the present application, a computer-readable storage medium is provided, on which computer instructions are stored. When the instructions are executed by a processor, the method described in any embodiment of the present application is implemented.

[0022] According to a fifth aspect of any embodiment of the present application, a computer program product is provided, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the method described in any embodiment of the present application is implemented.

[0023] The technical solution provided by this application may have the following beneficial effects:

[0024] According to the above embodiments, it can be known that by performing feature extraction on the received first time-frequency domain signal to obtain the first time-frequency domain feature, performing feature extraction on the second time-frequency domain signal to be played to obtain the second time-frequency domain feature, the first time-frequency domain feature and the second time-frequency domain feature are input into the human voice extraction model to obtain the time-frequency masking output by the human voice extraction model, based on the time-frequency masking, the acoustic factors in the first time-frequency domain signal are suppressed to obtain the human voice time-frequency domain signal, the human voice time-frequency domain signal is converted into the human voice time domain signal, and the human voice time domain signal is locally amplified, which can simultaneously suppress the howling, ambient noise, room reverberation and echo residual in the first time-frequency domain signal, so that the processed human voice time domain signal is closer to the user's original voice, thereby improving the clarity and audibility of local sound reinforcement.

[0025] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings, which are incorporated in the specification and constitute a part of this application, illustrate embodiments consistent with the application and, together with the description, serve to explain the principles of the application.

[0027] Figure 1 is a flow chart of a local sound reinforcement method according to an exemplary embodiment of the present application;

[0028] Figure 2 is a schematic diagram of a local sound reinforcement according to an exemplary embodiment of the present application;

[0029] Figure 3 is a flowchart of a method for training a human voice extraction model according to an exemplary embodiment of the present application;

[0030] Figure 4 is a structural schematic diagram of a post-filtering module according to an exemplary embodiment of the present application;

[0031] Figure 5 is a structural schematic diagram of a human voice extraction model according to an exemplary embodiment of the present application;

[0032] Figure 6 is a flow chart of another local sound reinforcement method according to an exemplary embodiment of the present application;

[0033] Figure 7 is a schematic diagram of the structure of a terminal device according to an exemplary embodiment of the present application;

[0034] Figure 8 It is a block diagram of a local sound reinforcement device according to an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0035] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0036] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0037] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0038] At present, in the process of local sound reinforcement, since the digital signal processing system in the terminal equipment also includes recording, amplification and playback functions, it is equivalent to the positive feedback of the signal. If the signal is directly amplified without processing, the signal will be continuously fed back, amplified, and eventually form a howling.

[0039] In local sound reinforcement scenarios, in addition to the impact of howling, there are usually adverse acoustic factors such as ambient noise, room reverberation and residual echo, which seriously affect the clarity and audibility of the sound.

[0040] In order to solve the above problems, the present application proposes a local sound reinforcement method. To further illustrate the present application, the following embodiments are provided:

[0041] See also Figure 1 , Figure 1 This is a flowchart of a local sound reinforcement method according to an exemplary embodiment of the present application. The local sound reinforcement method can be applied to terminal devices such as car machines, smart speakers, mobile phones, computers, messaging devices, tablet devices, personal digital assistants, and can be applied to local sound reinforcement scenarios such as karaoke, classrooms, meetings, and performances.

[0042] like Figure 1 As shown, the local sound reinforcement method may include the following steps:

[0043] Step 102: extracting features from the received first time-frequency domain signal to obtain first time-frequency domain features.

[0044] In this step, the terminal device can use a microphone or other sound collection device to receive the first time domain signal of the surrounding environment. Use methods such as Short Time Fourier Transform (STFT), Mel-Frequency Cepstral Coefficients (MFCC), Constant-Q Transform (CQT), etc. to convert the first time domain signal into the time-frequency domain to obtain the first time-frequency domain signal in the time-frequency domain.

[0045] The terminal device extracts features of the received first time-frequency domain signal by analyzing spectrum diagrams, deep learning models, and other methods to obtain first time-frequency domain features.

[0046] The first time domain signal is a time domain signal directly captured by a sound collection device. The time domain signal is an audio waveform that changes over time. The first time domain signal may include: human voice image, environmental noise and echo signal. The echo signal is the sound played by the speaker captured again by the microphone.

[0047] The vocal image is used to reflect the actual vocal situation of the user in a specific room environment. The vocal image may include: the vocal time domain signal and the room reverberation.

[0048] The human voice time domain signal is the original sound signal emitted directly by the user without being significantly changed by the room environment. The human voice time domain signal may include the user's direct sound, that is, the original human voice emitted directly by the user. The human voice time domain signal may also include early reverberation. Room reverberation is the late reverberation produced by the room environment on the original human voice.

[0049] The first time-frequency domain signal is used to describe the characteristics of the first time domain signal when both time and frequency information are considered. In the time-frequency domain, changes of the first time domain signal at different time points and different frequencies can be noted, providing a more comprehensive description of signal characteristics.

[0050] For example, the frequency of the first time domain signal changes with time, and the first time-frequency domain signal can simultaneously consider these two aspects of information, which is helpful to more accurately understand and analyze the human voice time-frequency domain signal.

[0051] The first time-frequency domain signal may include a human voice time-frequency domain signal, which is a speech signal in the time-frequency domain and is used to describe the original sound signal directly emitted by the user in the time domain and the frequency domain.

[0052] The first time-frequency domain feature is an audio feature that describes the characteristics of the first time-frequency domain signal, and is used to perform in-depth analysis and processing on the human voice time-frequency domain signal in the first time-frequency domain signal. It may be a spectrum, power spectrum, filter bank (FilterBank, FBank) feature, etc.

[0053] Step 104: extracting features from the second time-frequency domain signal to be played to obtain second time-frequency domain features.

[0054] In this step, the terminal device uses short-time Fourier transform, Mel spectrum transform, constant Q transform and other methods to transform the second time domain signal to be played in the previous frame to obtain a second time-frequency domain signal in the time-frequency domain.

[0055] The terminal device extracts features of the second time-frequency domain signal in the time-frequency domain by analyzing the spectrum diagram, deep learning model and other methods to obtain the second time-frequency domain features.

[0056] The second time domain signal is a time domain signal to be amplified and played by a loudspeaker or other sound reinforcement device in the terminal device in the previous frame, and is used to suppress adverse acoustic factors in the first time-frequency domain signal. The second time-frequency domain signal is a time-frequency domain signal obtained by converting the second time domain signal, and is used to provide distribution information of the second time domain signal in the time-frequency domain.

[0057] The second time-frequency domain feature is an audio feature that describes the characteristics of the second time-frequency domain signal, and is used for in-depth analysis and processing of the second time-frequency domain signal. It may be a spectrum, a power spectrum, an FBank feature, and the like.

[0058] In one embodiment, since the first time domain signal includes an echo signal, the terminal device may use an adaptive echo cancellation (Acoustic Echo Cancellation, AEC) algorithm to suppress the linear echo.

[0059] The terminal device adjusts the coefficients based on the adaptive filter to simulate the echo path and generate a predicted echo path. Based on the second time domain signal and the predicted echo path, the second time domain signal is convolved with the predicted echo path, the linear echo in the first time domain signal is calculated, and the linear echo in the first time domain signal is eliminated.

[0060] The predicted echo path is the acoustic path from the speaker to the microphone.

[0061] Exemplarily, the terminal device may obtain the first time domain signal after eliminating the linear echo according to the following formula (1):

[0062] (1)

[0063] Among them, y represents the first time domain signal after eliminating the linear echo, and x represents the first time domain signal captured by the microphone. represents the predicted echo path, r represents the second time domain signal, and * represents the convolution operation.

[0064] It can be understood that the adaptive echo cancellation algorithm described in the embodiment of the present application is only a preferred example, and other methods can also be used to eliminate the linear echo in the first time domain signal, which is not limited in the embodiment of the present application.

[0065] As described above, by calculating the linear echo in the first time domain signal based on the second time domain signal and the predicted echo path, and eliminating the linear echo in the first time domain signal, the complexity of the subsequent model for suppressing adverse acoustic factors can be reduced, and the overall efficiency and effect of audio processing can be improved.

[0066] To further introduce the local sound reinforcement process, Figure 2 A schematic diagram of local sound reinforcement is shown in FIG. Figure 2 As shown, the first time domain signal x captured by the terminal device using a microphone includes: , environmental noise v and echo signal e, the echo signal e is obtained by convolving the second time domain signal r with the unknown echo path b.

[0067] The terminal device convolves the second time domain signal r with the predicted echo path , calculate the linear echo in the first time domain signal x , to eliminate the linear echo in the first time domain signal x , and obtain the first time domain signal y after echo cancellation.

[0068] Step 106: Input the first time-frequency domain feature and the second time-frequency domain feature into the human voice extraction model to obtain the time-frequency mask output by the human voice extraction model, where the time-frequency mask is used to represent the human voice time-frequency domain signal in the first time-frequency domain signal.

[0069] In this step, the terminal device inputs the first time-frequency domain feature and the second time-frequency domain feature into the human voice extraction model, which is a trained deep convolutional neural network model (DCNN) that can identify the human voice time-frequency domain signal in the first time-frequency domain signal and extract the time-frequency mask in the first time-frequency domain signal. The terminal device obtains the time-frequency mask output by the human voice extraction model.

[0070] The time-frequency mask is used to represent the human voice time-frequency domain signal in the first time-frequency domain signal. The time-frequency mask is a two-dimensional matrix, in which the value of each element represents the intensity or existence of the human voice time-frequency domain signal at the corresponding time and frequency point.

[0071] Exemplarily, the value of each element in the time-frequency mask may be normalized to between 0 and 1, wherein 1 indicates that it completely belongs to the human voice time-frequency domain signal, and 0 indicates that it completely does not belong to the human voice time-frequency domain signal.

[0072] Specifically, the terminal device can obtain the time-frequency mask according to the following formula (2):

[0073] (2)

[0074] in, represents time-frequency masking, represents the first time-frequency domain feature and the second time-frequency domain feature, Λ represents the human voice extraction model operator, k represents the frequency band number, Indicates the data frame sequence number.

[0075] Step 108: Based on time-frequency masking, suppress the acoustic factors in the first time-frequency domain signal to obtain a human voice time-frequency domain signal; the acoustic factors include: howling, environmental noise, room reverberation and echo residue.

[0076] In this step, the terminal device adjusts the first time-frequency domain signal based on time-frequency masking to suppress the four acoustic factors of howling, ambient noise, room reverberation and echo residue in the first time-frequency domain signal to obtain a human voice time-frequency domain signal.

[0077] Among them, acoustic factors are acoustic characteristics that have a negative impact on the clarity of local sound reinforcement, and may include: howling, environmental noise, room reverberation, and echo residue. Echo residue is the residue of the echo signal caused by factors such as the predicted echo path not matching the actual echo and the nonlinear echo in the echo signal after eliminating the linear echo in the first time domain signal.

[0078] Exemplarily, the terminal device may obtain the human voice time-frequency domain signal according to the following formula (3):

[0079] (3)

[0080] in, Represents the time-frequency domain signal of human voice, represents time-frequency masking, represents the first time-frequency domain signal, k represents the frequency band number, Indicates the data frame sequence number.

[0081] Step 110: Convert the human voice time-frequency domain signal into a human voice time-domain signal, and locally amplify the human voice time-domain signal.

[0082] In this step, the terminal device uses methods such as Inverse Short-Time Fourier Transform (ISTFT), Overlap-Add Method, and Overlap-Save Method to convert the human voice time-frequency domain signal into a human voice time-domain signal. The human voice time-domain signal is amplified locally using a loudspeaker to achieve real-time local amplification.

[0083] The human voice time domain signal is a time domain signal to be amplified and played by the loudspeaker in the current frame.

[0084] Please continue reading Figure 2 , exemplarily, the post-filter module 20 in the terminal device can obtain the human voice time domain signal according to the following formula (4):

[0085] (4)

[0086] Among them, z represents the human voice time domain signal, y represents the first time domain signal, r represents the second time domain signal, H represents the post-filtering operator, and θ represents the human voice extraction model parameters.

[0087] Specifically, the post-filtering module 20 can convert and extract features from the second time domain signal r and the first time domain signal y after echo cancellation to obtain first time-frequency domain features and second time-frequency domain features. The post-filtering module 20 inputs the first time-frequency domain features and the second time-frequency domain features into the human voice extraction model to obtain time-frequency masking.

[0088] The post-filter module 20 applies time-frequency masking to the first time-frequency domain signal, suppresses the four acoustic factors of howling, environmental noise, room reverberation and echo residue in the first time-frequency domain signal, and obtains the human voice time-frequency domain signal. The post-filter module 20 converts the human voice time-frequency domain signal into the human voice time-domain signal z.

[0089] In one embodiment, the terminal device may perform gain control and delay processing on the human voice time domain signal to obtain a processed human voice time domain signal, superimpose the processed human voice time domain signal with the background audio signal to obtain a target signal, and use a speaker to locally amplify the target signal.

[0090] The background audio signal may be song accompaniment, background music, environmental sound effects, etc. The target signal is an audio signal to be locally amplified.

[0091] Please continue reading Figure 2Taking the local sound reinforcement scenario as a karaoke scenario as an example, the terminal device can perform gain control on the human voice time domain signal z, and g represents the amplified gain. Due to the algorithm processing, recording, and playback operations, a certain signal delay will be caused. The terminal device performs delay processing on the gain-controlled human voice time domain signal to obtain the processed human voice time domain signal. Indicates that caused The delay of each sampling point.

[0092] The terminal device combines the processed human voice time domain signal with the background audio signal Add new reverberation and sound effects to get the target signal. Use speakers to amplify the target signal locally.

[0093] As described above, by performing gain control and delay processing on the human voice time domain signal, a processed human voice time domain signal is obtained, the processed human voice time domain signal is superimposed with the background audio signal to obtain a target signal, and a loudspeaker is used to locally amplify the target signal. This can balance the volume and delay of the human voice time domain signal and the background audio signal, thereby achieving joint amplification of the human voice time domain signal and the background audio signal.

[0094] The local sound reinforcement method of this embodiment obtains a first time-frequency domain feature by performing feature extraction on a received first time-frequency domain signal, obtains a second time-frequency domain feature by performing feature extraction on a second time-frequency domain signal in the time-frequency domain, inputs the first time-frequency domain feature and the second time-frequency domain feature into a human voice extraction model, obtains a time-frequency mask output by the human voice extraction model, suppresses acoustic factors in the first time-frequency domain signal based on the time-frequency mask, obtains a human voice time-frequency domain signal, converts the human voice time-frequency domain signal into a human voice time-domain signal, and performs local sound reinforcement on the human voice time-domain signal, which can simultaneously suppress howling, environmental noise, room reverberation, and echo residue in the first time-frequency domain signal, so that the processed human voice time-domain signal is closer to the user's original voice, thereby improving the clarity and audibility of local sound reinforcement.

[0095] In the aforementioned embodiment, it is introduced that howling, environmental noise, room reverberation and echo residue in the first time-frequency domain signal are suppressed by using the time-frequency mask extracted by the human voice extraction model based on the first time-frequency domain feature and the second time-frequency domain feature, and the pure human voice time-domain signal is locally amplified. In the following embodiment, the training process and structure of the human voice extraction model will be described in more detail, and can be applied to any of the above embodiments.

[0096] In one embodiment, the terminal device may obtain training data through data simulation, and use the training data and label data to train the initial model to obtain a trained voice extraction model.

[0097] Specifically, the terminal device may superimpose the simulated second time domain signal of the previous frame and the simulated background audio signal to be amplified to obtain a superimposed simulated second time domain signal.

[0098] Exemplarily, the terminal device may obtain the superimposed simulated second time domain signal according to the following formula (5):

[0099] (5)

[0100] in, represents the simulated second time domain signal after superposition, represents the simulated second time domain signal of the previous frame, Represents the simulated background audio signal, Indicates the data frame sequence number.

[0101] The terminal device can simulate the echo path of the superimposed simulated second time domain signal to obtain a simulated echo signal. In a sound reinforcement scenario where no background audio signal needs to be added, the terminal device can also directly simulate the echo path of the simulated second time domain signal of the previous frame to obtain a simulated echo signal.

[0102] Exemplarily, the terminal device may obtain the analog echo signal according to the following formula (6):

[0103] (6)

[0104] in, represents the simulated echo signal, b represents the echo path, represents the simulated second time domain signal of the previous frame, Indicates the data frame sequence number.

[0105] The terminal device can simulate the room transmission function of the human voice source to obtain a simulated human voice image, wherein the simulated human voice image may include: a simulated human voice time domain signal and a simulated room reverberation.

[0106] Exemplarily, the terminal device may obtain the simulated human voice image according to the following formula (7):

[0107] (7)

[0108] in, It represents the simulated human voice image, s represents the human voice source, Represents the direct sound and early reverberation of the simulated room transmission function. The simulated room reverberation represents the late stage of the simulated room propagation function. Represents the simulated human voice time domain signal, Indicates the data frame sequence number.

[0109] The terminal device can superimpose the simulated echo signal, the simulated human voice image and the simulated environmental noise to obtain a simulated first time domain signal.

[0110] Exemplarily, the terminal device may obtain the simulated first time domain signal according to the following formula (8):

[0111] (8)

[0112] in, represents the simulated first time domain signal, represents the analog echo signal, It means simulating the human voice image. represents the simulated environmental noise, Indicates the data frame sequence number.

[0113] The terminal device uses an echo cancellation algorithm such as AEC to perform echo cancellation on the simulated first time domain signal to obtain an echo cancellation signal.

[0114] Based on the echo cancellation signal and the simulated second time domain signal, training data is generated. The echo cancellation signal and the simulated second time domain signal are respectively converted into time and frequency domains and feature extracted to obtain simulated first time and frequency domain features and simulated second time and frequency domain features. The simulated first time and frequency domain features and simulated second time and frequency domain features are used as training data.

[0115] Based on the simulated human voice time domain signal, label data corresponding to the training data is generated. The simulated human voice time domain signal is converted into a time-frequency domain to obtain a simulated human voice time-frequency domain signal, and the simulated human voice time-frequency domain signal is used as the corresponding label data.

[0116] The terminal device performs gain control and delay processing on the echo cancellation signal to obtain the simulated second time domain signal of the current frame. By recursively simulating the training data between different frames, four unfavorable acoustic factors, namely howling, environmental noise, room reverberation and echo residue, are simulated at the same time to help the model learn to process dynamically changing sound signals.

[0117] The simulated second time domain signal is a simulated second time-frequency domain signal that is not affected by the four acoustic factors of howling, environmental noise, room reverberation and echo residue. The simulated echo signal is used to simulate the echo generated by the reflection of the speaker sound to the microphone in the actual environment.

[0118] The human voice source is the original, unprocessed simulated real human voice. The simulated human voice image is the signal obtained by simulating the room transfer function (i.e. the frequency response characteristics of sound when propagating in the room) of the human voice source, which is used to train the model to identify and process the simulated room reverberation.

[0119] The simulated human voice time-frequency domain signal is the part of the simulated human voice image that comes directly from the human voice source, that is, the pure human voice that is not affected by the room reverberation. The simulated room reverberation is used to simulate the sound effect produced by the reflection and superposition of the human voice source in the room.

[0120] Simulated environmental noise simulates the background noise accompanying human voice in a real environment and is used to train the model to learn to extract human voice in a noisy environment.

[0121] The simulated first time domain signal is used to simulate the first time-frequency domain signal actually received by the microphone. The echo cancellation signal is a signal obtained after the simulated first time domain signal is subjected to echo cancellation processing, and is used to train the model to identify and process echo residues in the echo cancellation signal.

[0122] It is understandable that when generating training data and label data, other unfavorable acoustic factors can also be simulated, and the time-frequency masking can be predicted using the trained human voice extraction model. Based on the time-frequency masking, other acoustic factors in the first time-frequency domain signal can be suppressed. The embodiments of the present application do not limit this.

[0123] As described above, by simulating the echo path of the simulated second time domain signal of the previous frame to obtain a simulated echo signal, simulating the room transmission function of the human voice source to obtain a simulated human voice image, superimposing the simulated echo signal, the simulated human voice image and the simulated environmental noise to obtain a simulated first time domain signal, performing echo cancellation on the simulated first time domain signal to obtain an echo cancellation signal, generating training data based on the echo cancellation signal and the simulated second time domain signal, generating label data corresponding to the training data based on the simulated human voice time domain signal, performing gain control and delay processing on the echo cancellation signal to obtain a simulated second time domain signal of the current frame. Since the occurrence process of howling is recursive and progressive, the training data can better simulate howling by using a recursive data simulation process. The training data is also mixed with simulated environmental noise, simulated room reverberation and echo residue to ensure that the training data is closer to the real sound reinforcement data, so that the trained human voice extraction model has a better suppression effect.

[0124] In one embodiment, Figure 3 FIG. 4 is a flowchart showing a method for training a human voice extraction model. Figure 3 As shown, the training method may include the following steps:

[0125] Step 302: Initialize the initial model.

[0126] In this step, the terminal device selects or defines an initial neural network model, which may include the number of neural network model layers, the number of neurons in each layer, activation functions, etc. This initial model will serve as the starting point for training.

[0127] The terminal device divides a portion of the available training data into a validation set. The validation set is used to evaluate the performance of the model during the training process, but is not used to train the model itself to prevent overfitting.

[0128] Step 304: Based on the training progress of the initial model, select training data of different difficulty levels.

[0129] In this step, the terminal device dynamically selects training data of different difficulty levels based on the training progress of the initial model. For example, simpler training data is used in the early stage of training to help the initial model quickly learn basic features, and the difficulty of the training data is gradually increased as the training progresses.

[0130] The training progress can be determined according to the number of completed epochs, and the difficulty of the training data can be the signal-to-noise ratio of the training data.

[0131] Step 306: Generate a small batch of training data.

[0132] In this step, the terminal device randomly extracts a small batch of data from the training data for this training iteration. The small batch is a hyperparameter that can be adjusted according to actual conditions.

[0133] Step 308: Input the training data into the initial model to obtain the prediction result output by the initial model.

[0134] In this step, the terminal device inputs the batch data generated in step 306 into the model, performs forward propagation, and obtains the model's prediction results for each input data.

[0135] Step 310: Calculate the loss value based on the prediction results and label data, and update the parameters of the initial model.

[0136] In this step, the terminal device uses loss functions such as mean square error and cross entropy to calculate the loss value between the prediction result and the actual label data. According to the loss value, the model parameters are updated through the back propagation algorithm to reduce the loss value.

[0137] Repeat steps 306 to 310 to perform batch cycles until all training data of the current batch is processed.

[0138] Step 312: Calculate the average loss value (epoch loss) of the entire epoch of the initial model.

[0139] In this step, after an epoch (i.e., after traversing all the training data once) is completed, the terminal device calculates and records the average loss value of the entire round to evaluate the training effect of the epoch.

[0140] Step 314: Save the parameters and state of the initial model.

[0141] In this step, after each epoch, the terminal device saves the parameters and status of the current initial model as a checkpoint so that training can be resumed or further analysis can be performed later.

[0142] Step 316: Adjust the training progress of the initial model, select training data of the next difficulty level for model training, until the preset conditions are met, and obtain the human voice extraction model.

[0143] In this step, the terminal device adjusts the training progress of the initial model, selects the training data of the next difficulty, and continues to execute steps 304 to 316, and repeats rounds until the model meets preset conditions such as the loss value is lower than a certain threshold and the accuracy of the verification set no longer improves, thereby obtaining the final voice extraction model.

[0144] As described above, by selecting training data of different difficulty levels based on the training progress of the initial model, the training data is input into the initial model to obtain the prediction results of the initial model output, the loss value is calculated based on the prediction results and the label data, the parameters of the initial model are updated, the training progress of the initial model is adjusted, and the training data of the next difficulty level is selected for model training until the preset conditions are met to obtain a human voice extraction model. Using training data of different difficulty levels to gradually advance the training process will help the model gradually master the basic knowledge and steadily improve its ability to handle complex tasks. While ensuring the training effect, it avoids directly using high-difficulty data, which may cause the model to learn slowly and reduce unnecessary resource consumption.

[0145] In one embodiment, see Figure 4 , Figure 4 The schematic diagram of the structure of a post-filter module is shown. The post-filter module 20 may include: a STFT module 200, a feature extraction module 201, a human voice extraction model 202, a suppression module 203 and an ISTFT module 204.

[0146] The STFT module 200 can perform a short-time Fourier transform on the first time domain signal received by the microphone to obtain a first time-frequency domain signal. The STFT module 200 can perform a short-time Fourier transform on the second time domain signal to be played by the speaker to obtain a second time-frequency domain signal.

[0147] The STFT module 200 inputs the first time-frequency domain signal to the suppression module 203. The STFT module 200 inputs the first time-frequency domain signal and the second time-frequency domain signal to the feature extraction module 201, and the feature extraction module 201 extracts the first time-frequency domain feature corresponding to the first time-frequency domain signal and the second time-frequency domain feature corresponding to the second time-frequency domain signal.

[0148] The feature extraction module 201 inputs the first time-frequency domain feature and the second time-frequency domain feature to the human voice extraction model 202 , and the human voice extraction model 202 inputs the extracted time-frequency mask to the suppression module 203 .

[0149] The suppression module 203 applies the time-frequency masking to the first time-frequency domain signal to suppress the acoustic factors in the first time-frequency domain signal to obtain the human voice time-frequency domain signal. The suppression module 203 inputs the human voice time-frequency domain signal to the ISTFT module 204.

[0150] The ISTFT module 204 performs an inverse short-time Fourier transform on the human voice time-frequency domain signal to obtain a human voice time-domain signal in the time domain.

[0151] As described above, by using a microphone to receive a first time domain signal, performing a short-time Fourier transform on the first time domain signal to obtain a first time-frequency domain signal, performing a short-time Fourier transform on the second time domain signal to be played by the speaker to obtain a second time-frequency domain signal, and performing an inverse short-time Fourier transform on the human voice time-frequency domain signal to obtain a human voice time domain signal, the characteristics of the first time-frequency domain signal and the second time-frequency domain signal in two dimensions of time and frequency are captured, and short-time Fourier transform and inverse short-time Fourier transform can improve the computational efficiency of converting from time domain to time-frequency domain and then back to time domain, thereby ensuring the real-time performance of local sound reinforcement.

[0152] In one embodiment, the terminal device may splice the first time-frequency domain features of multiple channels such as stereo channels by concatenation, fusion, etc. to obtain the spliced ​​first splicing features. The terminal device may splice the second time-frequency domain features of multiple channels to obtain the spliced ​​second splicing features.

[0153] The first time-frequency domain feature is feature data of the first time-frequency domain signal on multiple channels, and the second time-frequency domain feature is feature data of the second time-frequency domain signal on multiple channels.

[0154] See also Figure 5 , Figure 5 The schematic diagram of the structure of a human voice extraction model is shown. The human voice extraction model may include: a nonlinear transformation unit 50, a multi-layer convolution unit 51 and a masking processing unit 52.

[0155] The nonlinear transformation unit 50 may include a fully connected layer (Affine) and a rectified linear unit (ReLU) activation function. The masking processing unit 52 may include a fully connected layer (Affine) and a tanh activation function.

[0156] The multi-layer convolution unit 51 may include a multi-layer feedforward sequential memory network (FSMN) unit, and each layer of the FSMN unit may include: a fully connected layer 510, a feedforward sequential memory network layer 511 and an activation layer 512.

[0157] The fully connected layer 510 is a linear layer, which is used to receive input data and generate output through linear transformation. The activation layer 512 is a ReLU (Affine) layer, which is used to perform nonlinear transformation on the output of the feedforward sequence memory network layer 511.

[0158] The feedforward sequence memory network layer 511 is composed of a convolutional structure, which is equivalent to the frequency domain signal processing algorithm frequency band filtering operation. Perform one-dimensional convolution on each feature dimension of . is the first splicing feature and the second splicing feature.

[0159] The terminal device will splice the features Input to the nonlinear transformation unit 50, the nonlinear transformation unit 50 performs concatenation feature Perform nonlinear transformation to obtain nonlinear transformation data. Nonlinear transformation data is used to enhance the nonlinear expression ability of the model and help the model learn more complex audio feature representations.

[0160] The nonlinear transformation unit 50 inputs the nonlinear transformation data to the multi-layer convolution unit 51, and the multi-layer convolution unit 51 performs convolution processing on the nonlinear transformation data, and constructs a deeper network structure by stacking multiple convolution layers to extract local features and reduce the spatial dimension of the features to obtain filtered data.

[0161] The multi-layer convolution unit 51 inputs the filtered data to the mask processing unit 52, and the mask processing unit 52 calculates and outputs the time-frequency mask based on the filtered data. In the fully connected layer of the mask processing unit 52, each input node is connected to the output node through a weight, and a bias term is added.

[0162] Through forward propagation, the values ​​of each output node are calculated, which correspond to the real and imaginary parts of the time-frequency mask. The activation function is applied to the output of the fully connected layer to obtain the final time-frequency mask consisting of the real and imaginary parts.

[0163] The time-frequency mask is a complex number consisting of a real part and an imaginary part. The real part and the imaginary part in the time-frequency mask jointly describe the characteristics of the human voice time-frequency domain signal in the time-frequency domain.

[0164] Understandably, Figure 5The structure of the human voice extraction model shown is only an example, and other neural network structures may also be used. As long as the human voice extraction model can implement the local sound amplification method described in the embodiment of the present application, the embodiment of the present application does not impose any restrictions on this.

[0165] As described above, by splicing the first time-frequency domain features of the first time-frequency domain signal on multiple channels to obtain the spliced ​​first spliced ​​features, and splicing the second time-frequency domain features of the second time-frequency domain signal on multiple channels to obtain the spliced ​​second spliced ​​features, multi-channel information fusion helps to more accurately capture the directionality, spatial distribution and other characteristics of the sound source, thereby improving the effect of human voice separation and enhancement; the nonlinear transformation unit performs nonlinear transformation on the first splicing feature and the second splicing feature to obtain nonlinear transformation data, and the multi-layer convolution unit performs convolution processing on the nonlinear transformation data to obtain filtered data, which can more accurately locate the position of the human voice time-frequency domain signal in the time-frequency domain, and effectively suppress interference factors such as environmental noise and echo residuals; the masking processing unit calculates and outputs the time-frequency mask composed of the real part and the imaginary part based on the filtered data, and the time-frequency mask can more accurately control the retention and suppression of the human voice time-frequency domain signal in the time-frequency domain, thereby improving the suppression effect of the four acoustic factors.

[0166] To further introduce the local sound reinforcement process, Figure 6 A flow chart of another local sound reinforcement method is shown. The local sound reinforcement method may include the following steps:

[0167] Step 602: recursively simulate training data and label data.

[0168] In this step, the terminal device uses the simulated second time domain signal of the previous frame to obtain a simulated echo signal. The simulated echo signal, the simulated human voice image and the simulated environmental noise are superimposed to obtain a simulated first time domain signal. The simulated first time domain signal is subjected to echo cancellation processing, training data is generated based on the echo cancellation signal and the simulated second time domain signal, and label data is generated based on the simulated human voice time domain signal in the simulated human voice image.

[0169] The echo cancellation signal is gain controlled and delayed to simulate the change of the signal in the next frame to obtain the simulated second time domain signal of the current frame. The simulated second time domain signal of the current frame is then used to obtain the simulated echo signal of the next frame, simulating the howling, ambient noise, room reverberation and echo residue in the real environment to realize recursive simulation training data.

[0170] Step 604: Use the training data and label data to train the initial model to obtain a trained voice extraction model.

[0171] In this step, the terminal device selects training data of different difficulty levels from the simulated training data based on the training progress of the initial model, inputs the training data into the initial model, obtains the prediction results output by the initial model, calculates the loss value based on the prediction results and the label data, and updates the parameters of the initial model.

[0172] The terminal device adjusts the training progress of the initial model and selects training data of the next difficulty level for model training until the preset conditions are met to obtain a trained voice extraction model.

[0173] Step 606: Eliminate the linear echo in the first time domain signal.

[0174] In this step, the terminal device convolves the second time domain signal with the predicted echo path, calculates the linear echo in the first time domain signal, and eliminates the linear echo in the first time domain signal.

[0175] Step 608: Perform short-time Fourier transform on the first time domain signal and the second time domain signal.

[0176] In this step, the terminal device performs short-time Fourier transform on the first time domain signal and the second time domain signal after the linear echo is eliminated.

[0177] Step 610: The terminal device extracts features from the first time-frequency domain signal to obtain first time-frequency domain features.

[0178] Step 612: The terminal device extracts features from the second time-frequency domain signal to obtain second time-frequency domain features.

[0179] Step 614: Input the first time-frequency domain feature and the second time-frequency domain feature into the human voice extraction model to obtain the time-frequency mask output by the human voice extraction model.

[0180] In this step, the terminal device splices the first time-frequency domain features to obtain a spliced ​​first spliced ​​feature, and splices the second time-frequency domain features to obtain a spliced ​​second spliced ​​feature.

[0181] The nonlinear transformation unit in the human voice extraction model performs nonlinear transformation on the first splicing feature and the second splicing feature to obtain nonlinear transformation data. The multi-layer convolution unit performs convolution processing on the nonlinear transformation data to obtain filtered data. The mask processing unit calculates and outputs the time-frequency mask based on the filtered data.

[0182] Step 616: Based on time-frequency masking, suppress the acoustic factors in the first time-frequency domain signal to obtain a human voice time-frequency domain signal.

[0183] In this step, the terminal device applies a time-frequency masking effect to the first time-frequency domain signal to suppress acoustic factors in the first time-frequency domain signal to obtain a human voice time-frequency domain signal in the time-frequency domain.

[0184] Step 618: Superimpose the gain-controlled and delay-processed human voice time domain signal with the background audio signal to obtain a target signal.

[0185] In this step, the terminal device performs gain control and delay processing on the human voice time domain signal, and superimposes the gain controlled and delayed human voice time domain signal with the background audio signal to obtain a target signal.

[0186] Step 620: Use a loudspeaker to locally amplify the target signal.

[0187] In this step, the terminal device uses a loudspeaker to locally amplify the target signal, thereby realizing a local amplification function that simultaneously suppresses four acoustic factors: howling, ambient noise, room reverberation, and residual echo.

[0188] Figure 7 1 is a schematic diagram of a terminal device according to an exemplary embodiment of the present application. The terminal device may be, for example, a mobile phone, a computer, a digital broadcast terminal, a message transceiver, a game console, a tablet device, a personal digital assistant, a server, a smart home appliance, a car computer, etc. Figure 7 At the hardware level, the terminal device includes a processor 702, an internal bus 704, a network interface 706, a memory 708, and a non-volatile memory 710, and may also include hardware required for other services. The processor 702 reads the corresponding computer program from the non-volatile memory 710 into the memory 708 and then runs it, forming a local sound reinforcement device at the logical level. Of course, in addition to the software implementation, this application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0189] Figure 8 is a block diagram of a local sound reinforcement device according to an exemplary embodiment of the present application. Figure 8 The device may include: a first extraction module 802, a second extraction module 804, a human voice extraction module 806, a factor suppression module 808 and a local sound reinforcement module 810, wherein:

[0190] The first extraction module 802 is used to extract features from the received first time-frequency domain signal to obtain first time-frequency domain features;

[0191] The second extraction module 804 is used to extract features of the second time-frequency domain signal to be played to obtain second time-frequency domain features;

[0192] The human voice extraction module 806 is used to input the first time-frequency domain feature and the second time-frequency domain feature into a human voice extraction model to obtain a time-frequency mask output by the human voice extraction model, wherein the time-frequency mask is used to represent the human voice time-frequency domain signal in the first time-frequency domain signal;

[0193] The factor suppression module 808 is used to suppress the acoustic factors in the first time-frequency domain signal based on the time-frequency masking to obtain the human voice time-frequency domain signal; the acoustic factors include: howling, environmental noise, room reverberation and echo residue;

[0194] The local sound reinforcement module 810 is used to convert the human voice time-frequency domain signal into a human voice time domain signal, and perform local sound reinforcement on the human voice time domain signal.

[0195] In one example, the human voice extraction module 806, before being used to input the first time-frequency domain features and the second time-frequency domain features into the human voice extraction model to obtain the time-frequency mask output by the human voice extraction model, further includes: using the training data and the label data to train the initial model to obtain the trained human voice extraction model; the human voice extraction module 806, when used to generate the training data and the label data, includes: simulating the echo path of the simulated second time domain signal of the previous frame to obtain a simulated echo signal; simulating the room transmission function of the human voice source to obtain a simulated human voice image The simulated human voice image includes: a simulated human voice time domain signal and a simulated room reverberation; superimposing the simulated echo signal, the simulated human voice image and the simulated environmental noise to obtain a simulated first time domain signal; performing echo cancellation on the simulated first time domain signal to obtain an echo cancellation signal; generating the training data based on the echo cancellation signal and the simulated second time domain signal; generating label data corresponding to the training data based on the simulated human voice time domain signal; performing gain control and delay processing on the echo cancellation signal to obtain a simulated second time domain signal of the current frame.

[0196] In one example, the human voice extraction module 806, when used to train the initial model using training data and label data to obtain the trained human voice extraction model, includes: selecting training data of different difficulties based on the training progress of the initial model; inputting the training data into the initial model to obtain the prediction results of the initial model output; calculating the loss value according to the prediction results and the label data, and updating the parameters of the initial model; adjusting the training progress of the initial model, selecting training data of the next difficulty for model training, until the preset conditions are met to obtain the human voice extraction model.

[0197] In one example, the local sound reinforcement module 810, when used to locally amplify the human voice time domain signal, includes: performing gain control and delay processing on the human voice time domain signal to obtain a processed human voice time domain signal; superimposing the processed human voice time domain signal with the background audio signal to obtain a target signal; and using a speaker to locally amplify the target signal.

[0198] In one example, the first extraction module 802, before being used to extract features from the received first time-frequency domain signal to obtain the first time-frequency domain features, further includes: receiving the first time-domain signal using a microphone; performing a short-time Fourier transform on the first time-domain signal to obtain the first time-frequency domain signal; performing a short-time Fourier transform on the second time-domain signal to be played by the speaker to obtain the second time-frequency domain signal; the local sound reinforcement module 810, when used to convert the human voice time-frequency domain signal into a human voice time-domain signal, includes: performing an inverse short-time Fourier transform on the human voice time-frequency domain signal to obtain the human voice time-domain signal in the time domain.

[0199] In one example, the first extraction module 802, before being used to perform short-time Fourier transform on the first time domain signal to obtain the first time-frequency domain signal, further includes: calculating the linear echo in the first time domain signal based on the second time domain signal and the predicted echo path; and eliminating the linear echo in the first time domain signal.

[0200] In one example, the human voice extraction model includes: a nonlinear transformation unit, a multi-layer convolution unit and a masking processing unit; the human voice extraction module 806, when used to input the first time-frequency domain features and the second time-frequency domain features into the human voice extraction model to obtain the time-frequency mask output by the human voice extraction model, includes: splicing the first time-frequency domain features to obtain a first spliced ​​feature after splicing, the first time-frequency domain features are feature data of the first time-frequency domain signal on multiple channels; splicing the second time-frequency domain features to obtain a second spliced ​​feature after splicing, the second time-frequency domain features are feature data of the second time-frequency domain signal on multiple channels; the nonlinear transformation unit performs nonlinear transformation on the first splicing feature and the second splicing feature to obtain nonlinear transformation data; the multi-layer convolution unit performs convolution processing on the nonlinear transformation data to obtain filtered data; the masking processing unit calculates and outputs the time-frequency mask based on the filtered data; the time-frequency mask consists of a real part and an imaginary part.

[0201] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0202] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying creative labor.

[0203] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory including instructions. The instructions can be executed by a processor of a local sound reinforcement device to implement any of the methods described in the above embodiments.

[0204] The non-temporary computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc., and the present application does not limit this.

[0205] In an exemplary embodiment, a computer program product including a computer program / instruction is further provided. The computer program / instruction can be executed by a processor of a local sound reinforcement device to implement any of the methods described in the above embodiments.

[0206] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0207] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the inventions claimed herein. The present application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

[0208] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A local sound reinforcement method, characterized in that: The method comprises: Extracting features from the received first time-frequency domain signal to obtain first time-frequency domain features; Perform feature extraction on the second time-frequency domain signal to be played to obtain second time-frequency domain features; Inputting the first time-frequency domain feature and the second time-frequency domain feature into a human voice extraction model to obtain a time-frequency mask output by the human voice extraction model, wherein the time-frequency mask is used to represent a human voice time-frequency domain signal in the first time-frequency domain signal, and the human voice extraction model is obtained by gradually training the model using training data of different difficulty levels based on the training progress of the initial model; Based on the time-frequency masking, suppressing acoustic factors in the first time-frequency domain signal to obtain the human voice time-frequency domain signal; the acoustic factors include: howling, environmental noise, room reverberation and echo residue; The human voice time-frequency domain signal is converted into a human voice time domain signal, and the human voice time domain signal is locally amplified.

2. The method according to claim 1, characterized in that Before inputting the first time-frequency domain feature and the second time-frequency domain feature into a human voice extraction model to obtain a time-frequency mask output by the human voice extraction model, the method further includes: Using the training data and the label data, the initial model is trained to obtain the trained human voice extraction model; The training data and label data are generated in a manner including: Simulating an echo path for the simulated second time domain signal of the previous frame to obtain a simulated echo signal; Simulating room transmission function for a human voice source to obtain a simulated human voice image, wherein the simulated human voice image includes: a simulated human voice time domain signal and a simulated room reverberation; Superimposing the simulated echo signal, the simulated human voice image and the simulated environmental noise to obtain a simulated first time domain signal; Performing echo cancellation on the simulated first time domain signal to obtain an echo cancelled signal; generating the training data based on the echo cancellation signal and the simulated second time domain signal; Based on the simulated human voice time domain signal, generating label data corresponding to the training data; Gain control and delay processing are performed on the echo cancellation signal to obtain a simulated second time domain signal of the current frame.

3. The method according to claim 2, characterized in that The initial model is trained using the training data and the label data to obtain the trained human voice extraction model, including: Based on the training progress of the initial model, selecting training data of different difficulty levels; Inputting the training data into the initial model to obtain a prediction result output by the initial model; Calculating a loss value according to the prediction result and the label data, and updating the parameters of the initial model; The training progress of the initial model is adjusted, and training data of the next difficulty level is selected for model training until preset conditions are met to obtain the human voice extraction model.

4. The method according to claim 1, characterized in that: The locally amplifying the human voice time domain signal includes: Performing gain control and delay processing on the human voice time domain signal to obtain a processed human voice time domain signal; Superimposing the processed human voice time domain signal and the background audio signal to obtain a target signal; The target signal is locally amplified using a loudspeaker.

5. The method according to claim 1, characterized in that Before extracting features from the received first time-frequency domain signal to obtain first time-frequency domain features, the method further includes: Receiving a first time domain signal using a microphone; Performing short-time Fourier transform on the first time domain signal to obtain the first time-frequency domain signal; Performing short-time Fourier transform on the second time domain signal to be played by the loudspeaker to obtain the second time-frequency domain signal; The converting the human voice time-frequency domain signal into a human voice time-domain signal comprises: Performing inverse short-time Fourier transform on the human voice time-frequency domain signal to obtain the human voice time-domain signal.

6. The method according to claim 5, characterized in that Before performing short-time Fourier transform on the first time domain signal to obtain the first time-frequency domain signal, the method further includes: Calculating a linear echo in the first time domain signal based on the second time domain signal and the predicted echo path; The linear echo in the first time domain signal is eliminated.

7. The method according to claim 1, characterized in that The human voice extraction model includes: a nonlinear transformation unit, a multi-layer convolution unit and a masking processing unit; The step of inputting the first time-frequency domain feature and the second time-frequency domain feature into a human voice extraction model to obtain a time-frequency mask output by the human voice extraction model comprises: Splicing the first time-frequency domain features to obtain a first spliced ​​feature, where the first time-frequency domain feature is feature data of the first time-frequency domain signal on multiple channels; splicing the second time-frequency domain features to obtain spliced ​​second splicing features, where the second time-frequency domain features are feature data of the second time-frequency domain signal on multiple channels; The nonlinear transformation unit performs nonlinear transformation on the first splicing feature and the second splicing feature to obtain nonlinear transformation data; The multi-layer convolution unit performs convolution processing on the nonlinear transformation data to obtain filtered data; The mask processing unit calculates and outputs the time-frequency mask based on the filtered data; the time-frequency mask consists of a real part and an imaginary part.

8. A local sound reinforcement device, characterized in that: The device comprises: A first extraction module, used to extract features from the received first time-frequency domain signal to obtain first time-frequency domain features; A second extraction module, used for extracting features of the second time-frequency domain signal to be played, to obtain second time-frequency domain features; A human voice extraction module, used for inputting the first time-frequency domain feature and the second time-frequency domain feature into a human voice extraction model, obtaining a time-frequency mask output by the human voice extraction model, wherein the time-frequency mask is used to represent the human voice time-frequency domain signal in the first time-frequency domain signal, and the human voice extraction model is obtained by gradually training the model using training data of different difficulty levels based on the training progress of the initial model; A factor suppression module, configured to suppress acoustic factors in the first time-frequency domain signal based on the time-frequency masking to obtain the human voice time-frequency domain signal; the acoustic factors include: howling, ambient noise, room reverberation and echo residue; The local sound reinforcement module is used to convert the human voice time-frequency domain signal into a human voice time domain signal, and perform local sound reinforcement on the human voice time domain signal.

9. A terminal device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 7 by running the executable instructions.

10. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product having a computer program / instructions stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images