A method, system and terminal device for separating target noise

Abstract

The present invention provides a method, system and terminal device for separating target noise, including: constructing a deep neural network model based on an encoder and training to obtain a deep neural network track separation model; obtaining a mixed audio signal to be separated having target noise and environmental noise, inputting its mixed noise amplitude feature time domain map into the deep neural network track separation model, fusing the context features and amplitude features extracted from the mixed noise amplitude feature time domain map through the fusion layer of the encoder to obtain the amplitude feature time domain map of the target noise; the context features characterize the distribution structure information of each sound source in the mixed audio signal to be separated; and obtaining the time domain signal of the target noise according to the amplitude feature time domain map of the target noise. The present invention can obtain high-precision amplitude features and context features by using the deep neural network track separation model, and can accurately separate environmental noise and target noise, solving the current defect that accuracy and fidelity cannot be taken into account at the same time.

Description

Technical Field

[0001] The present invention relates to the technical field of target noise separation, and in particular to a target noise separation method, system and terminal equipment. Background Art

[0002] With the growing demand for noise control, identifying the type of noise source has become a core function of monitoring equipment, and noise separation is an indispensable part of noise identification. Since the audio environment where the noise is located is mostly complex, mixed with many kinds of natural environmental sounds, such as: birdsong, wind, rain, etc., and multiple different sounds are mixed into one channel, making noise separation more and more difficult.

[0003] Current technology usually uses deep neural network models to achieve noise separation. This method has stronger feature extraction and learning capabilities than traditional methods. Generally, convolutional layers and pooling layers are added to the model to extract sound source feature information from the audio signal at a deeper level, thereby improving recognition accuracy.

[0004] However, as the number of convolution and pooling layers increases, more nonlinear processing will be introduced, which will cause greater distortion in the separation results. In other words, in the noise separation task, it is difficult for the existing technology to balance high accuracy and audio quality of the separation results. Summary of the invention

[0005] One of the purposes of the present invention is to provide a method, system and terminal device for separating target noise, which can improve the accuracy and audio quality of target noise separation. The embodiments of the present invention can be implemented as follows:

[0006] In a first aspect, the present invention provides a target noise separation method, the method comprising: step S1, constructing a deep neural network model based on an encoder, and training the deep neural network model to obtain a deep neural network track separation model; the encoder is provided with a fusion layer; step S2, obtaining a mixed audio signal to be separated; wherein the mixed audio signal to be separated has target noise and environmental noise; step S3, obtaining a mixed noise amplitude feature time domain map according to the mixed audio signal to be separated; step S4, inputting the mixed noise amplitude feature time domain map into the deep neural network track separation model, extracting context features and amplitude features from the mixed noise amplitude feature time domain map through the encoder, and fusing the context features and the amplitude features through the fusion layer to obtain the amplitude feature time domain map of the target noise; wherein the context features characterize the distribution structure information of each sound source in the mixed audio signal to be separated; step S5, obtaining the time domain signal of the target noise according to the amplitude feature time domain map of the target noise.

[0007] In an optional embodiment, the encoder includes an encoding part and a decoding part; the encoding part includes an input layer and multiple groups of first network layers, each group of the first network layers includes two convolutional layers and a maximum pooling layer, and the decoding part includes an output layer and multiple groups of second network layers, each group of the second network layers includes an upsampling layer, a fusion layer and two convolutional layers.

[0008] In an optional embodiment, step S1 specifically includes the following steps: step S11, obtaining a training sample; the training sample includes the target noise and at least one environmental noise; step S12, inputting the training sample into the deep neural network model for training to obtain the deep neural network track separation model.

[0009] In an optional implementation, the step S11 specifically includes the following steps: step S111, obtaining the audio signals of the target noise and at least one of the environmental noises; step S112, mixing the audio signals of the target noise and at least one of the environmental noises to obtain a plurality of mixed audio signals; step S113, amplifying or reducing the amplitude of each of the mixed audio signals to obtain mixed audio signals under different sound intensity states, and using the mixed audio signals under different sound intensity states as the training samples.

[0010] In an optional implementation, the step S3 specifically includes the following steps: step S31, downsampling the mixed audio signal to be separated to obtain a mixed audio signal after downsampling; step S32, performing short-time Fourier transform on the mixed audio signal after downsampling to obtain a time domain diagram of the mixed noise amplitude characteristics.

[0011] In an optional implementation, the step S5 specifically includes the following steps: step S51, performing an inverse short-time Fourier transform on the amplitude characteristic time domain map of the target noise to obtain an initial time domain signal; step S52, upsampling the initial time domain signal according to the sampling rate of the deep neural network track separation model to obtain the time domain signal of the target noise.

[0012] In a second aspect, the present invention provides a target noise separation method system, comprising: a construction and training module, which is used to construct a deep neural network model based on an encoder, and train the deep neural network model to obtain a deep neural network track separation model; the encoder is provided with a fusion layer; an acquisition module, which is also used to obtain a mixed audio signal to be separated; wherein the mixed audio signal to be separated has target noise and environmental noise; a transformation module, which is used to obtain a mixed noise amplitude feature time domain map based on the mixed audio signal to be separated; a separation module, which is used to input the mixed noise amplitude feature time domain map into the deep neural network track separation model, extract context features and amplitude features from the mixed noise amplitude feature time domain map through the encoder, and fuse the context features and the amplitude features through the fusion layer to obtain the amplitude feature time domain map of the target noise; wherein the context features represent the distribution structure information of each sound source in the mixed audio signal to be separated; the acquisition module is also used to obtain the time domain signal of the target noise based on the amplitude feature time domain map of the target noise.

[0013] In an optional embodiment, the encoder includes an encoding part and a decoding part; the encoding part includes an input layer and multiple groups of first network layers, each group of the first network layers includes two convolutional layers and a maximum pooling layer, and the decoding part includes an output layer and multiple groups of second network layers, each group of the second network layers includes an upsampling layer, a fusion layer and two convolutional layers.

[0014] In an optional embodiment, the acquisition module is specifically used to: acquire training samples; the training samples include the target noise and at least one environmental noise; input the training samples into the deep neural network model for training to obtain the deep neural network track separation model.

[0015] In a third aspect, the present invention provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the target noise separation method described in any one of the aforementioned embodiments when executing the computer program.

[0016] The present invention provides a method, system and terminal device for separating target noise. First, a deep neural network model is constructed based on an encoder, and the deep neural network model is trained to obtain a deep neural network track separation model. Then, the mixed noise amplitude feature time domain map of the mixed audio signal to be separated is input into the deep neural network track separation model. The encoder extracts and fuses the context feature and the amplitude feature. The context feature information can characterize the distribution structure of each sound source of the signal, which is helpful for locating and identifying the target noise. The amplitude feature finely expresses the details of the time domain change, which is conducive to accurately reconstructing the time domain waveform of the target signal. Making full use of multi-dimensional information can improve the separation effect. The fusion layer realizes the combination of feature levels to obtain the amplitude feature map of the target noise, which contains both global context and local detail information. According to the amplitude feature map of the target noise, the time domain signal of the target noise can be finely restored, greatly improving the separation quality. The deep neural network track separation model of the present invention can not only obtain high-precision amplitude features, but also obtain context features, so that the environmental noise and the target noise can be better separated, and the fidelity effect can be achieved, which solves the defect that the prior art cannot take into account both accuracy and fidelity at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 A schematic flow chart of a target noise separation method provided by an embodiment of the present invention;

[0019] Figure 2 A schematic diagram of the structure of a deep neural network audio track separation model provided by an embodiment of the present invention;

[0020] Figure 3 A functional module diagram of a target noise separation system 300 provided in an embodiment of the present invention;

[0021] Figure 4 A schematic diagram of the structure of a terminal device 400 provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0023] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0025] In the description of the present invention, it should be noted that if the terms "upper", "lower", "inside", "outside", etc. appear to indicate an orientation or position relationship, they are based on the orientation or position relationship shown in the accompanying drawings, or are the orientation or position relationship in which the product of the invention is usually placed when used. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0026] In addition, the terms “first”, “second”, etc., if used, are merely used to distinguish between the descriptions and should not be understood as indicating or implying relative importance.

[0027] It should be noted that, in the absence of conflict, the features in the embodiments of the present invention may be combined with each other.

[0028] The main difficulty in noise separation is that the ambient noise and the target noise are mixed to form a single-channel signal. In order to improve the separation accuracy, common deep learning models must add convolution and pooling layers, but this will inevitably introduce a greater degree of reconstruction distortion. Based on the time-frequency characteristics of mixed noise, an embodiment of the present invention provides a target noise separation method that can take into account both accuracy and fidelity in the noise separation task and achieve high-quality extraction of target noise.

[0029] See also Figure 1 , Figure 1 A schematic flow chart of a target noise separation method provided by an embodiment of the present invention may include the following steps:

[0030] Step S1, constructing a deep neural network model based on the encoder, and training the deep neural network model to obtain a deep neural network track separation model; the encoder is provided with a fusion layer;

[0031] Step S2, obtaining a mixed audio signal to be separated; wherein the mixed audio signal to be separated has target noise and environmental noise;

[0032] Step S3, obtaining a time domain graph of mixed noise amplitude characteristics according to the mixed audio signal to be separated;

[0033] Step S4, inputting the mixed noise amplitude feature time domain graph into the deep neural network track separation model, extracting context features and amplitude features from the mixed noise amplitude feature time domain graph through an encoder, and fusing the context features and amplitude features through a fusion layer to obtain the amplitude feature time domain graph of the target noise; wherein the context features represent the distribution structure information of each sound source in the mixed audio signal to be separated;

[0034] Step S5, obtaining the time domain signal of the target noise according to the amplitude characteristic time domain diagram of the target noise.

[0035] In the technical scheme of the above steps S1 to S5, a deep neural network model is first constructed based on an encoder, and the deep neural network model is trained to obtain a deep neural network track separation model, and then the mixed noise amplitude feature time domain map of the mixed audio signal to be separated is input into the trained deep neural network track separation model, and the context feature and amplitude feature are extracted and fused by the encoder. The context feature information can characterize the distribution structure of each sound source of the signal, which is helpful for locating and identifying the target noise. The amplitude feature finely expresses the details of the time domain change, which is conducive to accurately reconstructing the time domain waveform of the target signal. Making full use of multi-dimensional information can improve the separation effect, and the feature level combination is realized through the fusion layer to obtain the amplitude feature map of the target noise, which contains both global context and local detail information. According to the amplitude feature map of the target noise, the time domain signal of the target noise can be finely restored, greatly improving the separation quality. The deep neural network track separation model of the present invention can not only obtain high-precision amplitude features, but also obtain context features, so that the environmental noise and the target noise can be better separated, and the fidelity effect can be achieved, which solves the defect that the prior art cannot take into account both accuracy and fidelity at the same time.

[0036] The above steps are introduced and explained in detail below.

[0037] In step S1, in order to achieve high-quality extraction of target noise, the embodiment of the present invention provides a deep neural network track separation model, which improves the structure of the encoder in the traditional deep neural network, so that the deep neural network track separation model can minimize the distortion of the reconstruction result while ensuring high separation accuracy. At the same time, due to the small number of parameters, the model power consumption is also low.

[0038] In an optional implementation, the deep neural network track separation model provided by the embodiment of the present invention is as follows: Figure 2 As shown, Figure 2 A schematic diagram of the structure of an encoder for a deep neural network audio track separation model provided in an embodiment of the present invention.

[0039] like Figure 2 As shown, the encoder includes an encoding part and a decoding part. The encoding part includes an input layer and multiple groups of first network layers, each group of first network layers includes two convolutional layers and a maximum pooling layer, and the decoding part includes an output layer and multiple groups of second network layers, each group of second network layers includes an upsampling layer, a fusion layer and two convolutional layers.

[0040] Among them, N groups of network layers are set in the encoding part, each group of network layers consists of 2 convolutional layers and a pooling layer, and the convolutional layer activation function adopts ReLU. The decoding part is a mirror image of the encoding part and also contains three groups. Each group starts with an upsampling layer, followed by a fusion layer, and then two convolutional layers. The output layer is the (1, 1) convolutional layer output.

[0041] In the embodiment of the present invention, the limitations of the traditional autoencoder are taken into account: for example, if a large block of images is input, a pooling layer needs to be added, and too many pooling layers damage the accuracy of feature extraction. However, if a small block of subdivided images is input, the acquisition of context information by the neural network will be limited, and false detection and missed detection are prone to occur. Therefore, the embodiment of the present invention makes the above improvements to the encoder, and the improved encoder can extract context features through the fusion layer, and can provide structural information of the distribution of the sound source in the time and frequency domain. This structural information helps to identify the positional morphological features of each sound source in the time and frequency domain. In addition, the context features can also capture the characteristics of the entire mixed sound field. This is conducive to judging the relative position relationship of each sound source in the sound field. In addition, different types of sound sources, such as human voice and environmental noise, often have differences in time and frequency structure. Context features can identify these differences and help to classify and identify sound sources. In the fusion layer, the context features and amplitude features are effectively fused, which establishes the association between different parts of the signal through the context features. Give full play to the global dependency relationship, and the quality of track separation can be improved. In short, the context features extracted by the encoder provide rich structural information, scene information and classification information. These multidimensional features jointly improve the separation effect of target noise.

[0042] In an optional implementation, in order to improve the separation effect of the target noise, the deep neural network track separation model provided in the embodiment of the present invention, in addition to adding a fusion layer, also makes the following two improvements to the model based on the characteristics of the noisy audio:

[0043] First, the size of the convolution kernel is changed from the general (3, 3) to (5, 3). Since the height of the time-frequency graph is much larger than the width, the embodiment of the present invention increases the height of the convolution kernel.

[0044] The second point is to make the output size of the convolutional layer consistent with the input size, so that the output of the encoding part can be directly connected to the fusion layer of the decoding part. This can preserve the complete data to the greatest extent.

[0045] In summary, the deep neural network audio track separation model provided by the embodiment of the present invention has the advantages of simple separation framework, low latency, high speed, and a small number of weights. It can reduce distortion and output high-precision features that are not pooled.

[0046] In order to obtain the above-mentioned deep neural network track separation model, the embodiment of the present invention further provides a model training method, that is, the above-mentioned step S1 may include the following steps:

[0047] Step S11, obtaining a training sample; the training sample includes a target noise and at least one environmental noise;

[0048] In the embodiment of the present invention, step S11 may include the following steps:

[0049] Step S111, obtaining audio signals of target noise and at least one type of environmental noise;

[0050] Step S112, mixing the audio signals of the target noise and at least one ambient noise to obtain a plurality of mixed audio signals;

[0051] Step S113, amplifying or reducing the amplitude of each mixed audio signal to obtain mixed audio signals in different sound intensity states, and using the mixed audio signals in different sound intensity states as training samples.

[0052] By acquiring signals from different noise sources, complex mixed scenes can be restored. Mixing different noises to generate training samples can improve the generalization ability of the model under different conditions. Adjusting the sound intensity of the mixed signal allows the model to learn the impact of sound intensity on separation, thereby improving robustness under different noise ratios. Richer and more realistic training samples can help the model better learn signal features and mixing rules. During the training process, the model learns not only the features, but also the impact of environmental deviations such as sound intensity on the features. Finally, the applicability and separation stability of the model under complex and changing conditions are improved. Through data enhancement strategies and enriching sample diversity, the accuracy of the trained deep neural network track separation model in separating target noise can be improved.

[0053] Step S12: input the training samples into the deep neural network model for training to obtain a deep neural network track separation model.

[0054] In this embodiment of the present invention, the loss function used in the training process is as follows:

[0055]

[0056] Among them, cost is the loss function, s(t, g) is the mixed audio signal, and s1(t, g) is the audio signal of the target noise.

[0057] During the training process, the loss value of the loss function is calculated, and the loss value is back-propagated to the separation model, and the model parameters of the separation model are adjusted. When the loss function converges or reaches a preset number of iterations, the training is stopped. At this time, the separation model is the deep neural network audio track separation model provided in an embodiment of the present invention.

[0058] In step S3, in order to shorten the time of target noise separation and improve the efficiency of target noise separation, the embodiment of the present invention provides an implementation method, that is, step S3 may include:

[0059] Step S31, downsampling the mixed audio signal to be separated to obtain a mixed audio signal after downsampling;

[0060] Step S32, performing short-time Fourier transform on the mixed audio signal after the downsampling process to obtain a time domain diagram of the mixed noise amplitude characteristics.

[0061] Through downsampling, the time domain resolution of the mixed audio signal to be separated can be reduced. This is conducive to the subsequent short-time Fourier transform processing and reduces the impact of time-frequency resolution on algorithm performance and computational complexity. Compared with directly performing short-time Fourier transform on the original signal, downsampling can obtain a feature map with a higher balance of time resolution and frequency resolution. Perform a short-time Fourier transform on the downsampled signal to extract the time domain map of the mixed noise amplitude feature. During the short-time Fourier transform process, the window selected by the sliding window can be but is not limited to the Henning window. The time domain map of the mixed noise amplitude feature captures the strength and weakness change characteristics of the signal at different time and frequency points. This provides rich input features for the subsequent deep neural network track separation model and enhances the resolution ability of the target noise.

[0062] In step S5, in order to effectively improve the separation quality and restoration accuracy, the embodiment of the present invention provides an implementation method for obtaining the target noise time domain signal, that is, step S5 may include the following steps:

[0063] Step S51, performing an inverse short-time Fourier transform on the amplitude characteristic time domain image of the target noise to obtain an initial time domain signal;

[0064] Step S52, upsampling the initial time domain signal according to the sampling rate of the deep neural network track separation model to obtain the time domain signal of the target noise.

[0065] The mixed noise amplitude feature time domain map is transformed into the initial time domain signal through the inverse short-time Fourier transform. This restores the separation result from the frequency domain to the original time domain signal form. However, due to the time-frequency resolution limitation in the inverse short-time Fourier transform process, the resolution of the initial time domain signal may not match the model requirements. Therefore, according to the sampling rate of the deep neural network track separation model, the sampling rate of the initial signal is increased by upsampling to meet the sampling standard of the model output. This can ensure that the sampling of the separated target signal completely restores all the time domain detail information of the original audio. Upsampling can also make up for the loss of time-frequency resolution caused by the inverse short-time Fourier transform and restore the details of the target signal. The final output time domain signal can restore the real time domain waveform structure of the target noise to the greatest extent, effectively improving the separation quality and restoration accuracy.

[0066] Based on the same inventive concept, an embodiment of the present invention provides a target noise separation system, see Figure 3 , Figure 3 The functional module diagram of the target noise separation system provided by the embodiment of the present invention includes: a construction and training module 310, an acquisition module 320, a transformation module 330 and a separation module 340;

[0067] A construction and training module 310 is used to construct a deep neural network model based on the encoder, and train the deep neural network model to obtain a deep neural network track separation model; the encoder is provided with a fusion layer;

[0068] The acquisition module 320 is further used to acquire a mixed audio signal to be separated; wherein the mixed audio signal to be separated has target noise and environmental noise;

[0069] A transformation module 330 is used to obtain a time domain graph of mixed noise amplitude characteristics according to the mixed audio signal to be separated;

[0070] The separation module 340 is used to input the mixed noise amplitude feature time domain map into the deep neural network track separation model, extract context features and amplitude features from the mixed noise amplitude feature time domain map through an encoder, and fuse the context features and amplitude features through a fusion layer to obtain the amplitude feature time domain map of the target noise; wherein the context features represent the distribution structure information of each sound source in the mixed audio signal to be separated;

[0071] The acquisition module 320 is further used to obtain the time domain signal of the target noise according to the amplitude characteristic time domain diagram of the target noise.

[0072] It is understood that the construction and training module 310, the acquisition module 320, the transformation module 330 and the separation module 340 can be executed in a coordinated manner. Figure 1 Each step in the process is performed to achieve the corresponding technical effect.

[0073] In an optional embodiment, the encoder includes an encoding part and a decoding part; the encoding part includes an input layer and multiple groups of first network layers, each group of first network layers includes two convolutional layers and a maximum pooling layer, and the decoding part includes an output layer and multiple groups of second network layers, each group of second network layers includes an upsampling layer, a fusion layer and two convolutional layers.

[0074] In an optional embodiment, a construction and training module 310 is used to: obtain training samples; the training samples include target noise and at least one environmental noise; input the training samples into a deep neural network model for training to obtain a deep neural network track separation model.

[0075] In an optional embodiment, the acquisition module 320 is specifically used to obtain the audio signals of the target noise and at least one environmental noise; mix the audio signals of the target noise and at least one environmental noise to obtain multiple mixed audio signals; amplify or reduce the amplitude of each mixed audio signal to obtain mixed audio signals under different sound intensity states, and use the mixed audio signals under different sound intensity states as training samples.

[0076] In an optional implementation, the transformation module 330 is specifically used to downsample the mixed audio signal to be separated to obtain a downsampled mixed audio signal; and perform short-time Fourier transform on the downsampled mixed audio signal to obtain a time domain diagram of the mixed noise amplitude characteristics.

[0077] In an optional embodiment, the acquisition module 320 is further specifically used to perform an inverse short-time Fourier transform on the amplitude characteristic time domain map of the target noise to obtain an initial time domain signal; and upsample the initial time domain signal according to the sampling rate of the deep neural network track separation model to obtain the time domain signal of the target noise.

[0078] It should be noted that the division of modules in the above embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0079] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a terminal device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program code.

[0080] The embodiment of the present invention further provides a terminal device 400 for deploying Figure 3 The target noise separation system 300 shown in FIG. Figure 4 , Figure 4 The schematic diagram of the structure of the terminal device 400 provided in the embodiment of the present invention. The terminal device 400 may include a memory 401, a processor 402 and a communication interface 403. The memory 401, the processor 402 and the communication interface 403 are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0081] The memory 401 can be used to store software programs and modules, such as the program instructions / modules corresponding to the target noise separation system 300 provided in the present application. The processor 402 executes the software programs and modules stored in the memory 401 to perform various functional applications and data processing, thereby executing the steps of the live assistant interaction method provided in the present application. The communication interface 403 can be used to communicate signaling or data with other node devices.

[0082] Among them, the memory 401 can be, but is not limited to, random access memory (Random Access Memory, RAM), read only memory (Read Only Memory, ROM), programmable read-only memory (Programmable Read-Only Memory, PROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, EPROM), electrically erasable programmable read-only memory (Electric Erasable Programmable Read-Only Memory, EEPROM), etc.

[0083] The processor 402 may be an integrated circuit chip with signal processing capability. The processor 402 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0084] Understandably, Figure 4 The structure shown is for illustration only. The terminal device 400 may also include Figure 4 More or fewer components as shown, or with Figure 4 Different configurations are shown. Figure 4 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0085] Based on the above embodiments, the present application further provides a storage medium, in which a computer program is stored. When the computer program is executed by a computer, the computer executes the target noise separation method provided in the above embodiments.

[0086] Based on the above embodiments, the embodiments of the present application further provide a computer program. When the computer program is run on a computer, the computer executes the target noise separation method provided in the above embodiments.

[0087] Based on the above embodiments, an embodiment of the present application further provides a chip, which is used to read a computer program stored in a memory to execute the target noise separation method provided in the above embodiments.

[0088] A computer program product is also provided in an embodiment of the present application, including instructions, which, when executed on a computer, enable the computer to execute the target noise separation method provided in the above embodiment.

[0089] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by instructions. These instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0090] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0091] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0092] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A target noise separation method, characterized in that: The method comprises: Step S1, constructing a deep neural network model based on an encoder, and training the deep neural network model to obtain a deep neural network track separation model; the encoder is provided with a fusion layer; the encoder includes an encoding part and a decoding part; the encoding part includes an input layer and multiple groups of first network layers, each group of the first network layers includes two convolutional layers and a maximum pooling layer, the decoding part includes an output layer and multiple groups of second network layers, each group of the second network layers includes an upsampling layer, a fusion layer and two convolutional layers; the fusion layer is used to extract context features; In the decoding part, the input of the fusion layer in each of the second network layers is the output of the upsampling layer in the network layer; in the process of training the deep neural network track separation model, the loss function used in the training process is as follows: in, is the loss function, is a mixed audio signal, is the audio signal of the target noise; Step S2, obtaining a mixed audio signal to be separated; wherein the mixed audio signal to be separated has target noise and environmental noise; Step S3, obtaining a time domain graph of mixed noise amplitude characteristics according to the mixed audio signal to be separated; Step S4, inputting the mixed noise amplitude feature time domain graph into the deep neural network track separation model, extracting context features and amplitude features from the mixed noise amplitude feature time domain graph through the encoder, and fusing the context features and the amplitude features through the fusion layer to obtain the amplitude feature time domain graph of the target noise; wherein the context features represent the distribution structure information of each sound source in the mixed audio signal to be separated; Step S5, obtaining the time domain signal of the target noise according to the amplitude characteristic time domain diagram of the target noise.

2. A target noise separation method according to claim 1, characterized in that: Step S1 specifically includes the following steps: Step S11, obtaining a training sample; the training sample includes the target noise and at least one environmental noise; Step S12: input the training samples into the deep neural network model for training to obtain the deep neural network track separation model.

3. A target noise separation method according to claim 2, characterized in that: The step S11 specifically includes the following steps: Step S111, obtaining audio signals of the target noise and at least one of the environmental noises; Step S112, mixing the target noise with the audio signal of at least one of the environmental noises to obtain a plurality of mixed audio signals; Step S113, amplifying or reducing the amplitude of each mixed audio signal to obtain mixed audio signals in different sound intensity states, and using the mixed audio signals in different sound intensity states as the training samples.

4. A target noise separation method according to claim 1, characterized in that: The step S3 specifically comprises the following steps: Step S31, downsampling the mixed audio signal to be separated to obtain a mixed audio signal after downsampling; Step S32, performing short-time Fourier transform on the mixed audio signal after the downsampling process to obtain the time domain diagram of the mixed noise amplitude characteristics.

5. A target noise separation method according to claim 1, characterized in that: The step S5 specifically comprises the following steps: Step S51, performing an inverse short-time Fourier transform on the amplitude characteristic time domain image of the target noise to obtain an initial time domain signal; Step S52: up-sample the initial time domain signal according to the sampling rate of the deep neural network track separation model to obtain the time domain signal of the target noise.

6. A target noise separation system, characterized in that: include: A construction and training module, used to construct a deep neural network model based on an encoder, and train the deep neural network model to obtain a deep neural network track separation model; the encoder is provided with a fusion layer; the encoder includes an encoding part and a decoding part; the encoding part includes an input layer and multiple groups of first network layers, each group of the first network layers includes two convolutional layers and a maximum pooling layer, the decoding part includes an output layer and multiple groups of second network layers, each group of the second network layers includes an upsampling layer, a fusion layer and two convolutional layers; the fusion layer is used to extract context features; In the decoding part, the input of the fusion layer in each of the second network layers is the output of the upsampling layer in the network layer; in the process of training the deep neural network track separation model, the loss function used in the training process is as follows: in, is the loss function, is a mixed audio signal, is the audio signal of the target noise; The acquisition module is further used to acquire a mixed audio signal to be separated; wherein the mixed audio signal to be separated has target noise and environmental noise; A transformation module, used for obtaining a time domain diagram of mixed noise amplitude characteristics according to the mixed audio signal to be separated; A separation module, used for inputting the mixed noise amplitude feature time domain graph into the deep neural network track separation model, extracting context features and amplitude features from the mixed noise amplitude feature time domain graph through the encoder, and fusing the context features and the amplitude features through the fusion layer to obtain the amplitude feature time domain graph of the target noise; wherein the context features represent the distribution structure information of each sound source in the mixed audio signal to be separated; The acquisition module is further used to obtain the time domain signal of the target noise according to the amplitude characteristic time domain diagram of the target noise.

7. A target noise separation system according to claim 6, characterized in that: The acquisition module is specifically used to: acquire training samples; the training samples include the target noise and at least one environmental noise; input the training samples into the deep neural network model for training to obtain the deep neural network track separation model.

8. A terminal device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, a target noise separation method according to any one of claims 1 to 5 is implemented.

Images