Dynamic noise reduction and multi-speech enhancement method, system, device and medium

Abstract

The invention relates to the technical field of audio processing, in particular to a dynamic noise reduction and multi-speech enhancement method, system and device and a medium, and the method comprises the steps: obtaining a current sound signal and a historical sound signal, carrying out the audio curve capturing of the historical sound signal, obtaining an audio curve and a noise reduction label, training an audio curve noise reduction model according to the audio curve and the noise reduction label to obtain a trained audio curve noise reduction model, performing audio curve capture on the current sound signal to obtain a current audio curve of the current sound signal, and inputting the current audio curve into the audio curve noise reduction model for noise reduction to obtain a noise reduction result. And obtaining the current audio curve after noise reduction processing, generating a noise reduction sound signal based on the current audio curve after noise reduction processing, and sending the noise reduction sound signal to the user terminal equipment. According to the invention, the noise reduction effect of the audio is improved.

Description

Technical Field

[0001] The present application relates to the technical field of audio processing, and in particular to a method, system, device and medium for dynamic noise reduction and multi-speech enhancement. Background Art

[0002] When remotely recording or broadcasting video or remote video in classrooms, conference rooms and other places, usually only one microphone is set to collect the audio in the room. Different directions in the room are at different distances from the microphone, which makes the sound source at different distances. The collected audio is often unbalanced and interfered by external noise, resulting in poor sound quality output by the other receiving side, which seriously affects the audio-visual quality of users on the other receiving side and cannot meet the audio-visual needs.

[0003] At present, a preset noise model is often used to reduce noise and enhance the sound source of the sound collected by the indoor microphone. However, the preset noise model can only reflect the noise reduction effect in static environment scenes. For dynamically changing noise environments, there is insufficient noise reduction response, which leads to poor final noise reduction effect. Summary of the invention

[0004] In order to improve the noise reduction effect of audio and realize effective monitoring of the storage and reading activities of forensic data, the present application provides a method, system, device and medium for dynamic noise reduction and multi-speech enhancement.

[0005] In a first aspect, the present application provides a method for dynamic noise reduction and multi-speech enhancement, which adopts the following technical solution: A method for dynamic noise reduction and multi-speech enhancement, comprising: Obtain the current sound signal and historical sound signal collected by the microphone; Capturing an audio curve of the historical sound signal to obtain an audio curve corresponding to the historical sound signal and a noise reduction label corresponding to the audio curve, wherein the noise reduction label is a labeling of a classification result of the audio curve, and the classification result at least includes no noise reduction, noise reduction, a dynamic environment category when the audio curve is obtained, and a dynamic environment category when the historical sound signal is initially generated; The audio curve denoising model is trained according to the audio curve and the denoising label to obtain a trained audio curve denoising model; Capturing the audio curve of the current sound signal to obtain a current audio curve of the current sound signal, and inputting the current audio curve into the audio curve noise reduction model to perform noise reduction to obtain the current audio curve after noise reduction processing; A noise reduction audio signal is generated based on the current audio curve after the noise reduction processing, and the noise reduction audio signal is sent to a user terminal device.

[0006] 1. In a possible implementation, the audio curve denoising model is trained according to the audio curve and the denoising label to obtain a trained audio curve denoising model, including: Inputting the audio curve whose noise reduction label is not noise-reduced and meets a preset condition into the audio curve noise reduction model to generate a noise-reduced audio curve; Acquire an audio sample set, wherein the audio samples in the audio sample set include the denoised audio curve and the audio curve with a denoising label as denoised; Inputting the audio sample into the noise reduction determination model to obtain a determination result of the audio sample, wherein the determination result includes un-noised, noise-reduced, and a dynamic environment category of the audio sample; A loss function of the audio network is obtained according to the determination result, and parameters of the audio curve denoising model and the denoising determination model are adjusted respectively according to the loss function until the audio network converges, thereby determining a trained audio curve denoising model.

[0007] 2. In one possible implementation, the method further includes: Inputting any of the audio curves with the noise reduction label as not having noise reduction into a discrimination model to determine whether any of the audio curves with the noise reduction label as not having noise reduction belongs to noise, and the signal-to-noise ratio of the noise is lower than a preset signal-to-noise ratio; If the discrimination model determines that it does not belong to noise, then the noise reduction label is that any of the audio curves without noise reduction meets the preset condition.

[0008] 3. In a possible implementation, the discrimination model includes a linear layer and a residual network, and the determining whether any of the audio samples labeled as not denoised belongs to noise includes: Upgrading the dimension of any of the audio curves whose noise reduction label is not noise reduction through the linear layer, and adjusting the audio curve after the dimension increase into two-dimensional data through the reshape function; The two-dimensional data is input into a residual network for feature extraction to output a judgment result of whether it is noise.

[0009] 4. In a possible implementation, the audio curve denoising model adopts a U-shaped network, the U-shaped network includes an encoder and a decoder, and generating the denoised audio curve includes: Performing feature extraction on the audio curve whose noise reduction label is not noise reduced and meets a preset condition through an encoder, and obtaining a first feature; The decoder is used to perform dimension upgrading and information recovery on the first feature to obtain fused data with the same dimension as the audio curve whose noise reduction label is not noise reduced and meets the preset conditions; The fused data is added to the audio curve whose noise reduction label is not noise-reduced and meets a preset condition to obtain the noise-reduced audio curve.

[0010] 5. In a possible implementation, the noise reduction determination model includes a feature extraction layer, a maximum pooling layer, an average pooling layer, a first multi-layer perceptron, and a second multi-layer perceptron, and obtaining a determination result of the audio sample includes: Extracting features from the audio sample through the feature extraction layer to obtain a second feature; reducing the dimension of the second feature through the maximum pooling layer and the average pooling layer to obtain a third feature and a fourth feature respectively; The third feature and the fourth feature are input into a first multi-layer perceptron to obtain a judgment result on the dynamic environment category of the audio sample, and the third feature and the fourth feature are input into a second multi-layer perceptron to obtain a judgment result on whether the audio sample has been denoised.

[0011] In a second aspect, the present application provides a system for dynamic noise reduction and multi-speech enhancement, which adopts the following technical solution: A signal acquisition module is used to acquire the current sound signal and the historical sound signal collected by the microphone; A first capturing module is configured to capture an audio curve of the historical sound signal to obtain an audio curve corresponding to the historical sound signal and a noise reduction label corresponding to the audio curve, wherein the noise reduction label is a label for a classification result of the audio curve, and the classification result at least includes un-noised, noise-reduced, a dynamic environment category when the audio curve is obtained, and a dynamic environment category when the historical sound signal is initially generated; A model training module, used to train the audio curve denoising model according to the audio curve and the denoising label to obtain a trained audio curve denoising model; A second capturing module is used to capture the audio curve of the current sound signal to obtain the current audio curve of the current sound signal, and input the current audio curve into the audio curve noise reduction model to perform noise reduction to obtain the current audio curve after noise reduction processing; A signal sending module is used to generate a noise reduction audio signal based on the current audio curve after the noise reduction processing, and send the noise reduction audio signal to a user terminal device.

[0012] In a possible implementation, when the model training module trains the audio curve denoising model according to the audio curve and the denoising label to obtain the trained audio curve denoising model, it is specifically used to: Inputting the audio curve whose noise reduction label is not noise-reduced and meets a preset condition into the audio curve noise reduction model to generate a noise-reduced audio curve; Acquire an audio sample set, wherein the audio samples in the audio sample set include the denoised audio curve and the audio curve with a denoising label as denoised; Inputting the audio sample into the noise reduction determination model to obtain a determination result of the audio sample, wherein the determination result includes un-noised, noise-reduced, and a dynamic environment category of the audio sample; A loss function of the audio network is obtained according to the determination result, and parameters of the audio curve denoising model and the denoising determination model are adjusted respectively according to the loss function until the audio network converges, thereby determining a trained audio curve denoising model.

[0013] In another possible implementation, the system further includes: a noise reduction identification module and a condition determination module, wherein: The noise reduction identification module is used to input any of the audio curves with the noise reduction label as non-noise reduction into the identification model to determine whether any of the audio curves with the noise reduction label as non-noise reduction belongs to noise, and the signal-to-noise ratio of the noise is lower than a preset signal-to-noise ratio; The condition determination module is used for, when the discrimination model determines that it does not belong to noise, any of the audio curves with the noise reduction label as non-noise reduction meets the preset condition.

[0014] In another possible implementation, the discrimination model includes a linear layer and a residual network, and the noise reduction discrimination module is specifically used to: Upgrading the dimension of any of the audio curves whose noise reduction label is not noise reduction through the linear layer, and adjusting the audio curve after the dimension increase into two-dimensional data through the reshape function; The two-dimensional data is input into a residual network for feature extraction to output a judgment result of whether it is noise.

[0015] In another possible implementation, the audio curve denoising model adopts a U-shaped network, the U-shaped network includes an encoder and a decoder, and the generating of the denoised audio curve is specifically used for: Performing feature extraction on the audio curve whose noise reduction label is not noise reduced and meets a preset condition through an encoder, and obtaining a first feature; The decoder is used to perform dimension upgrading and information recovery on the first feature to obtain fused data with the same dimension as the audio curve whose noise reduction label is not noise reduced and meets the preset conditions; The fused data is added to the audio curve whose noise reduction label is not noise-reduced and meets a preset condition to obtain the noise-reduced audio curve.

[0016] In another possible implementation, the noise reduction determination model includes a feature extraction layer, a maximum pooling layer, an average pooling layer, a first multi-layer perceptron, and a second multi-layer perceptron, and obtaining the determination result of the audio sample is specifically used for: Extracting features from the audio sample through the feature extraction layer to obtain a second feature; reducing the dimension of the second feature through the maximum pooling layer and the average pooling layer to obtain a third feature and a fourth feature respectively; The third feature and the fourth feature are input into a first multi-layer perceptron to obtain a judgment result on the dynamic environment category of the audio sample, and the third feature and the fourth feature are input into a second multi-layer perceptron to obtain a judgment result on whether the audio sample has been denoised.

[0017] In a third aspect, the present application provides an electronic device, which adopts the following technical solution: An electronic device, comprising: at least one processor; Memory; At least one application, wherein the at least one application is stored in a memory and configured to be executed by at least one processor, and the at least one application is configured to: execute the above-mentioned method of dynamic noise reduction and multi-speech enhancement.

[0018] In a fourth aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution: A computer-readable storage medium includes: a computer program that can be loaded by a processor and execute the above-mentioned dynamic noise reduction and multi-speech enhancement method.

[0019] In summary, the present application includes at least one of the following beneficial technical effects: When collecting and transmitting indoor sound through a microphone, in order to reduce the noise in the sound collected by the microphone and enhance the collected voice, The present application adopts a method of dynamic noise reduction and multi-speech enhancement. Specifically, a current sound signal and a historical sound signal collected by a microphone are obtained, and an audio curve is captured on the historical sound signal after coal combustion to obtain an audio curve corresponding to the historical sound signal and a noise reduction label corresponding to the audio curve. The noise reduction label is a labeling of a classification result of the audio curve, and the classification result at least includes un-noised, noise-reduced, a dynamic environment category when the audio curve is obtained, and a dynamic environment category when the historical sound signal is initially generated. Then, an audio curve noise reduction model is trained according to the audio curve and the noise reduction label to obtain a trained audio curve noise reduction model, and then an audio curve is captured on the current sound signal to obtain a current audio curve of the current sound signal, and the current audio curve is input into the audio curve noise reduction model for noise reduction to obtain a current audio curve after noise reduction processing, and then a noise-reduced sound signal is generated based on the current audio curve after noise reduction processing, and the noise-reduced sound signal is sent to a user terminal device, so that before the sound collected by the microphone is transmitted to the user terminal device, the sound is subjected to noise reduction enhancement processing, thereby achieving an effect of improving audio noise reduction. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a schematic diagram of the architecture of a method for dynamic noise reduction and multi-speech enhancement in an embodiment of the present application; Figure 2 It is a structural diagram of a system for dynamic noise reduction and multi-speech enhancement in an embodiment of the present application; Figure 3 It is a structural schematic diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0021] The following is combined with Figure 1-3 This application is described in further detail.

[0022] After reading this specification, those skilled in the art may make non-creative modifications to this embodiment as needed, but such modifications are protected by patent law as long as they are within the scope of the claims of this application.

[0023] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0024] The embodiment of the present application provides a method for dynamic noise reduction and multi-speech enhancement, which is performed by an electronic device, which can be a server or a terminal device, wherein the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal device can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc., but is not limited thereto. The terminal device and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiment of the present application.

[0025] refer to Figure 1 , Figure 1 : is a flow chart of a method for dynamic noise reduction and multi-speech enhancement in an embodiment of the present application, the method comprising: Step S10: Acquire the current sound signal and the historical sound signal collected by the microphone.

[0026] For the embodiment of the present application, the microphone is a sound collection device, and its main function is to capture the sound signal in the environment in real time through the microphone array and other technologies. When it is necessary to obtain the current sound signal, the microphone will perform preliminary processing (such as amplification, noise reduction, etc.) on the sound signal collected by the microphone array, and then convert it into an electrical signal or a digital signal and output it to the electronic device.

[0027] Historical sound signals usually refer to sound data that has been collected and stored by a sound pickup. These data may be stored in a hard disk video recorder (DVR), a network storage device (NAS) or other forms of storage devices. When acquiring historical sound signals, the electronic device accesses these storage devices and finds the audio files under the corresponding time period or event label.

[0028] Step S11: Capture the audio curve of the historical sound signal to obtain the audio curve corresponding to the historical sound signal and the noise reduction label corresponding to the audio curve.

[0029] The noise reduction label is a label for the classification result of the audio curve, and the classification result at least includes un-noised, noise reduced, the dynamic environment category when the audio curve is obtained, and the dynamic environment category when the historical sound signal is initially generated.

[0030] For the embodiments of the present application, the audio curve, that is, the waveform representation of the sound signal, can intuitively display the characteristics of the sound signal changing over time. Capturing the audio curve generally includes the following steps: Signal reading: reading the audio file of the historical sound signal from the storage device. Signal preprocessing: performing necessary preprocessing on the audio signal, such as sampling rate conversion, normalization, etc., for subsequent analysis. Waveform drawing: using audio processing software or programming tools, the audio signal is converted into a waveform graph, that is, an audio curve. The audio curve generally shows the change of the amplitude (or volume) of the sound over time.

[0031] The noise reduction label is a label for the classification result of the audio curve, which is used to identify the noise reduction state of the audio signal and the category of the environment in which the sound is generated. The process of generating noise reduction labels usually includes the following steps: Feature extraction: Extract features that can characterize the characteristics of the audio signal from the audio curve, such as spectral features, time domain features, cepstrum features, etc. These features can reflect the sound quality, volume, pitch and other properties of the audio signal. Classifier training: Use machine learning or deep learning algorithms to train a classifier to identify the noise reduction state and dynamic environment category of the audio signal. The classifier can be trained using supervised learning, that is, using annotated training data sets to adjust the parameters of the model. The training data set should include audio signal samples with different noise reduction states and dynamic environment categories. Label generation: Apply the trained classifier to the audio curve of the historical sound signal, classify each audio curve, and generate the corresponding noise reduction label. The noise reduction label can include the labels of un-noised and noise-reduced, as well as the dynamic environment category when the audio curve was obtained (such as indoor, outdoor, traffic noise, etc.) and the dynamic environment category when the historical sound signal was initially generated (such as conference room, factory, concert, etc.).

[0032] Step S12: training the audio curve denoising model according to the audio curve and the denoising label to obtain a trained audio curve denoising model.

[0033] Specifically, an audio curve with a denoising label of not denoised and meeting preset conditions is input into an audio curve denoising model to generate a denoised audio curve, and an audio sample set is obtained. The audio samples in the audio sample set include denoised audio curves and audio curves with a denoising label of denoised. The audio samples are input into a denoising determination model to obtain a determination result of the audio samples, and the determination result includes the undenoised, denoised and dynamic environment categories of the audio samples. A loss function of the audio network is obtained according to the determination result, and the parameters of the audio curve denoising model and the denoising determination model are adjusted according to the loss function until the audio network converges, and a trained audio curve denoising model is determined.

[0034] Step S13: Capture the audio curve of the current sound signal to obtain the current audio curve of the current sound signal, and input the current audio curve into the audio curve noise reduction model to perform noise reduction to obtain the current audio curve after noise reduction processing.

[0035] For the embodiment of the present application, audio data of the current sound signal is obtained from a microphone or other audio acquisition device using audio processing software or programming tools. The audio data is converted into an audio curve (i.e., a waveform graph), which is usually achieved by drawing a graph of the amplitude of the audio signal changing over time. The audio curve is then preprocessed before the current audio curve is input into the audio curve denoising model. This includes but is not limited to: standardizing or normalizing the amplitude of the audio signal to ensure the consistency of the model input. The features of the audio curve are extracted, and these features characterize the denoising requirements of the audio signal. Model input: The preprocessed current audio curve (or its feature representation) is provided as input to the trained audio curve denoising model. Noise reduction processing: The audio curve denoising model will receive the current audio curve as input, and perform denoising processing on the audio curve based on its learned denoising capability and feature representation. The model outputs the feature representation of the denoised audio curve, or directly outputs the denoised audio data.

[0036] Step S14: generating a noise reduction audio signal based on the current audio curve after the noise reduction process, and sending the noise reduction audio signal to the user terminal device.

[0037] For the embodiment of the present application, when the sound in the room is collected and transmitted by the microphone, in order to reduce the noise in the sound collected by the microphone and enhance the collected voice, The present application adopts a method of dynamic noise reduction and multi-speech enhancement. Specifically, a current sound signal and a historical sound signal collected by a microphone are obtained, and an audio curve is captured on the historical sound signal after coal combustion to obtain an audio curve corresponding to the historical sound signal and a noise reduction label corresponding to the audio curve. The noise reduction label is a labeling of a classification result of the audio curve, and the classification result at least includes un-noised, noise-reduced, a dynamic environment category when the audio curve is obtained, and a dynamic environment category when the historical sound signal is initially generated. Then, an audio curve noise reduction model is trained according to the audio curve and the noise reduction label to obtain a trained audio curve noise reduction model, and then an audio curve is captured on the current sound signal to obtain a current audio curve of the current sound signal, and the current audio curve is input into the audio curve noise reduction model for noise reduction to obtain a current audio curve after noise reduction processing, and then a noise-reduced sound signal is generated based on the current audio curve after noise reduction processing, and the noise-reduced sound signal is sent to a user terminal device, so that before the sound collected by the microphone is transmitted to the user terminal device, the sound is subjected to noise reduction enhancement processing, thereby achieving an effect of improving audio noise reduction.

[0038] Furthermore, the method also includes: inputting any audio curve with a noise reduction label of not having noise reduction into a recognition model to determine whether any audio curve with a noise reduction label of not having noise reduction belongs to noise, and the signal-to-noise ratio of the noise is lower than a preset signal-to-noise ratio; if the recognition model determines that it does not belong to noise, then any audio curve with a noise reduction label of not having noise reduction meets the preset conditions.

[0039] Specifically, the discrimination model includes a linear layer and a residual network, and determines whether any audio sample labeled as not denoised is noise, including: upgrading the dimension of any audio curve labeled as not denoised through a linear layer, and adjusting the upgraded audio curve to two-dimensional data through a reshape function, and inputting the two-dimensional data into a residual network for feature extraction to output a judgment result of whether it is noise.

[0040] Furthermore, the audio curve denoising model adopts a U-shaped network, which includes an encoder and a decoder. Generating the denoised audio curve includes: extracting features of the audio curve whose denoising label is not denoised and meets preset conditions through the encoder, and obtaining a first feature, and using the decoder to perform dimension upgrading and information recovery on the first feature to obtain fused data with the same dimension as the audio curve whose denoising label is not denoised and meets the preset conditions, and adding the fused data to the audio curve whose denoising label is not denoised and meets the preset conditions to obtain the denoised audio curve.

[0041] Furthermore, the denoising judgment model includes a feature extraction layer, a maximum pooling layer, an average pooling layer, a first multi-layer perceptron and a second multi-layer perceptron, and the judgment result of the audio sample includes: extracting features of the audio sample through the feature extraction layer and obtaining a second feature, reducing the dimension of the second feature through the maximum pooling layer and the average pooling layer respectively to obtain a third feature and a fourth feature respectively, inputting the third feature and the fourth feature into the first multi-layer perceptron to obtain a judgment result of the dynamic environment category of the audio sample, and inputting the third feature and the fourth feature into the second multi-layer perceptron to obtain a judgment result of whether the audio sample has been denoised.

[0042] The above embodiment introduces a method of dynamic noise reduction and multi-speech enhancement from the perspective of method flow, and the following embodiment introduces a system of dynamic noise reduction and multi-speech enhancement from the perspective of a virtual module or a virtual unit. For details, please refer to the following embodiment.

[0043] The present application embodiment provides a system for dynamic noise reduction and multi-speech enhancement, such as Figure 2 As shown, the system may specifically include: The signal acquisition module 21 is used to acquire the current sound signal and the historical sound signal collected by the microphone; The first capture module 22 is used to capture the audio curve of the historical sound signal, and obtain the audio curve corresponding to the historical sound signal and the noise reduction label corresponding to the audio curve, where the noise reduction label is the annotation of the classification result of the audio curve, and the classification result at least includes no noise reduction, noise reduction, the dynamic environment category when the audio curve is obtained, and the dynamic environment category when the historical sound signal is initially generated; A model training module 23 is used to train the audio curve denoising model according to the audio curve and the denoising label to obtain a trained audio curve denoising model; The second capturing module 24 is used to capture the audio curve of the current sound signal to obtain the current audio curve of the current sound signal, and input the current audio curve into the audio curve noise reduction model to perform noise reduction to obtain the current audio curve after noise reduction processing; The signal sending module 25 is used to generate a noise reduction audio signal based on the current audio curve after the noise reduction process, and send the noise reduction audio signal to the user terminal device.

[0044] In a possible implementation of the embodiment of the present application, when the model training module 23 trains the audio curve denoising model according to the audio curve and the denoising label to obtain the trained audio curve denoising model, it is specifically used to: Inputting the audio curve whose noise reduction label is not noise reduction and meets the preset conditions into the audio curve noise reduction model to generate the audio curve after noise reduction; An audio sample set is obtained, where the audio samples in the audio sample set include audio curves after noise reduction and audio curves with noise reduction labels indicating that noise has been reduced; Inputting the audio sample into the noise reduction determination model to obtain a determination result of the audio sample, the determination result including un-noised, noise-reduced, and a dynamic environment category of the audio sample; The loss function of the audio network is obtained according to the judgment result, and the parameters of the audio curve denoising model and the denoising judgment model are adjusted according to the loss function respectively until the audio network converges, and the trained audio curve denoising model is determined.

[0045] In another possible implementation of the embodiment of the present application, the system 20 further includes: a noise reduction identification module and a condition determination module, wherein: A noise reduction identification module is used to input any audio curve with a noise reduction label of not having noise reduction into the identification model to determine whether any audio curve with a noise reduction label of not having noise reduction belongs to noise, and the signal-to-noise ratio of the noise is lower than a preset signal-to-noise ratio; The condition determination module is used to determine that when the discrimination model determines that it does not belong to noise, any audio curve with a noise reduction label of not denoised satisfies the preset condition.

[0046] In another possible implementation of the embodiment of the present application, the discrimination model includes a linear layer and a residual network, and the noise reduction discrimination module is specifically used to determine whether any audio sample labeled as non-noise-reduced is noise: Use the linear layer to increase the dimension of any audio curve whose denoising label is not denoised, and use the reshape function to adjust the audio curve after denoising to two-dimensional data; The two-dimensional data is input into the residual network for feature extraction to output the judgment result whether it is noise.

[0047] In another possible implementation of the embodiment of the present application, the audio curve noise reduction model adopts a U-shaped network, and the U-shaped network includes an encoder and a decoder, and generates a noise-reduced audio curve specifically for; Extract features of an audio curve whose noise reduction label is not noise reduced and meets a preset condition through an encoder, and obtain a first feature; The decoder is used to perform dimension upgrading and information recovery on the first feature to obtain fused data with the same dimension as the audio curve whose denoising label is not denoised and meets the preset conditions; The fused data is added to the audio curve whose denoising label is not denoised and meets the preset conditions to obtain the denoised audio curve.

[0048] In another possible implementation of the embodiment of the present application, the noise reduction determination model includes a feature extraction layer, a maximum pooling layer, an average pooling layer, a first multi-layer perceptron, and a second multi-layer perceptron to obtain a determination result of the audio sample, which is specifically used for: The feature extraction layer extracts features from the audio sample and obtains a second feature; the second feature is reduced in dimension by a maximum pooling layer and an average pooling layer, respectively, to obtain a third feature and a fourth feature; The third feature and the fourth feature are input into the first multi-layer perceptron to obtain a judgment result on the dynamic environment category of the audio sample, and the third feature and the fourth feature are input into the second multi-layer perceptron to obtain a judgment result on whether the audio sample has been denoised.

[0049] An electronic device provided in an embodiment of the present application is introduced below. The electronic device described below and the method for dynamic noise reduction and multi-speech enhancement described above can refer to each other.

[0050] The present application embodiment provides an electronic device, such as Figure 3 As shown, Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, Figure 3The electronic device 300 shown includes: a processor 301 and a memory 303. The processor 301 and the memory 303 are connected, such as through a bus 302. Optionally, the electronic device 300 may also include a transceiver 304. It should be noted that in actual applications, the transceiver 304 is not limited to one, and the structure of the electronic device 300 does not constitute a limitation on the embodiments of the present application.

[0051] The processor 301 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of the embodiments of the present application. The processor 301 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0052] The bus 302 may include a path to transmit information between the above components. The bus 302 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 302 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0053] The memory 303 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0054] The memory 303 is used to store application code for executing the solution of the embodiment of the present application, and the execution is controlled by the processor 301. The processor 301 is used to execute the application code stored in the memory 303 to implement the content shown in the above method embodiment.

[0055] Among them, electronic devices include but are not limited to: mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 3 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0056] A computer-readable storage medium provided in an embodiment of the present application is introduced below. The computer-readable storage medium described below and the method described above can be referenced to each other.

[0057] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method for dynamic noise reduction and multi-speech enhancement are implemented.

[0058] Since the embodiments of the computer-readable storage medium part and the embodiments of the method part correspond to each other, the embodiments of the computer-readable storage medium part refer to the description of the embodiments of the method part.

[0059] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0060] The above are only some implementation methods of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for dynamic noise reduction and multi-speech enhancement, characterized in that: include: Obtain the current sound signal and historical sound signal collected by the microphone; Capturing an audio curve of the historical sound signal to obtain an audio curve corresponding to the historical sound signal and a noise reduction label corresponding to the audio curve, wherein the noise reduction label is a labeling of a classification result of the audio curve, and the classification result at least includes no noise reduction, noise reduction, a dynamic environment category when the audio curve is obtained, and a dynamic environment category when the historical sound signal is initially generated; The audio curve denoising model is trained according to the audio curve and the denoising label to obtain a trained audio curve denoising model; Capturing the audio curve of the current sound signal to obtain a current audio curve of the current sound signal, and inputting the current audio curve into the audio curve noise reduction model to perform noise reduction to obtain the current audio curve after noise reduction processing; A noise reduction audio signal is generated based on the current audio curve after the noise reduction processing, and the noise reduction audio signal is sent to a user terminal device.

2. The method of dynamic noise reduction and multi-speech enhancement according to claim 1, characterized in that: The step of training the audio curve denoising model according to the audio curve and the denoising label to obtain a trained audio curve denoising model includes: Inputting the audio curve whose noise reduction label is not noise-reduced and meets a preset condition into the audio curve noise reduction model to generate a noise-reduced audio curve; Acquire an audio sample set, wherein the audio samples in the audio sample set include the denoised audio curve and the audio curve with a denoising label as denoised; Inputting the audio sample into the noise reduction determination model to obtain a determination result of the audio sample, wherein the determination result includes un-noised, noise-reduced, and a dynamic environment category of the audio sample; A loss function of the audio network is obtained according to the determination result, and parameters of the audio curve denoising model and the denoising determination model are adjusted respectively according to the loss function until the audio network converges, thereby determining a trained audio curve denoising model.

3. The method of dynamic noise reduction and multi-speech enhancement according to claim 2, characterized in that: The method further comprises: Inputting any of the audio curves with the noise reduction label as not having noise reduction into a discrimination model to determine whether any of the audio curves with the noise reduction label as not having noise reduction belongs to noise, and the signal-to-noise ratio of the noise is lower than a preset signal-to-noise ratio; If the discrimination model determines that it does not belong to noise, then the noise reduction label is that any of the audio curves without noise reduction meets the preset condition.

4. The method of dynamic noise reduction and multi-speech enhancement according to claim 3, characterized in that: The discrimination model includes a linear layer and a residual network, and the determining whether any of the audio samples labeled as not denoised belongs to noise includes: Upgrading the dimension of any of the audio curves whose noise reduction label is not noise reduction through the linear layer, and adjusting the audio curve after the dimension increase into two-dimensional data through the reshape function; The two-dimensional data is input into a residual network for feature extraction to output a judgment result of whether it is noise.

5. The method of dynamic noise reduction and multi-speech enhancement according to claim 2, characterized in that: The audio curve denoising model adopts a U-shaped network, the U-shaped network includes an encoder and a decoder, and generating a denoised audio curve includes: Performing feature extraction on the audio curve whose noise reduction label is not noise reduced and meets a preset condition through an encoder, and obtaining a first feature; The decoder is used to perform dimension upgrading and information recovery on the first feature to obtain fused data with the same dimension as the audio curve whose noise reduction label is not noise reduced and meets the preset conditions; The fused data is added to the audio curve whose noise reduction label is not noise-reduced and meets a preset condition to obtain the noise-reduced audio curve.

6. The method of dynamic noise reduction and multi-speech enhancement according to claim 2, characterized in that: The noise reduction determination model includes a feature extraction layer, a maximum pooling layer, an average pooling layer, a first multi-layer perceptron, and a second multi-layer perceptron, and the determination result of the audio sample includes: Extracting features from the audio sample through the feature extraction layer to obtain a second feature; reducing the dimension of the second feature through the maximum pooling layer and the average pooling layer to obtain a third feature and a fourth feature respectively; The third feature and the fourth feature are input into a first multi-layer perceptron to obtain a judgment result on the dynamic environment category of the audio sample, and the third feature and the fourth feature are input into a second multi-layer perceptron to obtain a judgment result on whether the audio sample has been denoised.

7. A system for dynamic noise reduction and multi-speech enhancement, characterized in that: include: A signal acquisition module is used to acquire the current sound signal and the historical sound signal collected by the microphone; A first capturing module is configured to capture an audio curve of the historical sound signal to obtain an audio curve corresponding to the historical sound signal and a noise reduction label corresponding to the audio curve, wherein the noise reduction label is a label for a classification result of the audio curve, and the classification result at least includes un-noised, noise-reduced, a dynamic environment category when the audio curve is obtained, and a dynamic environment category when the historical sound signal is initially generated; A model training module, used to train the audio curve denoising model according to the audio curve and the denoising label to obtain a trained audio curve denoising model; A second capturing module is used to capture the audio curve of the current sound signal to obtain the current audio curve of the current sound signal, and input the current audio curve into the audio curve noise reduction model to perform noise reduction to obtain the current audio curve after noise reduction processing; A signal sending module is used to generate a noise reduction audio signal based on the current audio curve after the noise reduction processing, and send the noise reduction audio signal to a user terminal device.

8. The system for dynamic noise reduction and multi-speech enhancement according to claim 7, characterized in that: The system further comprises: an audio attribution module and a condition determination module, wherein: The audio attribution module is used to input any of the audio curves with the noise reduction label as non-noise reduction into the discrimination model to determine whether any of the audio curves with the noise reduction label as non-noise reduction belongs to noise, and the signal-to-noise ratio of the noise is lower than a preset signal-to-noise ratio; The condition determination module is used for, when the discrimination model determines that it does not belong to noise, any of the audio curves with the noise reduction label as non-noise reduction meets the preset condition.

9. An electronic device, characterized in that: The electronic device includes: at least one processor; Memory; At least one application, wherein the at least one application is stored in a memory and configured to be executed by at least one processor, and the at least one application is configured to: execute the method of dynamic noise reduction and multi-speech enhancement according to any one of claims 1-6.

10. A computer-readable storage medium, characterized in that: include: A computer program is stored which can be loaded by a processor and execute the method for dynamic noise reduction and multi-speech enhancement as claimed in any one of claims 1 to 6.

Images