A training method, device and equipment for speech enhancement model

Abstract

The embodiments of the present application provide a method, apparatus, and device for training a speech enhancement model, which relates to the field of speech processing and recognition technology and is used to reduce speech distortion and suppress noise in audio through a speech enhancement model. In this method, a training sample set is first obtained; the first audio sample in the training sample set is preprocessed through the input layer of the speech enhancement model to obtain first audio sample data; the audio features of the first audio sample data are extracted through N hidden layers of the speech enhancement model; the audio features are respectively input into M output layers of the speech enhancement model to obtain M audio noise reduction results; the losses between the M audio noise reduction results and the audio masking results are determined through the loss functions corresponding to the M output layers to obtain M loss values; and the network parameters of the input layer, N hidden layers, and M output layers are adjusted according to the weighted results of the M loss values ​​to obtain a trained speech enhancement model.

Description

Technical Field

[0001] The present application relates to the field of speech processing and recognition technology, and in particular to a training method, apparatus and equipment for a speech enhancement model. Background Art

[0002] When speech enhancement technology is applied to call systems, it typically suppresses echo and noise in the audio to improve the signal-to-noise ratio and signal-to-response ratio, thereby enhancing the user experience. When applied to speech recognition systems, however, it needs to more completely preserve the voice signal and minimize distortion to improve recognition rates.

[0003] However, current technologies typically require switching between different speech enhancement models to suppress noise in call systems and reduce distortion in speech recognition systems. In other words, there's currently no single speech enhancement model that can simultaneously reduce distortion and suppress noise. Summary of the Invention

[0004] The embodiments of the present application provide a method, apparatus, and device for training a speech enhancement model, which can be used to reduce speech distortion and suppress noise in audio through a speech enhancement model.

[0005] In a first aspect, an embodiment of the present application provides a method for training a speech enhancement model, comprising:

[0006] Obtain a training sample set, wherein the training sample set includes multiple audio sample data and audio masking results corresponding to the multiple audio sample data respectively; obtain the first audio sample data in the training sample set through the input layer of the speech enhancement model; extract the audio features of the first audio sample data through the N hidden layers of the speech enhancement model, where N is a positive integer; input the audio features into the M output layers of the speech enhancement model respectively to obtain M audio noise reduction results, where M is an integer greater than 1; determine the losses between the M audio noise reduction results and the audio masking results respectively through the loss functions corresponding to the M output layers to obtain M loss values; adjust the network parameters of the input layer, the N hidden layers, and the M output layers according to the weighted results of the M loss values ​​to obtain the trained speech enhancement model.

[0007] Based on the above scheme, since the speech enhancement model includes multiple output layers, the speech enhancement model trained by this method can achieve different noise reduction processing effects on the processed speech signal by switching the network parameters of the output layer.

[0008] In one possible implementation, determining the losses between the M audio noise reduction results and the audio masking results respectively through loss functions corresponding to the M output layers to obtain M loss values ​​includes: determining a first loss value between the audio noise reduction result corresponding to a first output layer and the audio masking result through a first loss function; the first output layer is any output layer among the M output layers; and the first loss function satisfies the following formula:

[0009]

[0010] Among them, Loss1 is the first loss value, n is the frequency point number of the audio feature, W1 is the weight corresponding to the first loss function, y pred1,f is the audio noise reduction result corresponding to the first output layer of the fth frequency point of the audio feature, y true,f is the audio masking result of the f-th frequency point.

[0011] Based on the above scheme, the use of this loss function can make the penalty for residual noise greater than the penalty for excessive noise elimination, so that the speech enhancement model has a stronger noise reduction effect and improves the signal-to-noise ratio.

[0012] In one possible implementation, the weight corresponding to the first loss function is related to the audio masking result; the weight satisfies the following formula:

[0013]

[0014] Where W1 is the weight corresponding to the first loss function, α is a preset constant, and y true,f is the audio masking result of the f-th frequency point.

[0015] Based on the above scheme, the smaller the audio masking result is, the higher the weight is, so that during the training process of the speech enhancement model, the frequency point with a lower signal-to-noise ratio has a higher weight, thereby enhancing the noise reduction effect of the speech enhancement model.

[0016] In one possible implementation, the M output layers include a first output layer and a second output layer; determining the losses between the M audio noise reduction results and the audio masking results respectively through loss functions corresponding to the M output layers to obtain M loss values ​​includes: determining a second loss value between the audio noise reduction result corresponding to the second output layer and the audio masking result through a second loss function; the second output layer is any output layer among the M output layers that is different from the first output layer; and the second loss function satisfies the following formula:

[0017]

[0018] Among them, Loss2 is the second loss value, n is the frequency point number of the audio feature, W2 is the weight corresponding to the second loss function, y pred2,f is the audio noise reduction result corresponding to the second output layer of the fth frequency point of the audio feature, y true,f is the audio masking result of the f-th frequency point.

[0019] Based on the above scheme, the use of this loss function can make the penalty for residual noise smaller than the penalty for over-elimination of noise, so that the speech enhancement model can better retain the speech signal in the audio sample and reduce the distortion of the audio sample.

[0020] In one possible implementation, the weight corresponding to the second loss function is related to the audio masking result; the weight satisfies the following formula:

[0021]

[0022] Where W2 is the weight corresponding to the second loss function, β is a preset constant, and y true,f is the audio masking result of the f-th frequency point.

[0023] Based on the above scheme, the smaller the audio masking result is, the higher the weight is, so that during the training process of the speech enhancement model, the frequency point with a lower signal-to-noise ratio has a higher weight, thereby enhancing the noise reduction effect of the speech enhancement model.

[0024] In the second aspect, an embodiment of the present application provides an audio processing method based on a speech enhancement model, including: obtaining an audio signal to be processed; determining that the noise reduction mode for denoising the audio signal to be processed is a first mode; obtaining audio features of the audio signal to be processed through the input layer and at least one hidden layer of the speech enhancement model; the speech enhancement model also includes M output layers, M is an integer greater than 1, and the noise reduction mode includes M modes including the first mode, and the M modes correspond one-to-one to the M output layers, and the noise reduction degrees of the M modes are different, and the network parameters corresponding to the M output layers are different; the audio features are input into the first output layer corresponding to the M output layers of the first mode to obtain the audio noise reduction result of the audio signal to be processed.

[0025] Based on the above scheme, different audio noise reduction results can be obtained by inputting audio features into different output layers, so that switching between multiple noise reduction modes can be achieved through one model.

[0026] In a possible implementation, the speech enhancement model is trained using the training method of the speech enhancement model of the first aspect.

[0027] In a third aspect, an embodiment of the present application provides a training device for a speech enhancement model, comprising: an acquisition unit for acquiring a training sample set, wherein the training sample set includes multiple audio samples and audio masking results corresponding to the multiple audio samples; a processing unit for preprocessing the first audio sample in the training sample set through the input layer of the speech enhancement model to obtain first audio sample data; extracting the audio features of the first audio sample data through the N hidden layers of the speech enhancement model, where N is a positive integer; inputting the audio features into the M output layers of the speech enhancement model to obtain M audio noise reduction results, where M is an integer greater than 1; determining the losses between the M audio noise reduction results and the audio masking results through the loss functions corresponding to the M output layers to obtain M loss values; and adjusting the network parameters of the input layer, the N hidden layers, and the M output layers according to the weighted results of the M loss values ​​to obtain the trained speech enhancement model.

[0028] In one possible implementation, when the processing unit determines the losses between the M audio noise reduction results and the audio masking results respectively through the loss functions corresponding to the M output layers to obtain M loss values, it is used to: determine a first loss value between the audio noise reduction result corresponding to the first output layer and the audio masking result through a first loss function; the first output layer is any output layer among the M output layers; the first loss function satisfies the following formula:

[0029]

[0030] Among them, Loss1 is the first loss value, n is the frequency point number of the audio feature, W1 is the weight corresponding to the first loss function, y pred1,f is the audio noise reduction result corresponding to the first output layer of the fth frequency point of the audio feature, y true,f is the audio masking result of the f-th frequency point.

[0031] In one possible implementation, the weight corresponding to the first loss function is related to the audio masking result; the weight satisfies the following formula:

[0032]

[0033] Where W1 is the weight corresponding to the first loss function, α is a preset constant, and y true,f is the audio masking result of the f-th frequency point.

[0034] In one possible embodiment, the M output layers include a first output layer and a second output layer; when the processing unit determines the losses between the M audio noise reduction results and the audio masking results respectively through the loss functions corresponding to the M output layers to obtain M loss values, it is used to: determine a second loss value between the audio noise reduction result corresponding to the second output layer and the audio masking result through a second loss function; the second output layer is any output layer in the M output layers that is different from the first output layer; the second loss function satisfies the following formula:

[0035]

[0036] Among them, Loss2 is the second loss value, n is the frequency point number of the audio feature, W2 is the weight corresponding to the second loss function, y pred2,f is the audio noise reduction result corresponding to the second output layer of the fth frequency point of the audio feature, y true,f is the audio masking result of the f-th frequency point.

[0037] In one possible implementation, the weight corresponding to the second loss function is related to the audio masking result; the weight satisfies the following formula:

[0038]

[0039] Where W2 is the weight corresponding to the second loss function, β is a preset constant, and y true,f is the audio masking result of the f-th frequency point.

[0040] In a fourth aspect, an embodiment of the present application provides an audio processing device based on a speech enhancement model, comprising: an acquisition unit for acquiring an audio signal to be processed; a determination unit for determining that a noise reduction mode for performing noise reduction on the audio signal to be processed is a first mode; a processing unit for acquiring audio features of the audio signal to be processed through an input layer and at least one hidden layer of the speech enhancement model; the speech enhancement model also includes M output layers, M is an integer greater than 1, the noise reduction mode includes M modes including the first mode, the M modes correspond one-to-one to the M output layers, the noise reduction degrees of the M modes are different, and the network parameters corresponding to the M output layers are different; the audio features are input into the first output layer corresponding to the M output layers of the first mode to obtain the audio noise reduction result of the audio signal to be processed.

[0041] In one possible implementation, the speech enhancement model used by the device is trained using the above-mentioned speech enhancement model training method.

[0042] In a fifth aspect, an embodiment of the present application provides a chip system comprising a memory and a processor; the processor and the memory are coupled; wherein the memory comprises program instructions, and when the program instructions are executed by the processor, the chip system executes any one of the methods in the first and second aspects.

[0043] In a sixth aspect, an embodiment of the present application provides an electronic device, comprising: a communication interface for receiving a training sample set; a processor for obtaining the training sample set from the communication interface to execute the method described in the first aspect and different implementations of the first aspect.

[0044] In a seventh aspect, an embodiment of the present application provides a terminal device, comprising: a processor configured to execute the method described in the second aspect and the different implementations of the second aspect; a speaker configured to play an audio signal after audio noise reduction. In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium storing computer instructions that, when executed on a computer, cause the computer to execute any of the methods described in the first and second aspects.

[0045] For each of the above-mentioned aspects from the second to the seventh aspect and the technical effects that may be achieved by each aspect, please refer to the above-mentioned description of the technical effects that can be achieved by the first aspect or various possible solutions in the first aspect, and no further details will be given here. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present application.

[0047] Figure 1 A schematic diagram of a usage scenario of a display device provided in an embodiment of the present application;

[0048] Figure 2 A configuration block diagram of a control device 100 provided in an embodiment of the present application;

[0049] Figure 3 A hardware configuration block diagram of a display device 200 provided in an embodiment of the present application;

[0050] Figure 4 A schematic diagram of the software architecture of a terminal device provided in an embodiment of the present application;

[0051] Figure 5A A schematic diagram of the system architecture provided in an embodiment of the present application;

[0052] Figure 5BAnother system architecture diagram provided for an embodiment of the present application;

[0053] Figure 6 A schematic diagram of the electronic device structure provided in an embodiment of the present application;

[0054] Figure 7 This is one of the schematic diagrams of the speech enhancement model provided in the embodiment of the present application;

[0055] Figure 8 A schematic diagram of the hidden layer structure of the speech enhancement model provided in an embodiment of the present application;

[0056] Figure 9 A schematic diagram of the output layer structure of the speech enhancement model provided in an embodiment of the present application;

[0057] Figure 10 An exemplary flow chart of a method for training a speech enhancement model provided in an embodiment of the present application;

[0058] Figure 11 A schematic diagram of the structure of a speech enhancement model based on a convolutional neural network provided in an embodiment of the present application;

[0059] Figure 12 This is one of the schematic diagrams of the speech enhancement model structure provided in the embodiment of the present application;

[0060] Figure 13 This is one of the loss calculation diagrams provided in the embodiment of the present application;

[0061] Figure 14 This is one of the loss calculation diagrams provided in the embodiment of the present application;

[0062] Figure 15 A second loss function curve diagram provided in an embodiment of the present application;

[0063] Figure 16 This is one of the loss calculation diagrams provided in the embodiment of the present application;

[0064] Figure 17 An exemplary flow chart of an audio processing method based on a speech enhancement model provided in an embodiment of the present application;

[0065] Figure 18 This is a schematic diagram of one of the methods for the execution device to determine the noise reduction mode provided in an embodiment of the present application;

[0066] Figure 19 This is a schematic diagram of one of the methods for the execution device to determine the noise reduction mode provided in an embodiment of the present application;

[0067] Figure 20 This is one of the noise reduction mode switching diagrams provided in an embodiment of the present application;

[0068] Figure 21 This is one of the noise reduction mode switching diagrams provided in an embodiment of the present application. DETAILED DESCRIPTION

[0069] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of the technical solutions of this application, but not all of them. Based on the embodiments described in this application document, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the technical solutions of this application.

[0070] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for protection, but merely represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present application.

[0071] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

[0072] The Ideal Ratio Mask (IRM) used in the embodiments of this application calculates the energy ratio between the speech signal and noise, obtaining a value between 0 and 1. The energy of the time-frequency unit is then adjusted accordingly. The IRM is an evolution of the Ideal Binary Mask (IBM), reflecting the degree of noise suppression at each time-frequency unit, further improving the quality and intelligibility of the separated speech.

[0073] The following briefly introduces the application scenarios to which the technical solutions of the embodiments of the present application can be applied. It should be noted that the application scenarios described below are only used to illustrate the embodiments of the present application and are not limiting. In specific implementation, the technical solutions provided by the embodiments of the present application can be flexibly applied according to actual needs.

[0074] The training method of the speech enhancement model and the audio processing method based on the speech enhancement model provided in the embodiments of the present application can be implemented by an execution device. In some embodiments, the execution device for executing the training method of the speech enhancement model can be an electronic device, and the electronic device can include a server, a desktop computer, a laptop computer, etc. The execution device for executing the audio processing method based on the speech enhancement model can be a terminal device. The terminal device can be a display device with a voice interaction function. The display device can include: a mobile phone, a television, a tablet computer, a laptop computer, a PDA, a vehicle-mounted device, a wearable device, etc., which is not limited in this application.

[0075] The following describes the structure and application scenarios of the execution device by taking the execution device as a display device with a voice interaction function as an example. Figure 1 Schematic diagram of the use scenario of the display device in the embodiment. Figure 1 As shown, the display device 200 can also communicate data with the server 400, and the user can operate the display device 200 through the smart device 300 or the control device 100. In one possible example, the server 400 can execute the training method of the speech enhancement model and provide it to the display device 200, and the display device 200 can execute the audio processing method based on the speech enhancement model. In another possible example, the training method of the speech enhancement model can be executed by a server other than the server 400 and provided to the display device 200. The audio processing method based on the speech enhancement model can be executed by the server 400 or by the display device 200.

[0076] In some embodiments, the control device 100 may be a remote controller, and communication between the remote controller and the display device may include infrared protocol communication, Bluetooth protocol communication, or at least one other short-range communication method, to control the display device 200 wirelessly or wiredly. The user may control the display device 200 by inputting user commands through at least one of buttons on the remote controller, voice input, and control panel input.

[0077] In some embodiments, the smart device 300 may include any one of a mobile terminal, a tablet computer, a computer, a laptop computer, an AR / VR device, etc.

[0078] In some embodiments, the smart device 300 may also be used to control the display device 200. For example, the display device 200 may be controlled using an application running on the smart device.

[0079] In some embodiments, the smart device 300 and the display device 200 may also be used to communicate data.

[0080] In some embodiments, the display device 200 can also be controlled in a manner other than the control device 100 and the smart device 300. For example, the display device 200 can directly receive the user's voice command control through a module for obtaining voice commands configured inside the display device 200, or it can receive the user's voice command control through a voice control device set outside the display device 200.

[0081] In some embodiments, the display device 200 also communicates data with the server 400. The display device 200 may be connected to a local area network (LAN), a wireless local area network (WLAN), or other networks. The server 400 may provide various content and interactions to the display device 200. The server 400 may be a single cluster or multiple clusters, and may include one or more types of servers.

[0082] In some embodiments, the software steps executed by one step execution entity can be migrated to another step execution entity in data communication with it for execution as needed. For example, the software steps executed by a server can be migrated to a display device in data communication with it for execution as needed, and vice versa.

[0083] Figure 2 Schematically shows a block diagram of the configuration of the control device 100 according to an exemplary embodiment. Figure 2 As shown, the control device 100 includes a controller 110, a communication interface 130, a user input / output interface 140, a memory, and a power supply. The control device 100 can receive user input operation commands and convert the operation commands into commands that the display device 200 can recognize and respond to, acting as an interaction intermediary between the user and the display device 200.

[0084] In some embodiments, the communication interface 130 is used for external communication and includes at least one of a WIFI chip, a Bluetooth module, NFC, or an alternative module.

[0085] In some embodiments, the user input / output interface 140 includes at least one of a microphone, a touch pad, a sensor, a button, or alternative modules.

[0086] The embodiment will be described in detail below using the display device 200 as an example. It should be understood that Figure 3 The display device 200 shown is only an example, and the display device 200 may have more Figure 3 The more or less components shown in the figure can be combined with two or more components, or can have different component configurations. The various components shown in the figure can be implemented in hardware, software, or a combination of hardware and software including one or more signal processing and / or application specific integrated circuits.

[0087] Figure 3 FIG. 2 is a block diagram showing a hardware configuration of a display device 200 according to an exemplary embodiment.

[0088] In some embodiments, the display apparatus 200 includes at least one of a tuner 210 , a communicator 220 , a detector 230 , an external device interface 240 , a controller 250 , a display 260 , an audio output interface 270 , a memory, a power supply, and a user interface.

[0089] In some embodiments, the controller includes a central processing unit, a video processor, an audio processor, a graphics processor, a RAM, a ROM, and first to nth interfaces for input / output.

[0090] In some embodiments, the audio output interface 270 is used to output audio signals to other devices and may include components such as speakers and external audio output terminals for playing or transmitting audio. For example, the audio signal can be output to a speaker to play audio, or the audio signal can be output to other audio devices for playback or amplification.

[0091] In some embodiments, the display 260 includes a display screen component for presenting images, and a driving component for driving image display, a component for receiving image signals output from a controller, and a component for displaying video content, image content, and a menu control interface, as well as a user control UI interface, etc.

[0092] In some embodiments, the display 260 may be at least one of a liquid crystal display, an OLED display, and a projection display, and may also be a projection device and a projection screen.

[0093] In some embodiments, the tuner-demodulator 210 receives broadcast television signals via wired or wireless reception, and demodulates audio and video signals, such as EPG data signals, from a plurality of wireless or wired broadcast television signals.

[0094] In some embodiments, the communicator 220 is a component used to communicate with external devices or servers using various communication protocols. For example, the communicator may include at least one of a Wi-Fi module, a Bluetooth module, a wired Ethernet module, or other network communication protocol chip or a near-field communication protocol chip, as well as an infrared receiver. The display device 200 can use the communicator 220 to send and receive control signals and data signals with the control device 100 or the server 400.

[0095] In some embodiments, detector 230 is used to collect signals from the external environment or external interactions. For example, detector 230 may include a light receiver (not shown) for collecting ambient light intensity. Alternatively, detector 230 may include an image collector, such as a camera, for collecting external environmental scenes, user attributes, or user interaction gestures. Alternatively, detector 230 may include a sound collector, such as a microphone, for receiving external sounds.

[0096] In some embodiments, the external device interface 240 may include, but is not limited to, any one or more of the following: a high-definition multimedia interface (HDMI), an analog or digital high-definition component input interface (component), a composite video input interface (CVBS), a USB input interface (USB), an RGB port, etc. It may also be a composite input / output interface formed by multiple of the above interfaces.

[0097] In some embodiments, the controller 250 and the tuner 210 may be located in different separate devices, that is, the tuner 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0098] In some embodiments, controller 250 controls the operation of the display device and responds to user operations via various software control programs stored in memory. Controller 250 controls the overall operation of display device 200. For example, in response to receiving a user command to select a UI object for display on display 260, controller 250 may perform operations related to the object selected by the user command.

[0099] In some embodiments, the object may be any selectable object, such as a hyperlink, an icon, or other operable control. Operations associated with the selected object include: displaying a page, document, or image connected to the hyperlink, or executing a program corresponding to the icon.

[0100] In some embodiments, the controller includes at least one of a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), RAM Random Access Memory (RAM), ROM (Read-Only Memory, ROM), a first interface to an nth interface for input / output, a communication bus (Bus), etc.

[0101] The CPU processor is the control center of the display device 200, including the system-on-chip SOC, such as Figure 3As shown, the CPU is used to execute operating system and application instructions stored in memory, as well as various interactive instructions received from external input, to execute various applications, data, and content, ultimately displaying and playing various audio and video content. The CPU processor can include multiple processors, such as a main processor and one or more sub-processors.

[0102] In some embodiments, a graphics processor is configured to generate various graphical objects, such as at least one of icons, operation menus, and graphics displayed in response to user input commands. The graphics processor includes an arithmetic unit that performs operations based on various user input interactive commands and displays various objects according to display attributes; and a renderer that renders the various objects generated by the arithmetic unit. The rendered objects are then displayed on a display.

[0103] In some embodiments, the video processor is used to receive an external video signal and perform at least one of the following video processing operations, such as decompression, decoding, scaling, noise reduction, frame rate conversion, resolution conversion, and image synthesis, according to the standard codec protocol of the input signal, to obtain a signal that can be directly displayed or played on the display device 200.

[0104] In some embodiments, the video processor includes at least one of a demultiplexing module, a video decoding module, an image synthesis module, a frame rate conversion module, and a display formatting module. The demultiplexing module is used to demultiplex the input audio and video data stream. The video decoding module is used to process the demultiplexed video signal, including decoding and scaling. The image synthesis module, such as an image synthesizer, is used to superimpose and mix the GUI signal generated by the graphics generator according to user input or itself with the scaled video image to generate an image signal for display. The frame rate conversion module is used to convert the input video frame rate. The display formatting module is used to receive the frame rate converted video output signal and change the signal to conform to the display format signal, such as outputting an RGB data signal.

[0105] In some embodiments, the audio processor is used to receive an external audio signal, perform decompression and decoding according to the standard codec protocol of the input signal, and perform at least one of noise reduction, digital-to-analog conversion, and amplification processing to obtain a sound signal that can be played in a speaker.

[0106] In some embodiments, the user may input a user command through a graphical user interface (GUI) displayed on the display 260, and the user input interface receives the user input command through the graphical user interface (GUI). Alternatively, the user may input a user command through a specific sound or gesture, and the user input interface may recognize the sound or gesture through a sensor to receive the user input command.

[0107] In some embodiments, a "user interface" is a medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. A common form of user interface is a graphical user interface (GUI), which refers to a user interface related to computer operations that is displayed in a graphical manner. It can be an interface element such as an icon, window, or control displayed on the display screen of an electronic device, where a control can include at least one of a visual interface element such as an icon, button, menu, tab, text box, dialog box, status bar, navigation bar, or widget.

[0108] In some embodiments, the user interface 280 is an interface for receiving control input (eg, a physical button on the display device body, or other interfaces).

[0109] In some embodiments, the system of the display device may include a kernel, a command parser (shell), a file system, and applications. The kernel, shell, and file system together form the basic operating system structure, which allows users to manage files, run programs, and use the system. After power is turned on, the kernel starts, activates kernel space, abstracts hardware, initializes hardware parameters, runs and maintains virtual memory, schedulers, signals, and inter-process communication (IPC). After the kernel starts, the shell and user applications are loaded. After startup, the application is compiled into machine code to form a process.

[0110] See also Figure 4 In some embodiments, the system is divided into four layers, from top to bottom: the application layer (referred to as the "application layer"), the application framework layer (referred to as the "framework layer"), the Android runtime and system library layer (referred to as the "system runtime library layer"), and the kernel layer.

[0111] In some embodiments, at least one application runs in the application layer. These applications can be window programs, system settings programs, clock programs, etc. that come with the operating system, or applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the above examples.

[0112] The framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes predefined functions. The application framework layer acts as a processing center, determining the actions taken by applications in the application layer. Through the API, applications can access system resources and services during execution.

[0113] like Figure 4 As shown, in the embodiment of the present application, the application framework layer includes managers, content providers, etc., wherein the manager includes at least one of the following modules: an activity manager (ActivityManager) is used to interact with all activities running in the system; a location manager (Location Manager) is used to provide system services or applications with access to system location services; a package manager (Package Manager) is used to retrieve various information related to the application packages currently installed on the device; a notification manager (NotificationManager) is used to control the display and clearing of notification messages; a window manager (Window Manager) is used to manage icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0114] In some embodiments, the activity manager is used to manage the lifecycle of each application and common navigation back functions, such as controlling the exit, opening, and backing of an application. The window manager is used to manage all window programs, such as obtaining the display screen size, determining whether there is a status bar, locking the screen, taking screenshots, and controlling display window changes (such as shrinking the display window, shaking the display, distorting the display, etc.).

[0115] In some embodiments, the system runtime layer provides support for the upper layer, namely the framework layer. When the framework layer is used, the Android operating system will run the C / C++ library contained in the system runtime layer to implement the functions to be implemented by the framework layer.

[0116] In some embodiments, the kernel layer is a layer between hardware and software. Figure 4 As shown, the kernel layer includes at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0117] The display device 200 in the embodiment of the present application is not limited to a smart TV, but may also be an electronic device such as a mobile phone or a tablet computer.

[0118] In other embodiments, the execution device for executing the training method of the speech enhancement model can be implemented by one or more servers, and the server can be a local server or a cloud server.

[0119] See also Figure 5A As shown, the server 500 can be implemented by a physical server or a virtual server. The server can be implemented by a single server or a server cluster composed of multiple servers. The audio processing method based on the speech enhancement model or the training method of the speech enhancement model provided in this application can be implemented by a single server or a server cluster. Figure 5A In the example, a server 500 is connected to a terminal device 600 and a display device 200. For example, the server 500 can execute a method for training a speech enhancement model. The server 500 can train the speech enhancement model and then send the trained speech enhancement model to the display device 200 and / or the terminal device 600. The display device 200 or the terminal device 600 can process audio data using the speech enhancement model.

[0120] In some scenarios, a speech enhancement model is deployed in the server 500. The terminal device 600 or display device 200 sends an audio processing task to the server 500. The audio processing task includes an audio signal to be processed. The server 500 performs audio noise reduction on the audio signal to be processed and sends the noise-reduced audio signal to the terminal device 600 or display device 200. The terminal device 600 or display device 200 then plays the noise-reduced audio signal through a speaker.

[0121] like Figure 5B As shown, the server 500 is connected to the display device 200 as an example. For example, the server 500 can execute a training method for a speech enhancement model. In some scenarios, the server 500 can train the speech enhancement model and then send the trained speech enhancement model to the display device 200. The display device 200 can process audio data through the speech enhancement model. In some scenarios, a speech enhancement model is deployed in the server 500. The display device 200 sends an audio processing task to the server 500. The audio processing task includes an audio signal to be processed. The server 500 performs audio noise reduction on the audio signal to be processed and sends the noise-reduced audio signal to the display device 200. The display device 200 thus plays the noise-reduced audio signal through a speaker.

[0122] The following is an exemplary description of the structure of an electronic device for executing the training method of the speech enhancement model. Figure 6As shown, the electronic device may include a processor 510 and a communication interface 520. The electronic device may also include a memory 530. Of course, the electronic device may also include other components. Figure 6 Not shown in the figure.

[0123] The communication interface 520 is used to communicate with the display device, to receive the audio processing task sent by the display device, or to send the audio signal after noise reduction to the electronic device.

[0124] In the embodiments of the present application, the processor 510 may be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present application may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0125] The processor 510 is the control center of the electronic device. It connects the various parts of the electronic device using various interfaces and routes. It performs various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 530 and calling data stored in the memory 530. Optionally, the processor 510 may include one or more processing units. The processor 510 may be, for example, a processor, a microprocessor, a controller, or other control component. For example, it may be a general-purpose central processing unit (CPU), a general-purpose processor, a digital signal processing (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof.

[0126] The memory 530 can be used to store software programs and modules. The processor 510 executes various functional applications and data processing by running the software programs and modules stored in the memory 530. The memory 530 may mainly include a program storage area and a data storage area. The program storage area may store an operating system, at least one application required for a function, etc.; the data storage area may store data created based on business processing, etc. The memory 530 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs, and modules. The memory 530 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (RAM), a static random access memory (SRAM), a programmable read-only memory (PROM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic storage device, a magnetic disk, an optical disk, and the like. The memory 530 is any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. The memory 530 in the embodiment of the present application may also be a circuit or any other device capable of performing a storage function, for storing program instructions and / or data.

[0127] It should be noted that the above Figure 1-6 The structure shown is only an example and is not limited in the embodiment of the present invention.

[0128] Currently, speech enhancement technology often requires better suppression of echoes and noise in audio signals during phone calls. In speech recognition scenarios, it's often necessary to suppress noise while also reducing distortion. Switching audio signal processing for different scenarios typically requires switching between different speech enhancement models. This means applying processed audio signals to a wider range of scenarios requires building more speech enhancement models, significantly increasing the workload. Therefore, finding a single speech enhancement model that can both reduce distortion and suppress noise is an urgent need.

[0129] In view of this, the present application provides a training method for a speech enhancement model, in which the audio sample data in the training sample set and the audio masking result of each audio sample data are input into the speech enhancement model. Feature extraction is performed on the audio sample data through an input layer and multiple hidden layers, the extracted audio features are input into multiple output layers respectively, and the loss value between the audio noise reduction result and the audio masking result of each output layer is calculated. The network parameters of the speech enhancement model are adjusted according to the weighted results of the loss values ​​of the multiple output layers to obtain a trained speech enhancement model. By applying the trained speech enhancement model for audio processing, a variety of different modes of processing of audio signals can be achieved by switching the network parameters of the output layer, so that the audio signals processed by different output layers can be applied to different scenarios without the need to construct multiple speech enhancement models.

[0130] See Figure 7 This is one of the schematic diagrams of the speech enhancement model provided in the embodiment of the present application. In the embodiment of the present application, the speech enhancement model is described by taking a neural network as an example. Figure 7 The illustrated speech enhancement model 700 includes an input layer 701, a hidden layer 702, and an output layer 703. The input layer 701 is used to preprocess the audio samples input to the speech enhancement model 700. For example, the input layer 701 may perform preprocessing operations such as data standardization and normalization on the audio samples input to the speech enhancement model 700. The hidden layer 702 is used to extract features from the preprocessed audio sample data. The output layer 703 is used to output the audio noise reduction result after denoising the extracted audio features.

[0131] in, Figure 7 The hidden layer 702 in may include N hidden layers, where N is an integer greater than 0. Figure 8 This is a schematic diagram of the hidden layer structure of the speech enhancement model provided in the embodiment of the present application. Figure 8 As shown, the hidden layer 800 may include hidden layer 1, hidden layer 2, hidden layer 3...hidden layer N.

[0132] Figure 7 The output layer 703 in may include M output layers, where M is an integer greater than 1. Figure 9 This is a schematic diagram of the output layer structure of the speech enhancement model provided in the embodiment of this application. Figure 9 As shown, the output layer 900 may include output layer 1, output layer 2, output layer 3, ..., output layer M. The hidden layer inputs the output audio features into output layer 1, output layer 2, output layer 3, ..., output layer M in the output layer 900, respectively, to obtain audio noise reduction results 1, audio noise reduction results 2, audio noise reduction results 3, ..., audio noise reduction results M. The audio noise reduction results output by different output layers have different noise reduction levels.

[0133] It should be understood that Figure 7-Figure 9 The speech enhancement model shown can be implemented using one or more neural networks such as fully connected neural networks, deep neural networks (DNN), convolutional neural networks (CNN), and attention mechanisms, and this application does not limit this.

[0134] based on Figure 7 The speech enhancement model structure shown in Figure 10 This is an exemplary flow chart of a method for training a speech enhancement model provided in an embodiment of the present application. The method can be implemented by an execution device, such as Figure 1 The server 400 shown is implemented. The training method of the speech enhancement model includes the following process:

[0135] S1001: Obtain a training sample set.

[0136] The training sample set includes multiple audio samples and audio masking results corresponding to the multiple audio samples. The audio masking result corresponding to each audio sample can be pre-calibrated, and the value of the audio masking result can be a number greater than or equal to 0 and less than or equal to 1. The smaller the value of the audio masking result, the better the noise reduction effect of the audio when the trained speech enhancement model processes the audio. The larger the value of the audio masking result, the more complete the speech signal in the audio is retained when the trained speech enhancement model processes the audio.

[0137] In one possible implementation, the training sample set may be a continuous audio segment comprising T frames. Each frame of the audio segment may be an audio sample, T is an integer greater than 0, and t is an integer greater than 0 and less than T. For example, assuming the training sample set is a continuous audio segment comprising 20 frames, the first frame of the audio segment is an audio sample, the second frame is an audio sample, and so on, the training sample set may include 20 audio samples and the audio masking result corresponding to each audio sample.

[0138] In another possible implementation, the training sample set may also include one or more discontinuous audio segments. Each audio segment may be an audio sample. For example, assuming that the training sample set includes 30 discontinuous audio segments, the first audio segment is an audio sample, the second audio segment is an audio sample, and so on. The training sample set may include 30 audio samples and the audio masking result corresponding to each audio sample.

[0139] It should be understood that the audio samples in the audio sample set can be one or more of the following audio samples: time domain waveform, filter bank (Fbank), mel-frequency cepstrum coefficient (MFCC), amplitude spectrum, power spectrum, complex spectrum, phase spectrum, etc. This application is not limited to this.

[0140] S1002: Preprocess the first audio sample in the training sample set through the input layer of the speech enhancement model to obtain first audio sample data.

[0141] The first audio sample in the training sample set obtained in S1001 and the audio masking result corresponding to the first audio sample are input as follows: Figure 7 The input layer of the speech enhancement model shown is preprocessed to obtain first audio sample data.

[0142] S1003: Extract audio features of the first audio sample data through N hidden layers of the speech enhancement model.

[0143] The first audio sample data acquired in S1002 is input into N hidden layers to perform feature extraction on the first audio sample data to obtain audio features of the first audio sample data.

[0144] Optionally, when Figure 7 The speech enhancement model shown applies different neural networks. Figure 8 The structure of the hidden layer shown may be different. For example, when the speech enhancement model is a speech enhancement model based on a convolutional neural network, the hidden layer may include a convolutional layer, a pooling layer, and a fully connected layer. In one example, see Figure 11 This is a schematic diagram of the structure of a speech enhancement model based on a convolutional neural network provided in an embodiment of the present application. Figure 11 As shown, the speech enhancement model 1100 includes an input layer 1101, a hidden layer 1102 and an output layer 1103. The hidden layer 1102 includes a pooling layer 1, a convolutional layer, a pooling layer 2, a fully connected layer 1 and a fully connected layer 2. It should be understood that Figure 11 The structure of the convolutional neural network shown is only exemplary. The number of convolutional layers, pooling layers and fully connected layers in the convolutional neural network can be one or more, and this application does not limit this.

[0145] S1004: Input the audio features into M output layers of the speech enhancement model respectively to obtain M audio noise reduction results.

[0146] The audio features extracted in S1003 are respectively input into the M output layers of the speech enhancement model. Each output layer can output an audio noise reduction result respectively through a mask-based speech enhancement algorithm, so that M audio noise reduction results can be obtained. Wherein, M is an integer greater than 1, and different output layer network parameters are different. It should be understood that the mask-based speech enhancement algorithm can be any one of the mask-based speech enhancement algorithms such as IRM, ideal amplitude mask (IAM) and phase sensitive mask (PSM), and this application is not limited to this.

[0147] Specifically, the M output layers of the speech enhancement model may include a first output layer and a second output layer. After the audio features extracted in S1003 are respectively input into the first output layer and the second output layer of the speech enhancement model, a first audio noise reduction result can be obtained in the first output layer according to the mask-based speech enhancement algorithm. A second audio noise reduction result can be obtained in the second output layer according to the mask-based speech enhancement algorithm. The network parameters of the first output layer and the second output layer are different.

[0148] For example, assume that the speech enhancement model consists of 1 input layer, 1 hidden layer, and 2 output layers. Figure 12 This is one of the schematic diagrams of the speech enhancement model structure provided in the embodiment of the present application. Figure 12 The speech enhancement model 1200 shown in FIG. 1 includes an input layer 1201, a hidden layer 1202, a first output layer 1203, and a second output layer 1204. For example, the first output layer 1203 and the second output layer 1204 of the speech enhancement model 1200 are fully connected layers. The network parameters of the first output layer 1203 and the second output layer 1204 may include weights w and biases b. The network parameters of the first output layer 1203 are w1 and b1. The network parameters of the second output layer 1204 are w2 and b2. Inputting an audio sample into the speech enhancement model 1200, through the input layer 1201, hidden layer 1202, and first output layer 1203, yields a first audio denoising result. Inputting an audio sample into the speech enhancement model 1200, through the input layer 1201, hidden layer 1202, and second output layer 1204, yields a second audio denoising result.

[0149] S1005 , determining the losses between the M audio denoising results and the audio masking results using the loss functions corresponding to the M output layers respectively, to obtain M loss values.

[0150] See also Figure 13 This is one of the loss calculation diagrams provided in the embodiments of the present application. Figure 13The speech enhancement model shown includes output layer A and output layer B. The loss function corresponding to output layer A is loss function A, and the loss function corresponding to output layer B is loss function B. Audio samples pass through the input layer, hidden layer, and output layer A to output audio denoising result A, and audio samples pass through the input layer, hidden layer, and output layer B to output audio denoising result B. Loss function A calculates the loss between audio denoising result A and the audio masking result to obtain loss value LosA. Loss function B calculates the loss between audio denoising result B and the audio masking result to obtain loss value LosB.

[0151] See also Figure 14 This is one of the loss calculation diagrams provided in the embodiments of the present application. Figure 14 The speech enhancement model shown includes output layer 1, output layer 2, ..., output layer M, for a total of M output layers. The loss function corresponding to output layer 1 is loss function 1, and the loss function corresponding to output layer 2 is loss function 2. The audio sample passes through the input layer, hidden layer, and output layer 1 to output audio denoising result 1, and the audio sample passes through the input layer, hidden layer, and output layer 2 to output audio denoising result 2. Loss function 1 calculates the loss between audio denoising result 1 and the audio masking result, resulting in loss value Loss1. Loss function 2 calculates the loss between audio denoising result 2 and the audio masking result, resulting in loss value Loss2. Similarly, Loss1 - LossM are obtained, for a total of M loss values.

[0152] In one possible implementation, a first loss value between a first audio denoising result output by a first output layer and an audio masking result corresponding to the first audio sample can be determined using a first loss function. The first output layer is any one of the M output layers, and the first loss function is a loss function corresponding to the first output layer.

[0153] Optionally, the first loss function may be determined according to the first target noise reduction result and the first audio noise reduction result. The first loss function may be as shown in formula (1).

[0154]

[0155] As shown in the above formula, Loss1 represents the first loss value, n represents the number of frequency points of the audio feature, W1 represents the weight corresponding to the first loss function, and y pred1,f The first audio noise reduction result of the fth frequency point of the audio feature, y target1,f The target noise reduction result of the fth frequency point representing the audio feature can be shown as formula (2), y true,f Indicates the audio masking result of the f-th frequency point of the audio feature.

[0156] y target1,f =1-ytrue,f Formula (2)

[0157] As shown in the above formula, y true,f is the audio masking result of the f-th frequency point of the audio feature.

[0158] The target noise reduction result of the fth frequency point of the audio feature in formula (1) can be expressed by formula (2), and the first loss function can be expressed as shown in formula (3).

[0159]

[0160] Optionally, the weight corresponding to the first loss function may be independent of the audio masking result of the audio sample. For example, the weight corresponding to the first loss function may be set to a constant, such as W1 may be 4, 5, or 6, etc., which is not limited in this application.

[0161] Alternatively, the weight corresponding to the first loss function may also be related to the audio masking result of the audio sample.

[0162] In one example, the smaller the audio masking result is, the greater the weight corresponding to the first loss function is, and the weight corresponding to the first loss function can satisfy formula (4).

[0163]

[0164] As shown in the above formula, W1 represents the weight corresponding to the first loss function. α can be a preset constant, such as 3, 4, or 5. The larger the value of α, the greater the weight corresponding to the first loss function. This application does not limit this.

[0165] In another example, the weight corresponding to the first loss function may also be as shown in formula (5).

[0166] W1=γ-y true,f Formula (5)

[0167] As shown in the above formula, γ can be a preset constant, such as γ can take a value of 2, 3 or 4, etc., which is not limited in this application.

[0168] Based on the above scheme, since the first loss function shown in formula (3) imposes a greater penalty on residual noise in the audio signal than on excessive noise elimination, the noise reduction effect of the speech enhancement model can be enhanced.

[0169] Optionally, when the training sample set obtained in S1001 is different, the formula of the first loss function may be different. For example, when the training sample set obtained in S1001 includes multiple discontinuous audio segments, the first loss function may be as shown in formula (1) or formula (3), which will not be repeated here.

[0170] In one possible scenario, when the training sample set obtained in S1001 may be a continuous audio segment comprising T frames, the first audio sample may be the t-th frame of the audio segment, where T is an integer greater than 0 and t is an integer greater than 0 and less than T.

[0171] Then the first loss function is the loss function of the t-th frame audio, which can be shown as formula (6).

[0172]

[0173] As shown in the above formula, Loss1 represents the first loss value, n represents the number of frequency points of the audio feature, W1 represents the weight corresponding to the first loss function, and y pred1，(t,f) The audio denoising result corresponding to the first output layer of the fth frequency point of the audio feature of this frame, y true,(t,f) Indicates the audio masking result of the f-th frequency point.

[0174] In one possible implementation, the M output layers may further include a second output layer. A second loss value is determined between an audio noise reduction result of the second output layer and an audio masking result corresponding to the second audio sample using a second loss function. The second output layer is any one of the M output layers except the first output layer, and the second loss function is a loss function corresponding to the second output layer.

[0175] The second loss function is shown in formula (7).

[0176]

[0177] As shown in the above formula, Loss2 represents the second loss value, n represents the frequency of the audio feature, W2 represents the weight corresponding to the second loss function, and y pred2,f The audio noise reduction result corresponding to the second output layer of the fth frequency point of the audio feature, y target2,f represents the second target noise reduction result of the fth frequency point, satisfying formula (8).

[0178] y target2,f =y true,f Formula (8)

[0179] The second target denoising result in formula (7) is expressed by formula (8), and the second loss function can be shown as formula (9).

[0180]

[0181] In one possible implementation, the weight corresponding to the second loss function can be set by referring to the weight corresponding to the first loss function mentioned above, which will not be repeated here.

[0182] Optionally, the weight corresponding to the second loss function may be the same as the weight corresponding to the first loss function. For example, the weight corresponding to the first loss function and the weight corresponding to the second loss function may both be 4.

[0183] Alternatively, the weight corresponding to the second loss function may also be different from the weight corresponding to the first loss function. In one example, the formula satisfied by the weight corresponding to the first loss function is the same as the formula satisfied by the weight corresponding to the second loss function, but the preset constant in the formula satisfied by the weight corresponding to the first loss function is different from the preset constant in the formula satisfied by the weight corresponding to the second loss function. For example, when the weight corresponding to the first loss function and the weight corresponding to the second loss function both satisfy formula (4), α in the weight corresponding to the first loss function can take a value of 4, and α in the weight corresponding to the second loss function can take a value of 5. In another example, the formula satisfied by the weight corresponding to the first loss function and the formula satisfied by the weight corresponding to the second loss function may be different. For example, the weight corresponding to the first loss function satisfies formula (4), and the weight corresponding to the second loss function satisfies formula (5).

[0184] Based on the above scheme, since the penalty for residual audio noise in the second loss function shown in formula (9) is less than the penalty for over-eliminating the noise in the audio signal, the audio can be retained more completely, so that the audio processed by the speech enhancement model is not distorted.

[0185] See also Figure 15 This is a graph of the second loss function provided in an embodiment of the present application. Figure 15 Assuming the audio masking result is 0.2, it can be seen that the second loss value is minimum when the audio noise reduction result is 0.2, the second loss value is set to Loss2 when the audio noise reduction result is 0.1, and the second loss value is set to Loss1 when the audio noise reduction result is 0.3, and Loss1 is less than Loss2. In other words, when the audio masking results are the same and the difference between the audio noise reduction result and the audio masking result is the same, the first loss value when the audio noise reduction result is greater than the audio masking result is less than the first loss value when the audio noise reduction result is less than the audio masking result.

[0186] Optionally, when the training sample set obtained in S1001 is different, the formula of the second loss function may be different. For example, when the training sample set obtained in S1001 includes multiple discontinuous audio segments, the second loss function may be as shown in Formula (7) or Formula (9), which will not be repeated here.

[0187] In one possible scenario, when the training sample set obtained in S1001 may be a continuous audio segment including T frames, the second audio sample may be the tth frame of the audio segment. Then, the second loss function is the loss function of the tth frame of audio, which may be shown in formula (10).

[0188]

[0189] As shown in the above formula, Loss2 represents the second loss value, n represents the frequency of the audio feature, W2 represents the weight corresponding to the second loss function, and y pred2，(t,f) The second audio denoising result of the fth frequency point of the audio feature of this frame, y true,(t,f) Indicates the audio masking result of the f-th frequency point.

[0190] In one possible implementation, the loss functions corresponding to the M output layers may be different. Specifically, different loss functions may be set for different output layers by adjusting the logarithmic function and / or weighting function in the loss function shown in Formula (3) or Formula (9). For example, see Figure 16 This is one of the loss calculation diagrams provided in the embodiments of the present application. Figure 16 The output layers of the speech enhancement model in include output layer 1, output layer 2, and output layer 3. The loss function corresponding to output layer 1 satisfies formula (3), where W1 satisfies formula (4) and α is 4. Then, the loss value Loss1 obtained by the loss function corresponding to output layer 1 satisfies formula (11).

[0191]

[0192] The loss function corresponding to the output layer 2 satisfies formula (9). Where W2 satisfies formula (4) and α is 4, then the loss value Loss2 obtained by the loss function corresponding to the output layer 2 can satisfy formula (12).

[0193]

[0194] The loss function corresponding to the output layer 3 can satisfy formula (9) where W2 satisfies formula (5) and γ is 2.

[0195] Then the loss value Loss3 obtained by the loss function corresponding to the output layer 3 can satisfy formula (13).

[0196]

[0197] S1006, adjusting the network parameters of the input layer, N hidden layers, and M output layers according to the weighted results of the M loss values ​​to obtain a trained speech enhancement model.

[0198] The total loss value of the speech enhancement model can be determined according to the weighted result of the M loss values. The speech enhancement model is trained according to the total loss value of the speech enhancement model, and the network parameters of the input layer, N hidden layers, and M output layers are adjusted to obtain the trained speech enhancement model.

[0199] The total loss value of the speech enhancement model can satisfy formula (14).

[0200] Loss=a1Loss1+a2Loss2+……+a m Lossm formula(14)

[0201] As shown in the above formula, Loss represents the total loss value of the speech enhancement model. a1~a m Represents the weighted values ​​corresponding to the M loss values, and a1~a m The sum of Loss1 to Lossm is 1. Loss1 to Lossm represent the M loss values ​​obtained in S1005.

[0202] Optionally, the speech enhancement model may terminate training when the total loss value of the model no longer converges, obtaining a trained speech enhancement model. Alternatively, the speech enhancement model may terminate training when the number of iterations reaches a certain number, obtaining a trained speech enhancement model. For example, the speech enhancement model may terminate training when the number of iterations reaches 50. This application is not limited to this.

[0203] See Figure 17 , is an exemplary flow chart of an audio processing method based on a speech enhancement model provided in an embodiment of the present application. The process can be executed by an execution device, which can be Figure 3 The display device 200 shown in FIG. 2 may specifically perform audio processing via the controller 250 in the display device 200. Alternatively, the execution device may be Figure 6 The electronic device shown in FIG. 5 can specifically perform audio processing by the processor 510 in the electronic device. The speech enhancement model applied during audio processing can be Figure 1 The server 400 shown in FIG. Figure 10 The speech enhancement model training method shown is used to train the speech enhancement model, and the trained speech enhancement model is provided to the execution device. Specifically, it includes:

[0204] S1701: The execution device obtains an audio signal to be processed.

[0205] The audio signal to be processed may be obtained through a network, or may be recorded by an execution device, which is not limited in this application.

[0206] S1702: The execution device determines that the noise reduction mode for performing noise reduction on the audio signal to be processed is the first mode.

[0207] The noise reduction mode for performing noise reduction on the audio signal to be processed may include M noise reduction modes. Different noise reduction modes have different noise reduction degrees on the audio signal to be processed.

[0208] Optionally, the execution device may determine a noise reduction mode for performing noise reduction on the audio signal to be processed according to the usage scenario. Figure 18 This is one of the schematic diagrams of the method for the execution device to determine the noise reduction mode provided in the embodiment of the present application. Figure 18 As shown, taking a mobile phone as an example, when the executing device detects that the usage scenario is a call, the noise reduction mode for the processed audio signal can be determined to be the first mode. When the executing device detects that the usage scenario is a recording scenario, the noise reduction mode for the processed audio signal can be determined to be the second mode, and so on.

[0209] Alternatively, the execution device may determine the noise reduction mode for performing noise reduction on the audio signal to be processed according to the received instruction. For example, the execution device may determine that the noise reduction mode for performing noise reduction on the audio signal to be processed is the first mode in response to the instruction of the user selecting the noise reduction mode as the first mode. Figure 19 This is one of the schematic diagrams of the method for the execution device to determine the noise reduction mode provided in the embodiment of the present application. Figure 19 As shown, taking the execution device as a mobile phone with a touch screen as an example, the touch screen can display a display interface for selecting a noise reduction mode, which includes a first noise reduction mode and a second noise reduction mode. When the user clicks the first noise reduction mode, the execution device determines that the noise reduction mode for the processed audio signal is the first mode in response to the user's click. When the user clicks the second noise reduction mode, the execution device determines that the noise reduction mode for the processed audio signal is the second mode in response to the user's click.

[0210] It should be understood that Figure 18 and Figure 19 The method for the execution device to determine the noise reduction mode is only exemplary and is not limited in this application.

[0211] S1703: The execution device obtains audio features of the audio signal to be processed through the input layer and at least one hidden layer of the speech enhancement model.

[0212] Among them, the speech enhancement model can be as follows Figure 10 The speech enhancement model is trained by the speech enhancement model training method shown in FIG. The structure of the speech enhancement model can be as follows: Figure 7-Figure 9 As shown, no further details are given here.

[0213] S1704: The execution device inputs the audio features to the first output layer of the M output layers corresponding to the first mode to obtain an audio noise reduction result of the audio signal to be processed.

[0214] Among them, the M noise reduction modes can correspond one-to-one to the M output layers of the speech enhancement model, and the network parameters of each output layer are different.

[0215] In one possible implementation, the execution device can switch the noise reduction mode by switching the network parameters of the output layer, thereby obtaining different audio noise reduction results. For example, see Figure 20 This is one of the noise reduction mode switching diagrams provided in an embodiment of the present application. Figure 20 In the example, the noise reduction mode of the execution device includes the first mode and the second mode. The first output layer corresponds to the first mode, and the second output layer corresponds to the second mode. The network parameters of the first output layer are w1 and b1, and the network parameters of the second output layer are w2 and b2. Assuming that the current noise reduction mode of the execution device is the first mode, when the execution device detects that the current usage scenario is the usage scenario corresponding to the second mode, the execution device can switch the noise reduction mode of the execution device from the first mode to the second mode by switching the network parameters of the speech enhancement model output layer from w1 and b1 to w2 and b2.

[0216] In another possible implementation, the execution device can also switch the noise reduction mode by switching the network parameters of the last k layers of the speech enhancement model, thereby obtaining different audio noise reduction results. The last k layers of the speech enhancement model may include an output layer and a hidden layer, and k is an integer greater than 1. Figure 21 This is one of the noise reduction mode switching diagrams provided in the embodiment of the present application. Figure 21 As shown, assuming that the speech enhancement model includes 5 hidden layers, the execution device switches the noise reduction mode by switching the network parameters of the last 3 layers of the speech enhancement model. When the noise reduction mode is the first mode, the audio signal to be processed input to the speech enhancement model can obtain the audio noise reduction result of the audio signal to be processed through the input layer, hidden layer 1, hidden layer 2, hidden layer 3, hidden layer 4, hidden layer 5 and output layer 1. When the noise reduction mode is the second mode, the audio signal to be processed input to the speech enhancement model can obtain the audio noise reduction result of the audio signal to be processed through the input layer, hidden layer 1, hidden layer 2, hidden layer 3, hidden layer 6, hidden layer 7 and output layer 2. Then the current noise reduction mode of the execution device is the first mode. When the execution device detects that the current usage scenario is the usage scenario corresponding to the second mode, it can switch the hidden layer 4, hidden layer 5 and output layer 1 to the hidden layer 6, hidden layer 7 and output layer 2 respectively by switching the network parameters, thereby switching the noise reduction mode of the execution device to the second mode.

[0217] Based on the above solution, by switching the network parameters of the output layer or the network parameters of the last k layers of the speech enhancement model, it is possible to switch between multiple different noise reduction modes through one speech enhancement model.

[0218] Based on the same technical concept, an embodiment of the present application further provides a computer-readable storage medium on which computer instructions are stored. When the computer instructions are executed on a computer, the computer executes the steps of any of the above methods.

[0219] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be implemented using hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0220] Although the specific embodiments of the present application have been described above, it should be understood by those skilled in the art that these are merely examples, and the scope of protection of the present application is defined by the appended claims. Those skilled in the art may make various changes or modifications to these embodiments without departing from the principles and essence of the present application, but these changes and modifications all fall within the scope of protection of the present application. Although the preferred embodiments of the present application have been described, once those skilled in the art become aware of the basic creative concepts, additional changes and modifications may be made to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0221] Obviously, those skilled in the art may make various changes and modifications to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include these modifications and variations.

Claims

1. A method for training a speech enhancement model, characterized in that: include: Acquire a training sample set, where the training sample set includes a plurality of audio samples and audio masking results corresponding to the plurality of audio samples respectively; The value of the audio masking result is a number greater than or equal to 0 and less than or equal to 1; the smaller the value of the audio masking result, the better the noise reduction effect of the audio when the trained speech enhancement model processes the audio; the larger the value of the audio masking result, the more complete the speech signal in the audio is retained when the trained speech enhancement model processes the audio; Preprocessing the first audio sample in the training sample set through the input layer of the speech enhancement model to obtain first audio sample data; Extracting audio features of the first audio sample data through N hidden layers of the speech enhancement model, where N is a positive integer; Inputting the audio features into M output layers of the speech enhancement model respectively to obtain M audio noise reduction results, where M is an integer greater than 1; Determining the losses between the M audio noise reduction results and the audio masking results respectively by using the loss functions corresponding to the M output layers to obtain M loss values; determining the losses between the M audio noise reduction results and the audio masking results respectively by using the loss functions corresponding to the M output layers to obtain M loss values, including: Determine a first loss value between the audio noise reduction result corresponding to the first output layer and the audio masking result by using a first loss function; the first output layer is any output layer among the M output layers; the first loss function satisfies the following formula: Among them, Loss1 is the first loss value, n is the frequency point number of the audio feature, W1 is the weight corresponding to the first loss function, y pred1,f is the audio noise reduction result corresponding to the first output layer of the fth frequency point of the audio feature, y true,f is the audio masking result of the f-th frequency point; The weight corresponding to the first loss function is related to the audio masking result; the weight satisfies the following formula: Among them, α is a preset constant; The network parameters of the input layer, the N hidden layers, and the M output layers are adjusted according to the weighted results of the M loss values ​​to obtain the trained speech enhancement model.

2. The method according to claim 1, characterized in that The M output layers include a first output layer and a second output layer; and determining the losses between the M audio noise reduction results and the audio masking results using loss functions corresponding to the M output layers to obtain M loss values ​​includes: Determining a second loss value between the audio noise reduction result corresponding to a second output layer and the audio masking result by a second loss function; the second output layer is any output layer among the M output layers that is different from the first output layer; The second loss function satisfies the following formula: Among them, Loss2 is the second loss value, n is the frequency point number of the audio feature, W2 is the weight corresponding to the second loss function, y pred2,f is the audio noise reduction result corresponding to the second output layer of the fth frequency point of the audio feature, y true,f is the audio masking result of the f-th frequency point.

3. The method according to claim 2, characterized in that The weight corresponding to the second loss function is related to the audio masking result; The weight satisfies the following formula: Where W2 is the weight corresponding to the second loss function, β is a preset constant, and y true,f is the audio masking result of the f-th frequency point.

4. An audio processing method based on a speech enhancement model, characterized in that: The method comprises: Obtaining the audio signal to be processed; Determining a first noise reduction mode for performing noise reduction on the audio signal to be processed; The audio features of the audio signal to be processed are obtained through an input layer and at least one hidden layer of the speech enhancement model; the speech enhancement model further includes M output layers, where M is an integer greater than 1; the noise reduction mode includes M modes including the first mode; the M modes correspond one-to-one to the M output layers; the M modes have different noise reduction degrees, and the network parameters corresponding to the M output layers are different; the speech enhancement model is trained using the training method according to any one of claims 1 to 3; The audio features are input into a first output layer of the M output layers corresponding to the first mode to obtain an audio noise reduction result of the audio signal to be processed.

5. A chip system, characterized in that: It comprises a memory and a processor; the processor and the memory are coupled; wherein the memory includes program instructions, and when the program instructions are executed by the processor, the chip system executes the method according to any one of claims 1 to 4.

6. An electronic device, characterized in that: include: A communication interface for receiving a training sample set; A processor, configured to obtain the training sample set from the communication interface to execute the method according to any one of claims 1 to 3.

7. A terminal device, characterized in that: include: A processor, configured to execute the method according to claim 4; The speaker is used to play the audio signal after the audio noise is reduced.

8. A computer-readable storage medium, characterized in that include: The computer-readable storage medium stores computer instructions, which, when executed on a computer, enable the computer to perform the method according to any one of claims 1 to 4.

Images