Speech enhancement method and related device

Abstract

The invention discloses a speech enhancement method and a related device, and relates to the technical field of artificial intelligence, and the method comprises the steps: carrying out the enhancement of target speech data based on a speech enhancement model, and obtaining enhanced speech data, the speech enhancement model is constructed based on a U-shaped network added with a joint attention mechanism module composed of a self-attention module and a global space attention module, and on the basis of the feature modeling capability of the U-shaped network, on one hand, the self-attention module pays attention to the correlation of context information; and on the other hand, global features are fully modeled through a global space attention module, and intra-class compactness is improved, so that feature representation is optimized, the speech enhancement effect of the speech enhancement model is improved, and the speech quality of a speech enhancement result is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a speech enhancement method and related devices. Background Art

[0002] In interactive systems, voice interaction is one of the common interactive methods, and the quality of voice directly affects the fluency and comfort of using interactive systems. Taking voice interaction in a banking system as an example, the financial equipment installed at the front end of the banking system is equipped with a built-in microphone. The microphone collects user voices and sends them to the back-end voice recognition system, so that the back-end voice recognition system can recognize the user voice and execute the corresponding business, thereby achieving the purpose of supporting customers to perform voice operations. However, in a real environment, in addition to collecting the customer's voice, the microphone will also collect other noises, such as the rotation sound of the fan in the equipment, the current sound generated by the microphone itself, and the noise voice in the external environment. In addition, the voice signal-to-noise ratio collected by the microphone is relatively low, and it contains less effective voice information. Yes, at present, the user voice collected by the banking system through the microphone has the disadvantage of low quality, resulting in a low recognition accuracy rate of the back-end voice recognition system for the user voice, and ultimately resulting in low availability of the voice input function on the financial equipment end. Summary of the invention

[0003] In view of the above problems, the present application provides a speech enhancement method and related devices to achieve the purpose of improving speech quality. The specific scheme is as follows:

[0004] The first aspect of the present application provides a speech enhancement method, comprising:

[0005] Acquire target speech data, wherein the target speech data includes frequency domain amplitude spectrum features of the speech signal to be enhanced;

[0006] The target speech data is input into a pre-trained speech enhancement model to obtain enhanced speech data output by the speech enhancement model; the speech enhancement model includes a U-type network and a joint attention mechanism module, the joint attention mechanism module includes a self-attention module and a global space attention module, the U-type network includes a U-type network encoder, a U-type network decoder and an output layer, the output end of the U-type network encoder is connected to the input end of the joint attention mechanism module, the output end of the joint attention mechanism module is connected to the input end of the U-type network decoder, and the output end of the U-type network decoder is connected to the output layer.

[0007] Optionally, acquiring the target voice data includes:

[0008] The speech signal to be enhanced is acquired, and the speech signal to be enhanced is framed, sampled, and short-time Fourier transformed to obtain frequency domain amplitude spectrum features of the speech signal to be enhanced.

[0009] Optionally, the output end of the U-type network encoder is connected to the input end of the joint attention mechanism module, including: the output end of the U-type network encoder is connected to the input end of the self-attention module and the input end of the global space attention module respectively, for inputting the output data input of the U-type network encoder to the self-attention module and the global space attention module respectively;

[0010] The output end of the joint attention mechanism module is connected to the input end of the U-shaped network decoder including:

[0011] The output end of the self-attention module, the output end of the global spatial attention module, and the output end of the U-type network encoder are connected to the input end of the U-type network decoder through a feature fusion layer. The feature fusion layer is used to fuse the output data of the self-attention module, the output data of the global spatial attention module, and the output data of the U-type network encoder, and then input them into the U-type network decoder.

[0012] Optionally, the speech enhancement model further includes a multi-scale skip connection, wherein the multi-scale skip connection includes a plurality of series blocks and a plurality of sampling layers;

[0013] The input of the first concatenation block is connected to the output of the first decoder block and connected to the output of the joint attention mechanism module through the sampling layer;

[0014] The input end of the nth series block is connected to the output end of the nth decoder block, and is connected to the output end of the n-1th series block through the sampling layer; the output end of the nth series block is connected to the input end of the n+1th decoder block, and is connected to the input end of the n+1th series block through the sampling layer, where n is greater than 1 and less than N, and N is the number of decoder blocks;

[0015] The input end of the Nth series block is connected to the output end of the Nth decoder block, and is connected to the output end of the N-1th series block through the sampling layer; the output end of the Nth series block is the output end of the U-type network decoder;

[0016] The series connection block is used to perform series connection and splicing on the input data, and the sampling layer is used to perform bilinear interpolation up-sampling on the input data.

[0017] Optionally, the speech enhancement method further includes:

[0018] Acquire original speech data, wherein the original speech data includes a noisy speech signal and a clean speech signal;

[0019] Performing frame division, sampling, and short-time Fourier transformation on each speech signal in the original speech data to obtain frequency domain amplitude spectrum characteristics of each speech signal in the original speech data;

[0020] Acquire training data and test data, wherein the training data includes frequency domain amplitude spectrum features of multiple noisy speech signals and frequency domain amplitude spectrum features of corresponding clean speech signals, and the test data includes frequency domain amplitude spectrum features of multiple noisy speech signals;

[0021] Training an original speech enhancement model based on the training data to obtain a candidate speech enhancement model;

[0022] The candidate speech enhancement model is tested based on the test data to obtain a trained speech enhancement model.

[0023] A second aspect of the present application provides a speech enhancement device, comprising:

[0024] A data preprocessing unit, used to obtain target speech data, wherein the target speech data includes frequency domain amplitude spectrum features of the speech signal to be enhanced;

[0025] A speech enhancement unit is used to input the target speech data into a pre-trained speech enhancement model to obtain enhanced speech data output by the speech enhancement model; the speech enhancement model includes a U-type network and a joint attention mechanism module, the joint attention mechanism module includes a self-attention module and a global space attention module, the U-type network includes a U-type network encoder, a U-type network decoder and an output layer, the output end of the U-type network encoder is connected to the input end of the joint attention mechanism module, the output end of the joint attention mechanism module is connected to the input end of the U-type network decoder, and the output end of the U-type network decoder is connected to the output layer.

[0026] A third aspect of the present application provides a speech enhancement system, comprising: a speech enhancement device and a pre-trained speech enhancement model;

[0027] The speech enhancement device is used to obtain target speech data and input the target speech data into a pre-trained speech enhancement model; the target speech data includes frequency domain amplitude spectrum features of the speech signal to be enhanced;

[0028] The speech enhancement model is used to output enhanced speech data; the speech enhancement model includes a U-type network and a joint attention mechanism module, the joint attention mechanism module includes a self-attention module and a global space attention module, the U-type network includes a U-type network encoder, a U-type network decoder and an output layer, the output end of the U-type network encoder is connected to the input end of the joint attention mechanism module, the output end of the joint attention mechanism module is connected to the input end of the U-type network decoder, and the output end of the U-type network decoder is connected to the output layer.

[0029] The fourth aspect of the present application provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the speech enhancement method of the above-mentioned first aspect or any implementation manner of the first aspect.

[0030] A fifth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0031] The memory is used to store computer programs;

[0032] The processor is used to execute the computer program so that the electronic device can implement the speech enhancement method of the above-mentioned first aspect or any implementation manner of the first aspect.

[0033] A fifth aspect of the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the speech enhancement method of the above-mentioned first aspect or any implementation of the first aspect.

[0034] By means of the above technical scheme, the present application provides a speech enhancement method and related devices, obtains target speech data, the target speech data includes the frequency domain amplitude spectrum features of the speech signal to be enhanced; inputs the target speech data into a pre-trained speech enhancement model to obtain enhanced speech data output by the speech enhancement model; the speech enhancement model includes a U-type network and a joint attention mechanism module, the joint attention mechanism module includes a self-attention module and a global spatial attention module, the U-type network includes a U-type network encoder, a U-type network decoder and an output layer, the output end of the U-type network encoder is connected to the input end of the joint attention mechanism module, the output end of the joint attention mechanism module is connected to the input end of the U-type network decoder, and the output end of the U-type network decoder is connected to the output layer. Based on the feature modeling capability of the U-type network, the speech enhancement model, on the one hand, focuses on the relevance of context information through the self-attention module, and on the other hand, fully models the global features through the global spatial attention module, improves the compactness within the class and optimizes the feature representation, thereby improving the speech enhancement effect of the speech enhancement model and improving the speech quality of the speech enhancement result. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0036] Figure 1 A schematic diagram of a system architecture is shown;

[0037] Figure 2 An optional hardware structure diagram of the terminal 100 is shown;

[0038] Figure 3 A schematic diagram of the structure of a server 200 is provided;

[0039] Figure 4 An example of a spectrogram of a noisy speech signal with low signal-to-noise comparison is shown;

[0040] Figure 5 A flowchart of a speech enhancement method provided in an embodiment of the present application;

[0041] Figure 6 A method for constructing a speech enhancement model provided in an embodiment of the present application;

[0042] Figure 7 An example of a traditional U-shaped network structure diagram is shown;

[0043] Figure 8A U-shaped network based on a joint attention mechanism module provided in an embodiment of the present application;

[0044] Fig. 9 A functional schematic diagram of a self-attention module provided in an embodiment of the present application;

[0045] Fig.10 A functional schematic diagram of a global spatial attention module provided in an embodiment of the present application;

[0046] Fig.11 A schematic diagram of the structure of a speech enhancement model based on multi-scale skip connections provided in an embodiment of the present application;

[0047] Fig.12 A schematic diagram of a speech enhancement effect provided in an embodiment of the present application;

[0048] Fig.13 A schematic diagram of a speech enhancement device provided in an embodiment of the present application

[0049] Fig.14 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0051] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0052] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0053] The present application embodiment provides a speech enhancement method, which can be applied to, but not limited to, an application program having a speech enhancement method function or a cloud service provided by a cloud-side server, etc., which are described below respectively:

[0054] See also Figure 1 , Figure 1 A schematic diagram of a system architecture is shown. The system may include a terminal 100 and a server 200. The server 200 may include one or more servers ( Figure 1 In the example, a server is included, and the server 200 can provide the method provided in the embodiment of the present application for one or more terminals.

[0055] Among them, a voice enhancement application can be installed on the terminal 100. The above application and web page can provide an interface. The terminal 100 can receive relevant parameters entered by the user on the voice enhancement interface and send the above parameters to the server 200. The server 200 can obtain processing results based on the received parameters and return the processing results to the terminal 100.

[0056] It should be understood that in some optional implementations, the terminal 100 can also complete the action of obtaining the processing result based on the received parameters by itself without the cooperation of the server, and the embodiments of the present application are not limited to this.

[0057] Next describe Figure 1 The product form of the mid-terminal 100;

[0058] The terminal 100 in the embodiment of the present application can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the embodiment of the present application does not impose any limitation on this.

[0059] Figure 2 An optional hardware structure diagram of the terminal 100 is shown.

[0060] refer to Figure 2 As shown, the terminal 100 may include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160 (optional), a speaker 161 (optional), a microphone 162 (optional), an earphone jack 163 (optional), a processor 170, an external interface 180, a power supply 190 and other components. Those skilled in the art will appreciate that Figure 2 These are merely examples of terminals or multi-function devices and do not constitute limitations on the terminals or multi-function devices, which may include more or fewer components than those shown in the figures, or combinations of certain components, or different components.

[0061] The input unit 130 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the portable multifunctional device. Specifically, the input unit 130 may include a touch screen 131 (optional) and / or other input devices 132. The touch screen 131 can collect the user's touch operations on or near it (such as the user's operation on or near the touch screen using any suitable object such as fingers, joints, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 170, and can receive and execute the command sent by the processor 170; the touch signal at least includes the touch point coordinate information. The touch screen 131 can provide an input interface and an output interface between the terminal 100 and the user. In addition, the touch screen can be implemented using multiple types such as resistive, capacitive, infrared and surface acoustic wave. In addition to the touch screen 131, the input unit 130 can also include other input devices. Specifically, other input devices 132 may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0062] Among them, the input device 132 can receive input data and the like.

[0063] The display unit 140 may be used to display information input by the user or provided to the user, various menus of the terminal 100, interactive interfaces, file displays, and / or playback of any multimedia file. In an embodiment of the present application, the display unit 140 may be used to display an interface for speech enhancement, processing results, and the like.

[0064] The memory 120 can be used to store instructions and data. The memory 120 can mainly include an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files, texts, etc.; the instruction storage area can store software units such as operating systems, applications, instructions required for at least one function, or their subsets and extensions. It can also include a non-volatile random access memory; provide the processor 170 with hardware, software and data resources including management of computing and processing equipment, and support control software and applications. It is also used for the storage of multimedia files, and the storage of running programs and applications.

[0065] The processor 170 is the control center of the terminal 100. It uses various interfaces and lines to connect various parts of the entire terminal 100. By running or executing instructions stored in the memory 120 and calling data stored in the memory 120, it executes various functions of the terminal 100 and processes data, thereby controlling the terminal device as a whole. Optionally, the processor 170 may include one or more processing units; preferably, the processor 170 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 170. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips. The processor 170 may also be used to generate corresponding operation control signals, send them to corresponding components of the computing and processing device, read and process data in the software, especially read and process data and programs in the memory 120, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0066] Among them, the memory 120 can be used to store software codes related to the speech enhancement method, the processor 170 can execute the steps of the speech enhancement method, and can also schedule other units (such as the above-mentioned input unit 130 and display unit 140) to achieve corresponding functions.

[0067] The radio frequency unit 110 (optional) can be used for receiving and sending information or receiving and sending signals during a call, for example, after receiving the downlink information of the base station, it is sent to the processor 170 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA), a duplexer, etc. In addition, the radio frequency unit 110 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (Global System of Mobile communication, GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.

[0068] In this embodiment of the present application, the RF unit 110 can send data to the server 200 and receive processing results sent by the server 200.

[0069] It should be understood that the radio frequency unit 110 is optional and can be replaced by other communication interfaces, such as a network port.

[0070] The terminal 100 also includes a power supply 190 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 170 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.

[0071] The terminal 100 further includes an external interface 180 , which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the terminal 100 to communicate with other devices, or to connect a charger to charge the terminal 100 .

[0072] Although not shown, the terminal 100 may also include a flashlight, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which are not described in detail here. Some or all of the methods described below may be applied in the following embodiments. Figure 2 In the terminal 100 shown.

[0073] Next describe Figure 1 The product form of the server 200;

[0074] Figure 3 A structural diagram of a server 200 is provided, such as Figure 3 As shown, the server 200 includes a bus 201, a processor 202, a communication interface 203, and a memory 204. The processor 202, the memory 204, and the communication interface 203 communicate with each other via the bus 201.

[0075] The bus 201 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0076] The processor 202 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0077] The memory 204 may include a volatile memory, such as a random access memory (RAM). The memory 204 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0078] The memory 204 may be used to store software codes related to the speech enhancement method, and the processor 202 may execute the steps of the speech enhancement method of the chip, and may also schedule other units to implement corresponding functions.

[0079] It should be understood that the above-mentioned terminal 100 and server 200 can be centralized or distributed devices, and the processors in the above-mentioned terminal 100 and server 200 (such as processor 170 and processor 202) can be hardware circuits (such as application specific integrated circuit (ASIC), field-programmable gate array (FPGA), general-purpose processor, digital signal processor (DSP), microprocessor or microcontroller, etc.), or a combination of these hardware circuits. For example, the processor can be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0080] This application can be applied to the field of artificial intelligence technology, and specifically to voice interaction scenarios in banking systems. In voice interaction scenarios, microphones are used to collect user voices on the banking and financial equipment side. However, due to the noise generated by the operation of the equipment, the current sound generated by the microphone itself, and other environmental noises and human voices in the bank, the voice quality collected by the banking and financial equipment is poor. In addition, affected by the position of the microphone and the quality of the equipment, the signal-to-noise ratio of the voice is relatively low. Due to the above reasons, the performance of the voice recognition system for voice recognition has greatly declined, resulting in low accuracy of the voice recognition results, poor usability of the voice interaction function, and poor user experience.

[0081] Figure 4 An example of a spectrogram of a noisy speech signal with low signal-to-noise comparison. Figure 4 The low-mid frequency part representing speech (marked by a white rectangular box) and the high-frequency part representing speech (marked by a green rectangular box) are marked.

[0082] In the scenario of voice collection by financial equipment, the running sound of the fan in the equipment and the dull noise of people talking mainly affect the voice quality of the low-frequency part, while the harsh noise of the current sound of the microphone mainly affects the voice quality of the high-frequency part. The purpose of a voice enhancement method provided in an embodiment of the present application is to remove noise of various frequencies (including medium and low frequencies and high frequencies) in the voice, thereby improving the voice quality and further improving the accuracy of the background recognition system of the equipment end.

[0083] Existing speech interaction technologies include spectral subtraction, filter method, wavelet transform, time-frequency domain analysis, adaptive filtering and other traditional speech enhancement technologies. However, traditional speech enhancement technologies have high limitations, which are manifested in that speech enhancement technologies are highly dependent on noise types, have limited enhancement effects on speech with low signal-to-noise ratios, and have severe distortion after enhancement. Therefore, the processing effect of traditional speech enhancement technologies on complex environmental noise is often not ideal.

[0084] The speech enhancement technology based on deep learning uses a deep neural network model to stably and reliably model speech signals in complex acoustic scenes, thereby obtaining a more accurate speech enhancement effect. Among them, since the U-type network has a simple structure and a strong feature modeling capability, the speech enhancement model based on the U-type network can significantly improve the speech enhancement effect compared to the traditional speech enhancement technology. However, according to the research of the inventor of the present application, on the one hand, due to the loss of fine details caused by the stacking of convolutional layers and continuous sampling operations in the encoder part of the traditional U-type network, it is difficult to model the global features and cannot make good use of the global feature information. On the other hand, since the jump connection used by the traditional U-type network is only performed on the feature data of the same scale, the relationship between the feature data of different scales (decoding stage) is not explored. The above two aspects lead to the poor effect of traditional speech enhancement based on the traditional U-type network.

[0085] In order to solve the above problems, an embodiment of the present application proposes a speech enhancement method, which constructs a speech enhancement model based on a U-shaped network with a joint attention mechanism module and a multi-scale jump connection, and uses the speech enhancement model to enhance the speech. In order to address problems such as large equipment noise, large environmental noise, and poor sound collection, the method can fully extract effective information from the speech, improve the speech quality, and significantly improve the accuracy of the back-end recognition system.

[0086] Figure 5 A flow chart of a speech enhancement method provided in an embodiment of the present application is shown as follows: Figure 5 As shown, the method may include the following steps:

[0087] S501: Acquire target voice data.

[0088] In this embodiment, the target speech data includes frequency domain amplitude spectrum features of the speech signal to be enhanced.

[0089] In an optional embodiment, the method for obtaining target voice data includes:

[0090] A speech signal to be enhanced is obtained, and the speech signal to be enhanced is framed, sampled, and short-time Fourier transformed to obtain frequency domain amplitude spectrum features of the speech signal to be enhanced.

[0091] S502: Input the target speech data into a pre-trained speech enhancement model to obtain enhanced speech data output by the speech enhancement model.

[0092] In this embodiment, the speech enhancement model is constructed based on the joint attention mechanism module and the U-shaped network. Figure 5 As shown, the speech enhancement model includes a U-type network and a joint attention mechanism module, the joint attention mechanism module includes a self-attention module and a global space attention module, the U-type network includes a U-type network encoder, a U-type network decoder and an output layer, the output end of the U-type network encoder is connected to the input end of the joint attention mechanism module, and the output end of the joint attention mechanism module is connected to the input end of the U-type network decoder.

[0093] It can be seen from the above technical scheme that a speech enhancement method provided in an embodiment of the present application enhances target speech data based on a speech enhancement model to obtain enhanced speech data. The speech enhancement model is constructed based on a U-shaped network with a joint attention mechanism module composed of a self-attention module and a global spatial attention module. On the basis of the feature modeling capability of the U-shaped network, on the one hand, the self-attention module focuses on the relevance of contextual information, and on the other hand, the global features are fully modeled through the global spatial attention module to improve the compactness within the class and thus optimize the feature representation, thereby improving the speech enhancement effect of the speech enhancement model and improving the speech quality of the speech enhancement results.

[0094] It should be noted that Figure 6 A method for constructing a speech enhancement model provided in an embodiment of the present application is as follows: Figure 6 As shown, the method for constructing the speech enhancement model specifically includes:

[0095] S601. Construct a U-shaped network based on the joint attention mechanism module.

[0096] In this embodiment, the joint attention mechanism module includes a self-attention module and a global spatial attention module.

[0097] In this embodiment, the method for constructing a U-type network based on a joint attention mechanism module includes:

[0098] 1. Build a traditional U-shaped network.

[0099] Figure 7 A traditional U-shaped network structure diagram is shown as follows: Figure 7As shown in the figure, the traditional U-shaped network W1 is composed of a cascade of an encoder, a decoder, and an output layer, wherein the encoder is composed of multiple encoder blocks, and the encoder block in the encoder includes a convolution layer. Each convolution layer is used to extract features from the input speech and output feature data. As the number of encoder blocks increases, the number of channels of the feature data output by the encoder will gradually increase, so that more abundant feature information can be extracted. The decoder is composed of multiple decoder blocks, and the decoder block includes a transposed convolution layer and a convolution layer. The decoder block is used to upsample the feature data output by the encoder, and fuse the upsampled features with the feature data output by the corresponding (symmetrically deployed) encoder block through a jump connection. Each transposed convolution layer in the decoder block is used to restore the low-resolution feature data to its original size. The output layer is composed of a convolution layer, and the output layer is used to regress the feature data output by the decoder to obtain the final speech enhancement result.

[0100] In an optional embodiment, each encoder consists of 2 layers of 2D convolutional layers with a kernel size of 3 and a stride of 1. Each 2D convolutional layer in the encoder is followed by a batch normalization layer and an activation function layer using a linear rectification (ReLU, Rectified linear Unit) function. Each decoder consists of a transposed convolutional layer with a kernel size of 2 and a stride of 2 and 2 layers of 2D convolutional layers with a kernel size of 3 and a stride of 3. Each 2D convolutional layer in the decoder is followed by a batch normalization layer and an activation function layer using a linear rectification function. The output layer is a deconvolution layer with a kernel size of 1 and a stride of 1.

[0101] 2. Add a joint attention mechanism module to the traditional U-shaped network.

[0102] In this embodiment, the joint attention mechanism module is composed of a self-attention module and a global spatial attention module in parallel. The joint attention mechanism module is configured between the encoder and the decoder in the traditional U-shaped network, and is used to connect the last convolution layer of the encoder and the first transposed convolution layer of the decoder.

[0103] Figure 8 A U-shaped network based on a joint attention mechanism module is provided in an embodiment of the present application, such as Figure 8As shown, in the U-type network W2 based on the joint attention mechanism module, the self-attention module and the global space attention module are configured in parallel between the traditional U-type network encoder and decoder, the input ends of the self-attention module and the global space attention module are connected to the encoder, and the output ends of the self-attention module, the global space attention module and the encoder are connected to the decoder input end through a feature fusion layer, wherein the feature fusion layer is a feature addition layer, which is used to fuse the output data of the self-attention module, the output data of the global space attention module, and the output data of the U-type network encoder, and then input them into the U-type network decoder.

[0104] In this embodiment, the self-attention module can calculate the correlations between different positions in the feature data (such as a feature map), and use these correlations to calculate the weighted sum of each position, thereby improving the model's ability to understand global features.

[0105] Fig. 9 A functional schematic diagram of the self-attention module provided in the embodiment of the present application, such as Fig. 9 As shown in Figure 1, in order to obtain context information, the self-attention module first introduces the learned position encoding into the output of the encoder, that is, into the input feature map of the self-attention module, thereby preserving the position information of the sequence. Then, using the output of the new encoder , generate the query matrix , the bond matrix Sum Matrix , the calculation formula is as follows (1):

[0106] (1);

[0107] in, , and represents different linear transformation matrices. Then, The transpose of is scaled dot product operation and normalized using normalized exponential function to generate context attention map . express The given element and Then, the context attention map Multiply , thereby obtaining the feature data after attention weighting The calculation formula is shown in formula (2):

[0108] (2);

[0109] in, is the dimension of the query / key-value sequence.

[0110] Finally, by transforming the size of the feature data, we obtain the output feature map of the self-attention module. .

[0111] In this embodiment, the global spatial attention module can selectively aggregate global information into the learned features, thereby further improving the compactness within the class and optimizing the feature representation.

[0112] Fig.10 A functional schematic diagram of the global spatial attention module provided in an embodiment of the present application, such as Fig.10 As shown in the figure, the global spatial attention module firstly processes the output of the underlying encoder, which is also the input feature map of the self-attention module. , using two different types of convolution operations to generate feature data and ,in . Then, The size of the image is transformed and transposed to generate feature data respectively and ,Will Transpose Then, and Perform matrix multiplication and normalize using the normalized exponential function to obtain the position attention map Among them Position to Position Impact The calculation formulas are shown in equations (3) and (4):

[0113] (3);

[0114] (4).

[0115] Finally, and Multiply them together to get the global spatial attention feature , the calculation formula is shown in formula (5):

[0116] (5).

[0117] It should be noted that the size of the output global spatial attention feature is adjusted to generate the output feature map of the global spatial attention module .

[0118] In order to make full use of the contextual information and global information obtained, the joint attention mechanism module obtains the feature data output by the joint attention mechanism module through weighted addition of the input features (i.e., the encoder output features), the output features of the self-attention module, and the output features of the global spatial attention module. The calculation formula is shown in formula (6):

[0119] (6);

[0120] in, Represents the output of the encoder, which is also the input feature data of the joint attention mechanism module. and They represent the weight coefficients of the self-attention module and the global spatial attention module respectively. The initial values ​​of these two parameters are set to 0 and are adaptively adjusted during the training process so that the decoder can obtain the optimal potential features.

[0121] S602. Add multi-scale skip connections to the U-shaped network based on the joint attention mechanism module to obtain an initial speech enhancement model.

[0122] In this embodiment, the initial speech enhancement model is the speech enhancement model to be trained, the model parameters of the speech enhancement model are initialization parameters, the multi-scale jump connection includes multiple series blocks and sampling layers, and the multi-scale jump connection aggregates features of different semantic scales through upsampling, series connection and convolution.

[0123] Fig.11 A schematic diagram of the structure of a speech enhancement model based on multi-scale skip connections provided in an embodiment of the present application.

[0124] like Fig.11 As shown, the input of the first series block is connected to the output of the first decoder block, and is connected to the output of the joint attention mechanism module through the sampling layer;

[0125] The input end of the nth series block is connected to the output end of the nth decoder block, and is connected to the output end of the n-1th series block through the sampling layer; the output end of the nth series block is connected to the input end of the n+1th decoder block, and is connected to the input end of the n+1th series block through the sampling layer, where n is greater than 1 and less than N, and N is the number of decoder blocks;

[0126] The input end of the Nth series block is connected to the output end of the Nth decoder block, and is connected to the output end of the N-1th series block through the sampling layer; the output end of the Nth series block is the output end of the U-type network decoder;

[0127] The series connection block is used to perform series connection and splicing on the input data, and the sampling layer is used to perform bilinear interpolation up-sampling on the input data.

[0128] Each concatenation block is used to concatenate the input data, and each sampling layer is used to perform bilinear interpolation upsampling on the input data.

[0129] For the target decoder block (any encoder block), the feature data of the decoder block channel input to the target decoder block is upsampled to the output feature data size through bilinear interpolation, and then concatenated with the feature data output by the target decoder block as the input of the decoder block after the target decoder block.

[0130] In this regard, The decoder block receives input from the decoder part The calculation formula is shown in formula (7):

[0131] (7)

[0132] in, and Respectively represent A 2D convolution operation and an upsampling operation.

[0133] by Fig.11 For example, the decoder block F4 receives the input from the decoder part. The calculation formula is: .

[0134] It should be noted that the multi-scale skip connection can gradually aggregate features from multiple decoders of different scales, thereby generating discriminative feature representations. Furthermore, the multi-scale skip connection scheme can alleviate the problem of some detailed information being ignored during the decoding process, and can also solve the problems of vanishing gradients and overfitting.

[0135] In summary, by adding multi-scale skip connections and joint attention mechanism modules to the traditional U-shaped network, we can construct Fig.11 The initial speech enhancement model shown. The initial speech enhancement model provided in the embodiment of the present application is based on the U-shaped network, and an attention module is innovatively added to the network, so that the network can fully model the global features. And a multi-scale jump connection scheme is used to aggregate features of different semantic scales, alleviating the problem that some detailed information is ignored during the decoding process.

[0136] S603: Acquire original voice data.

[0137] In this embodiment, the original speech data includes a noisy speech signal and a clean speech signal.

[0138] S604: Frame, sample, and perform short-time Fourier transform on the original speech data to obtain frequency domain amplitude spectrum features of the original speech data.

[0139] In this embodiment, the voice data is a continuous analog signal. In order to perform digital signal processing and analysis, it needs to be framed and sampled. In this embodiment, the original voice data is framed and sampled to obtain a clean voice signal. and noisy speech signal Among them, the frame length of the frame is , frame shift is ms, the sampling rate is In this embodiment, the framed and sampled speech is subjected to a short-time Fourier transform to obtain the frequency domain amplitude spectrum of the clean speech signal and the noisy speech signal. and The window function used in the short-time Fourier transform is the Hamming window, and the window length is , the sampling rate is The standard short-time Fourier transform formula is shown in equation (8):

[0140] (8)

[0141] in, For time, is the time domain signal, is the window function, for The corresponding frequency domain amplitude spectrum.

[0142] It should be noted that the specific methods of framing, sampling and short-time Fourier transform can be referred to in the prior art.

[0143] S605. Acquire training data and test data based on the frequency domain amplitude spectrum features of the original speech data.

[0144] In this embodiment, the training data includes frequency domain amplitude spectrum features of multiple noisy speech signals and frequency domain amplitude spectrum features of corresponding clean speech signals. The test data includes frequency domain amplitude spectrum features of multiple noisy speech signals.

[0145] S606: Train an initial speech enhancement model based on the training data to obtain a candidate speech enhancement model.

[0146] In this embodiment, the frequency domain amplitude spectrum features are input into the initial speech enhancement model to train the model parameters of the initial speech enhancement model.

[0147] In this embodiment, the input of the initial speech enhancement model is the frequency domain amplitude spectrum features of the clean speech signal and the speech to be enhanced (that is, the noisy speech signal), and the loss function used is the L1 loss function, which is shown in formula (9):

[0148] (9)

[0149] in, The frequency domain amplitude spectrum characteristics of the clean speech signal represented by Represents the frequency domain amplitude spectrum characteristics of enhanced speech.

[0150] In this embodiment, the Adam optimization algorithm is used to train the randomly sampled training data during the training process. Optionally, the decay rate of the Adam optimizer Set to 0.9, the decay rate It is set to 0.999.

[0151] S607. Test the candidate speech enhancement model based on the test data to obtain performance evaluation parameters of the candidate speech enhancement model.

[0152] In this embodiment, the frequency domain amplitude spectrum features of the noisy speech signal are input into the trained candidate speech enhancement model to obtain the enhanced speech corresponding to the noisy speech signal, and the speech evaluation parameters of each enhanced speech are calculated, for example, the perceptual speech quality evaluation, the short-time objective intelligibility and the segmented signal-to-noise ratio value. Based on the speech evaluation parameters of each enhanced speech, the model performance evaluation parameters of the candidate speech enhancement model are evaluated and obtained. The model performance evaluation parameters include various parameters for evaluating the speech enhancement performance of the candidate model.

[0153] S608: If the model performance evaluation parameters of the candidate speech enhancement model meet the preset performance conditions, the candidate speech enhancement model is used as the trained speech enhancement model.

[0154] Fig.12 A schematic diagram of a speech enhancement effect provided in an embodiment of the present application is shown in FIG. Fig.12 The following example shows a noisy speech signal with a signal-to-noise ratio of 5 dB, the corresponding clean speech signal, the enhanced speech after being enhanced by the U-shaped network, and the U-shaped network based on attention (referring to Figure 8 Compared with the traditional U-shaped network, the enhanced speech output by the speech enhancement model provided in the embodiment of the present application contains significantly less noise in the low- and medium-frequency (marked by white rectangular boxes) and high-frequency areas (marked by green rectangular boxes) of the speech, and is closer to the spectrum of the original clean speech signal. It can be seen that the speech enhancement model provided in the embodiment of the present application can more effectively enhance the input speech, improve the quality of the input speech, and thus improve the recognition rate of the back-end system of the banking and financial equipment end.

[0155] In summary, considering the characteristics of noise at the banking and financial equipment end in real environments, this scheme adds a joint attention mechanism and multi-scale jump connections on the basis of the traditional U-shaped network, thereby improving the speech enhancement effect.

[0156] Specifically, the joint attention mechanism module combines the self-attention module and the global space attention module. The self-attention module in the attention module can effectively learn the global information of the input sequence, and the global space attention module further learns the global information of the input sequence, so that the network can fully model the global features. Therefore, the joint attention mechanism module uses the self-attention mechanism and the global space attention mechanism to effectively establish the contextual interaction and global spatial relationship between the semantic features of the encoder, and fully extract the frequency domain features of the speech. Further, this scheme uses learnable parameters to adaptively adjust the weights of the outputs of the self-attention and global space attention modules, so that the model can obtain the optimal enhancement effect. Further, the multi-scale connection structure in this scheme uses a series of operations such as upsampling, concatenation and convolution to fuse the feature data of each decoder block with the output of the decoder block on the previous layer, and the fused feature data is used as part of the input of the next decoder, thereby realizing the aggregation of semantic features of different scales and alleviating the problem that some detail information is ignored during the decoding process. In summary, compared with the prior art, a speech enhancement model provided in an embodiment of the present application can extract more effective information from the input speech, thereby improving the enhancement effect of the speech.

[0157] The above introduces a speech enhancement method provided in an embodiment of the present application. The following will introduce a system for executing the above speech enhancement method. A speech enhancement system provided in an embodiment of the present application includes a speech enhancement device and a speech enhancement model.

[0158] See also Fig.13 , Fig.13 This is a schematic diagram of the structure of a speech enhancement device provided in an embodiment of the present application. Fig.13 As shown, the speech enhancement device 1300 includes:

[0159] The data preprocessing unit 1301 is used to obtain target speech data, wherein the target speech data includes frequency domain amplitude spectrum characteristics of the speech signal to be enhanced;

[0160] The speech enhancement unit 1302 is used to input the target speech data into a pre-trained speech enhancement model to obtain enhanced speech data output by the speech enhancement model; the speech enhancement model includes a U-type network and a joint attention mechanism module, the joint attention mechanism module includes a self-attention module and a global spatial attention module, the U-type network includes a U-type network encoder, a U-type network decoder and an output layer, the output end of the U-type network encoder is connected to the input end of the joint attention mechanism module, the output end of the joint attention mechanism module is connected to the input end of the U-type network decoder, and the output end of the U-type network decoder is connected to the output layer.

[0161] Optionally, when the data preprocessing unit is used to obtain the target voice data, it is specifically used to:

[0162] The speech signal to be enhanced is acquired, and the speech signal to be enhanced is framed, sampled, and short-time Fourier transformed to obtain frequency domain amplitude spectrum features of the speech signal to be enhanced.

[0163] Optionally, the output end of the U-type network encoder is connected to the input end of the joint attention mechanism module, including: the output end of the U-type network encoder is connected to the input end of the self-attention module and the input end of the global space attention module respectively, for inputting the output data input of the U-type network encoder to the self-attention module and the global space attention module respectively;

[0164] The output end of the joint attention mechanism module is connected to the input end of the U-shaped network decoder including:

[0165] The output end of the self-attention module, the output end of the global spatial attention module, and the output end of the U-type network encoder are connected to the input end of the U-type network decoder through a feature fusion layer. The feature fusion layer is used to fuse the output data of the self-attention module, the output data of the global spatial attention module, and the output data of the U-type network encoder, and then input them into the U-type network decoder.

[0166] Optionally, the speech enhancement model further includes a multi-scale skip connection, wherein the multi-scale skip connection includes a plurality of series blocks and a plurality of sampling layers;

[0167] The input of the first concatenation block is connected to the output of the first decoder block and connected to the output of the joint attention mechanism module through the sampling layer;

[0168] The input end of the nth series block is connected to the output end of the nth decoder block, and is connected to the output end of the n-1th series block through the sampling layer; the output end of the nth series block is connected to the input end of the n+1th decoder block, and is connected to the input end of the n+1th series block through the sampling layer, where n is greater than 1 and less than N, and N is the number of decoder blocks;

[0169] The input end of the Nth series block is connected to the output end of the Nth decoder block, and is connected to the output end of the N-1th series block through the sampling layer; the output end of the Nth series block is the output end of the U-type network decoder;

[0170] The series connection block is used to perform series connection and splicing on the input data, and the sampling layer is used to perform bilinear interpolation up-sampling on the input data.

[0171] Optionally, the speech enhancement device also includes: a model building unit, which is used to: obtain original speech data, wherein the original speech data includes a noisy speech signal and a clean speech signal; frame, sample, and short-time Fourier transform each speech signal in the original speech data to obtain frequency domain amplitude spectrum features of each speech signal in the original speech data; obtain training data and test data, wherein the training data includes frequency domain amplitude spectrum features of multiple noisy speech signals and corresponding frequency domain amplitude spectrum features of clean speech signals, and the test data includes frequency domain amplitude spectrum features of multiple noisy speech signals; train the original speech enhancement model based on the training data to obtain a candidate speech enhancement model; and test the candidate speech enhancement model based on the test data to obtain a trained speech enhancement model.

[0172] The present application also provides an electronic device in an embodiment. Fig.14 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Fig.14 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0173] like Fig.14As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1402 or a program loaded from a storage device 1408 to a random access memory (RAM) 1403. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 1403. The processing device 1401, the ROM 1402, and the RAM 1403 are connected to each other via a bus 1404. An input / output (I / O) interface 1405 is also connected to the bus 1404.

[0174] Typically, the following devices may be connected to the I / O interface 1405: an input device 1406 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 1407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1408 including, for example, a memory card, a hard disk, etc.; and a communication device 1409. The communication device 1409 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0175] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any of the speech enhancement methods provided in the embodiments of the present application.

[0176] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the speech enhancement methods provided in the embodiments of the present application.

[0177] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0178] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0179] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0180] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A speech enhancement method, characterized in that: include: Acquire target speech data, wherein the target speech data includes frequency domain amplitude spectrum features of the speech signal to be enhanced; The target speech data is input into a pre-trained speech enhancement model to obtain enhanced speech data output by the speech enhancement model; the speech enhancement model includes a U-type network and a joint attention mechanism module, the joint attention mechanism module includes a self-attention module and a global space attention module, the U-type network includes a U-type network encoder, a U-type network decoder and an output layer, the output end of the U-type network encoder is connected to the input end of the joint attention mechanism module, the output end of the joint attention mechanism module is connected to the input end of the U-type network decoder, and the output end of the U-type network decoder is connected to the output layer.

2. The speech enhancement method according to claim 1, characterized in that: Obtaining target voice data includes: The speech signal to be enhanced is acquired, and the speech signal to be enhanced is framed, sampled, and short-time Fourier transformed to obtain frequency domain amplitude spectrum features of the speech signal to be enhanced.

3. The speech enhancement method according to claim 1, characterized in that: The output end of the U-shaped network encoder is connected to the input end of the joint attention mechanism module, including: the output end of the U-shaped network encoder is connected to the input end of the self-attention module and the input end of the global space attention module respectively, for inputting the output data input of the U-shaped network encoder to the self-attention module and the global space attention module respectively; The output end of the joint attention mechanism module is connected to the input end of the U-shaped network decoder including: The output end of the self-attention module, the output end of the global spatial attention module, and the output end of the U-type network encoder are connected to the input end of the U-type network decoder through a feature fusion layer. The feature fusion layer is used to fuse the output data of the self-attention module, the output data of the global spatial attention module, and the output data of the U-type network encoder, and then input them into the U-type network decoder.

4. The speech enhancement method according to claim 3, characterized in that: The speech enhancement model further includes a multi-scale jump connection, wherein the multi-scale jump connection includes a plurality of series blocks and a plurality of sampling layers; The input of the first concatenation block is connected to the output of the first decoder block and connected to the output of the joint attention mechanism module through the sampling layer; The input end of the nth series block is connected to the output end of the nth decoder block, and is connected to the output end of the n-1th series block through the sampling layer; the output end of the nth series block is connected to the input end of the n+1th decoder block, and is connected to the input end of the n+1th series block through the sampling layer, where n is greater than 1 and less than N, and N is the number of decoder blocks; The input end of the Nth series block is connected to the output end of the Nth decoder block, and is connected to the output end of the N-1th series block through the sampling layer; the output end of the Nth series block is the output end of the U-type network decoder; The series connection block is used to perform series connection and splicing on the input data, and the sampling layer is used to perform bilinear interpolation up-sampling on the input data.

5. The speech enhancement method according to claim 4, characterized in that: The speech enhancement method further comprises: Acquire original speech data, wherein the original speech data includes a noisy speech signal and a clean speech signal; Performing frame division, sampling, and short-time Fourier transformation on each speech signal in the original speech data to obtain frequency domain amplitude spectrum characteristics of each speech signal in the original speech data; Acquire training data and test data, wherein the training data includes frequency domain amplitude spectrum features of multiple noisy speech signals and frequency domain amplitude spectrum features of corresponding clean speech signals, and the test data includes frequency domain amplitude spectrum features of multiple noisy speech signals; Training an original speech enhancement model based on the training data to obtain a candidate speech enhancement model; The candidate speech enhancement model is tested based on the test data to obtain a trained speech enhancement model.

6. A speech enhancement device, characterized in that: include: A data preprocessing unit, used to obtain target speech data, wherein the target speech data includes frequency domain amplitude spectrum features of the speech signal to be enhanced; A speech enhancement unit is used to input the target speech data into a pre-trained speech enhancement model to obtain enhanced speech data output by the speech enhancement model; the speech enhancement model includes a U-type network and a joint attention mechanism module, the joint attention mechanism module includes a self-attention module and a global space attention module, the U-type network includes a U-type network encoder, a U-type network decoder and an output layer, the output end of the U-type network encoder is connected to the input end of the joint attention mechanism module, the output end of the joint attention mechanism module is connected to the input end of the U-type network decoder, and the output end of the U-type network decoder is connected to the output layer.

7. A speech enhancement system, characterized in that: include: Speech enhancement device and pre-trained speech enhancement model, The speech enhancement device is used to obtain target speech data and input the target speech data into a pre-trained speech enhancement model; the target speech data includes frequency domain amplitude spectrum features of the speech signal to be enhanced; The speech enhancement model is used to output enhanced speech data; the speech enhancement model includes a U-type network and a joint attention mechanism module, the joint attention mechanism module includes a self-attention module and a global space attention module, the U-type network includes a U-type network encoder, a U-type network decoder and an output layer, the output end of the U-type network encoder is connected to the input end of the joint attention mechanism module, the output end of the joint attention mechanism module is connected to the input end of the U-type network decoder, and the output end of the U-type network decoder is connected to the output layer.

8. A computer program product, characterized in that It comprises computer-readable instructions, and when the computer-readable instructions are executed on an electronic device, the electronic device implements the speech enhancement method as claimed in any one of claims 1 to 5.

9. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the speech enhancement method as described in any one of claims 1 to 5.

10. A computer storage medium, characterized in that: The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the speech enhancement method as described in any one of claims 1 to 5.

Images