Speech enhancement method, speech enhancement device, electronic device, and storage medium

Abstract

The application provides a speech enhancement method, a speech enhancement device, an electronic device and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: obtaining original speech data; inputting the original speech data into a speech enhancement model, performing encoding processing on the original speech data through an encoder to obtain encoded speech features; performing first speech enhancement on the encoded speech features through a first speech enhancement module to obtain first enhanced speech features; performing second speech enhancement on the first enhanced speech features through a second speech enhancement module to obtain second enhanced speech features; performing feature fusion on the first enhanced speech features, the second enhanced speech features and the encoded speech features through a feature fusion module to obtain target enhanced speech features; and performing decoding processing on the target enhanced speech features through a decoder to obtain enhanced speech data. The application can improve the speech quality of the enhanced speech. The application can be used in scenes such as e-commerce and online transactions.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a speech enhancement method, a speech enhancement device, an electronic device and a storage medium. BACKGROUND

[0002] Currently, in various financial scenarios such as e-commerce and online transactions, conversations between objects are often implemented in the form of voice communication.

[0003] Taking insurance recommendation as an example, when a staff member introduces an insurance product to an object through network video, network audio or other means, if the staff member directly transmits the description voice of the insurance product to the object, the clarity of the description voice received by the object will be greatly affected by factors such as the environment in which the staff member is located and the quality of the transmission device. For example, when the staff member is in a relatively noisy environment and has a conversation with the object, there will be a lot of environmental noise in the description voice of the insurance product, and the voice quality of the description voice is poor. At this time, the object may not be able to clearly determine the specific content contained in the description voice, and may not be able to judge whether this insurance product meets its own needs, thereby affecting the normal recommendation of the insurance product.

[0004] In related technologies, a speech enhancement method based on deep learning is often used to perform speech enhancement on conversation voice to improve voice quality. However, the speech enhancement method in related technologies often ignores the influence of noise signals, which can result in poor voice quality of the enhanced voice. SUMMARY

[0005] The main purpose of the embodiments of the present application is to propose a speech enhancement method, a speech enhancement device, an electronic device and a storage medium, which aims to improve the voice quality of the enhanced voice.

[0006] To achieve the above-mentioned purpose, a first aspect of the embodiments of the present application proposes a speech enhancement method, which comprises:

[0007] obtaining original voice data, wherein the original voice data is voice data containing noise;

[0008] inputting the original voice data into a preset speech enhancement model, wherein the speech enhancement model comprises an encoder, a speech enhancement network and a decoder, and the speech enhancement network comprises a first speech enhancement module, a second speech enhancement module and a feature fusion module;

[0009] performing encoding processing on the original voice data through the encoder to obtain encoded voice features;

[0010] performing first speech enhancement on the encoded voice features through the first speech enhancement module to obtain first enhanced voice features;

[0011] performing second speech enhancement on the first enhanced speech feature through the second speech enhancement module to obtain a second enhanced speech feature;

[0012] performing feature fusion on the first enhanced speech feature, the second enhanced speech feature, and the encoded speech feature through the feature fusion module to obtain a target enhanced speech feature;

[0013] performing decoding processing on the target enhanced speech feature through the decoder to obtain enhanced speech data, wherein the enhanced speech data contains less noise than the original speech data.

[0014] In some embodiments, the encoder comprises a first convolutional layer, a first normalization layer, and a first activation function, and the encoding processing of the original speech data through the encoder comprises:

[0015] performing feature extraction on the original speech data through the first convolutional layer to obtain a preliminary speech feature;

[0016] performing normalization processing on the preliminary speech feature through the first normalization layer to obtain a preliminary normalized feature;

[0017] performing activation processing on the preliminary normalized feature through the first activation function to obtain an encoded speech feature.

[0018] In some embodiments, the first speech enhancement module comprises a first branch network and a second branch network, and the first speech enhancement processing of the encoded speech feature through the first speech enhancement module comprises:

[0019] performing dimension reduction processing on the encoded speech feature to obtain a low-dimensional speech feature;

[0020] performing speech enhancement on the low-dimensional speech feature through the first branch network to obtain a third enhanced speech feature;

[0021] performing speech enhancement on the low-dimensional speech feature through the second branch network to obtain a fourth enhanced speech feature;

[0022] performing splicing processing on the third enhanced speech feature and the fourth enhanced speech feature to obtain a spliced speech feature;

[0023] performing dimension increasing processing on the spliced speech feature to obtain the first enhanced speech feature, wherein the feature dimensions of the first enhanced speech feature and the encoded speech feature are the same.

[0024] In some embodiments, the speech enhancement on the low-dimensional speech feature through the first branch network comprises:

[0025] The low-dimensional speech feature is activated through the first branch network to obtain a first activated speech feature;

[0026] The first activated speech feature is normalized to obtain a first normalized speech feature;

[0027] The first normalized speech feature is convoluted through a dilated convolution layer of the first branch network to obtain the third enhanced speech feature.

[0028] In some embodiments, the speech enhancement on the low-dimensional speech feature through the second branch network comprises:

[0029] The low-dimensional speech feature is activated through the second branch network to obtain a second activated speech feature;

[0030] The second activated speech feature is normalized to obtain a second normalized speech feature;

[0031] The second normalized speech feature is convoluted through a dilated convolution layer of the second branch network to obtain a convoluted speech feature;

[0032] The convoluted speech feature is adjusted in feature distribution through a preset function of the second branch network to obtain the fourth enhanced speech feature.

[0033] In some embodiments, before the original speech data is input into the preset speech enhancement model, the method further comprises pre-training the speech enhancement model, specifically comprising:

[0034] Obtaining sample speech data, wherein the sample speech data is speech data containing noise;

[0035] The sample speech data is encoded through a preset original model to obtain sample encoded features;

[0036] The sample encoded features are speech enhanced through the original model to obtain sample enhanced speech features;

[0037] The sample enhanced speech features are decoded through the original model to obtain sample enhanced speech data;

[0038] The original model is loss calculated based on the sample enhanced speech data, the sample speech data, and reference speech data obtained in advance to obtain a model loss value;

[0039] perform parameter updating on the original model based on the model loss value, to obtain the speech enhancement model.

[0040] In some embodiments, the loss calculation on the original model based on the sample enhanced speech data, the sample speech data and the pre-acquired reference speech data to obtain a model loss value comprises:

[0041] perform first loss calculation based on the sample enhanced speech data and the sample speech data to obtain a mean square error loss value;

[0042] perform second loss calculation based on the sample enhanced speech data, the sample speech data and the reference speech data to obtain a contrast loss value;

[0043] obtain the model loss value based on the mean square error loss value and the contrast loss value.

[0044] To achieve the above object, a second aspect of embodiments of the present application proposes a speech enhancement device, the device comprising:

[0045] To achieve the above object, a third aspect of embodiments of the present application proposes an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the method of the first aspect when executing the computer program.

[0046] To achieve the above object, a fourth aspect of embodiments of the present application proposes a computer readable storage medium, the computer readable storage medium storing a computer program, and the computer program implementing the method of the first aspect when executed by a processor.

[0047] The voice enhancement method, the voice enhancement device, the electronic equipment and the storage medium provided by the application can realize extraction of voice feature information in the original voice data by obtaining original voice data, inputting the original voice data into a preset voice enhancement model, and performing encoding processing on the original voice data by an encoder to obtain encoded voice features. Then, the first voice enhancement module is used to perform first voice enhancement on the encoded voice features to obtain first enhanced voice features, the second voice enhancement module is used to perform second voice enhancement on the first enhanced voice features to obtain second enhanced voice features, and the feature fusion module is used to perform feature fusion on the first enhanced voice features, the second enhanced voice features and the encoded voice features to obtain target enhanced voice features. The target enhanced voice features can contain voice enhancement information generated based on the first voice enhancement module and voice enhancement information generated based on the second voice enhancement module, and the feature richness and the feature comprehensiveness of the target enhanced voice features are improved. Finally, the decoder is used to perform decoding processing on the target enhanced voice features to obtain enhanced voice data, which can effectively weaken the noise signal in the original voice data, improve the voice quality of the enhanced voice data, and enable each object in various financial scenarios such as e-commerce and online transactions to clearly receive specific content described by another conversation object in the enhanced voice data, thereby improving the conversation efficiency of the voice conversation. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 is a flowchart of the voice enhancement method provided by the embodiment of the application;

[0049] Figure 2 is another flowchart of the voice enhancement method provided by the embodiment of the application;

[0050] Figure 3 is a flowchart of step S205 in Figure 2

[0051] Figure 4 is a structural schematic diagram of the voice enhancement model provided by the embodiment of the application;

[0052] Figure 5 is a structural schematic diagram of the voice enhancement network of the voice enhancement model in Figure 4

[0053] Figure 6 is a structural schematic diagram of the first voice enhancement module in Figure 5

[0054] Figure 7 is a flowchart of step S103 in Figure 1

[0055] Figure 8 is a flowchart of step S104 in Figure 1 ​​​​​

[0056] Figure 9 is Figure 8 a flowchart of step S802 in

[0057] Figure 10 is Figure 8 a flowchart of step S803 in

[0058] Figure 11 is a structural schematic diagram of a voice enhancement device provided by an embodiment of the present application;

[0059] Figure 12 is a hardware structure schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0061] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a manner different from the module division in the device or the order in the flowchart. The terms "first", "second", and the like in the specification and claims and the above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0063] First, the terms involved in the present application are analyzed:

[0064] Artificial intelligence (AI): is a new technical science of researching, developing, simulating, extending and expanding human intelligence, and is a branch of computer science. Artificial intelligence attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. The research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. Artificial intelligence can simulate the information process of human consciousness and thinking. Artificial intelligence is also the theory, method, technology and application system of using digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, to perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0065] Natural language processing (NLP): NLP uses computers to process, understand and use human languages (such as Chinese, English, etc.). NLP is a branch of artificial intelligence and is an interdisciplinary subject of computer science and linguistics, and is also commonly referred to as computational linguistics. Natural language processing includes syntax analysis, semantic analysis, discourse understanding, etc. Natural language processing is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent recognition, information extraction and filtering, text classification and clustering, public opinion analysis and opinion mining, etc. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research related to language processing, and linguistic research related to language computing.

[0066] Information extraction (Information Extraction, NER): A text processing technology that extracts specified types of entities, relationships, events, etc. from natural language text and forms structured data output. Information extraction is a technology for extracting specific information from text data. Text data is composed of specific units such as sentences, paragraphs, and chapters, and text information is composed of specific units such as words, phrases, sentences, paragraphs, or combinations of these specific units. Extracting noun phrases, names, and places from text data is text information extraction, and of course, the information extracted by the text information extraction technology can be various types of information.

[0067] Currently, in the scenarios of e-commerce, online transactions, etc., the conversation between objects is often implemented in the form of voice communication.

[0068] Taking the recommendation of insurance and financial products as an example, when the staff introduces the insurance and financial products to the object through network video, network audio, etc., if the staff directly transmits the description voice of the insurance and financial products to the object, the clarity of the description voice received by the object will be greatly affected by factors such as the environment in which the staff is located, the quality of the transmission device, etc.

[0069] For example, when the staff is in a relatively noisy environment and has a conversation with the object, there will be a lot of environmental noise in the description voice of the insurance and financial products, and the voice quality of the description voice is poor. At this time, the object may not be able to clearly determine the specific content contained in the description voice, and cannot well judge whether this insurance and financial product meets its own needs, affecting the normal recommendation of the insurance and financial product.

[0070] In the related art, a speech enhancement method based on deep learning is often used to perform speech enhancement on conversational speech to improve speech quality. However, the speech enhancement method in the related art often ignores the influence of noise signals, which can result in poor speech quality of the enhanced speech.

[0071] Based on this, the embodiments of the present application provide a speech enhancement method, a speech enhancement device, an electronic device and a storage medium, aiming to improve the speech quality of the enhanced speech.

[0072] The speech enhancement method and device, the electronic device and the storage medium provided by the embodiments of the present application are specifically described by the following embodiments. First, the speech enhancement method in the embodiments of the present application is described.

[0073] The embodiments of the present application can acquire and process related data based on artificial intelligence technology. Artificial intelligence (AI) is the use of digital computers or computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0074] The basic technology of artificial intelligence generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. The software technology of artificial intelligence mainly includes computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0075] The speech enhancement method provided by the embodiments of the present application relates to the field of artificial intelligence technology. The speech enhancement method provided by the embodiments of the present application can be applied in a terminal, can also be applied in a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc.; the server can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and basic cloud computing services such as big data and artificial intelligence platforms; the software can be an application that implements the speech enhancement method, but is not limited to the above forms.

[0076] The application is operable in a variety of general purpose or special purpose computer systems environments or configurations. Examples of well-known computing systems, environments, and / or configurations that can be suitable for use with the application include personal computers, server computers, handheld or laptop devices, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. The application can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like, that perform particular tasks or implement particular abstract data types. The application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in local and remote computer storage media including memory storage devices.

[0077] It should be noted that in each of the specific embodiments of the present application, when it is necessary to perform relevant processing according to object information, object behavior data, object historical data, and object location information, and other data related to the identity or characteristics of the object, the permission or consent of the object will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiments of the present application need to obtain personal information of the object, the separate permission or separate consent of the object will be obtained through a pop-up window or by jumping to a confirmation page, and after obtaining the separate permission or separate consent of the object, the necessary object-related data for enabling the embodiments of the present application to operate normally will be obtained.

[0078] Figure 1 is an optional flowchart of the voice enhancement method provided by the embodiments of the present application, Figure 1 The method in the above embodiment can include but is not limited to including steps S101 to S107.

[0079] Step S101, obtaining original voice data;

[0080] Step S102, inputting the original voice data into a preset voice enhancement model;

[0081] Step S103, performing encoding processing on the original voice data through an encoder to obtain encoded voice features;

[0082] Step S104, performing first voice enhancement on the encoded voice features through a first voice enhancement module to obtain first enhanced voice features;

[0083] Step S105, performing second voice enhancement on the first enhanced voice features through a second voice enhancement module to obtain second enhanced voice features;

[0084] In step S106, the first enhanced speech feature, the second enhanced speech feature, and the encoded speech feature are fused by a feature fusion module to obtain a target enhanced speech feature.

[0085] In step S107, the target enhanced speech feature is decoded by a decoder to obtain enhanced speech data.

[0086] The steps S101 to S107 shown in the embodiments of the present application can realize the extraction of the speech feature information in the original speech data by obtaining the original speech data, inputting the original speech data into a preset speech enhancement model, and encoding the original speech data by an encoder to obtain an encoded speech feature. Then, the first enhanced speech feature is obtained by performing first speech enhancement on the encoded speech feature by a first speech enhancement module, the second enhanced speech feature is obtained by performing second speech enhancement on the first enhanced speech feature by a second speech enhancement module, and the target enhanced speech feature is obtained by fusing the first enhanced speech feature, the second enhanced speech feature, and the encoded speech feature by a feature fusion module. The target enhanced speech feature can contain speech enhancement information generated based on the first speech enhancement module and speech enhancement information generated based on the second speech enhancement module, thereby improving the feature richness and feature comprehensiveness of the target enhanced speech feature. Finally, the enhanced speech data is obtained by decoding the target enhanced speech feature by a decoder, which can effectively reduce the noise signal in the original speech data, improve the speech quality of the enhanced speech data, and enable each object in the e-commerce, online transaction, and other financial scenarios to clearly receive the specific content described by another conversation object in the enhanced speech data, thereby improving the conversation efficiency of the speech conversation. In addition, this method can also effectively improve the conversation experience of the object and increase the trust of the object to the staff, which is conducive to improving the transaction rate of business recommendations.

[0087] In step S101 of some embodiments, the original speech data can be conversation data generated in online transactions and other businesses, or speech data generated by an intelligent customer service robot in the digital financial field in a dialogue service with each object. The object is a person who has certain business needs in the insurance business, online shopping business, financial transaction, and other business scenarios.

[0088] When obtaining the original speech data, a network crawler can be used to crawl the speech data on a preset data source to obtain the original speech data, where the preset data source can be an audio collector on the object terminal, etc. The audio data generated in the conversation process can also be collected by using an audio collection module on the object terminal or the intelligent customer service robot to obtain the original speech data. It should be noted that the original speech data can also be collected by other methods, which are not limited to this.

[0089] Since the positions, scenes and environments of the objects are different when the objects are conversing, and various environmental noises or interference sounds generated by other objects exist in various scenes in daily life, the obtained original speech data is often speech data containing noises.

[0090] Please refer to Figure 2 Before step S102 of some embodiments, the speech enhancement method further includes pre-training a speech enhancement model, which can specifically include but is not limited to steps S201 to S206:

[0091] Step S201, obtaining sample speech data;

[0092] Step S202, encoding the sample speech data by using a preset original model to obtain sample encoded features;

[0093] Step S203, performing speech enhancement on the sample encoded features by using the original model to obtain sample enhanced speech features;

[0094] Step S204, decoding the sample enhanced speech features by using the original model to obtain sample enhanced speech data;

[0095] Step S205, performing loss calculation on the original model based on the sample enhanced speech data, the sample speech data and pre-obtained reference speech data to obtain a model loss value;

[0096] Step S206, performing parameter updating on the original model based on the model loss value to obtain a speech enhancement model.

[0097] In step S201 of some embodiments, the sample speech data is speech data containing noises. The specific implementation process of obtaining the sample speech data is similar to the above steps. The difference is that step S201 obtains sample speech data, while step S101 obtains original speech data. To save space, no further description is given.

[0098] In step S202 of some embodiments, the sample speech data is encoded by using the original model to obtain sample encoded features.

[0099] In step S203 of some embodiments, the sample encoded features are enhanced by using the original model to reduce noise feature information in the sample encoded features and enhance non-noise feature information in the sample encoded features, thereby obtaining sample enhanced speech features.

[0100] In step S204 of some embodiments, the sample enhanced speech features are decoded by the original model to convert the sample enhanced speech features from a spectral form to a waveform form to obtain sample enhanced speech data.

[0101] Referring to Figure 3 In some embodiments, step S205 can include, but is not limited to, steps S301 to S303.

[0102] In step S301, a first loss calculation is performed based on the sample enhanced speech data and the sample speech data to obtain a mean square error loss value.

[0103] In step S302, a second loss calculation is performed based on the sample enhanced speech data, the sample speech data, and the reference speech data to obtain a contrast loss value.

[0104] In step S303, a model loss value is obtained based on the mean square error loss value and the contrast loss value.

[0105] In step S301 of some embodiments, the specific process of performing the first loss calculation based on the sample enhanced speech data and the sample speech data to obtain the mean square error loss value can be represented as shown in formula (1).

[0106] L2 = || |g| - |X||| 2 Formula (1)

[0107] Wherein, L2 is the mean square error loss value, |g| is the amplitude of the sample enhanced speech data g, and |X| is the amplitude of the sample speech data X.

[0108] In step S302 of some embodiments, the specific process of performing the second loss calculation based on the sample enhanced speech data, the sample speech data, and the reference speech data to obtain the contrast loss value can be represented as shown in formula (2).

[0109]

[0110] Wherein, L cr is the contrast loss value; f(|g|) is the feature vector of the amplitude of the sample enhanced speech data g, f(|X|) is the feature vector of the amplitude of the sample speech data X, and f(|c|) is the feature vector of the amplitude of the reference speech data c.

[0111] In step S303 of some embodiments, the specific process of obtaining the model loss value based on the mean square error loss value and the contrast loss value can be represented as shown in formula (3).

[0112] L = a*L2 + b*L cr Formula (3)

[0113] Where L is the model loss value, L2 is the mean squared error loss value, and L... cr To compare the loss values, 'a' is the weight of the average error loss value and 'b' is the weight of the comparison loss value. a + b = 1. The specific values ​​of a and b can be set according to actual needs and are not restricted.

[0114] It should be noted that the reference speech data can be directly retrieved from the preset speech database. The reference speech data is speech data without noise signals. The reference speech data and the sample speech data both come from the same sample object, and the reference speech data and the sample speech data represent the same semantic information.

[0115] In step S206 of some embodiments, when updating the parameters of the original model based on the model loss value to obtain the speech enhancement model, the model parameters of the original model are continuously adjusted according to the model loss value and a preset loss threshold. The original model is iteratively trained until the model loss value is less than the loss threshold, at which point the updating of the original model stops, and the model parameters of the original model at this point are taken as the final model parameters. The original model at this point is then used as the speech enhancement model. The speech enhancement model includes an encoder, a speech enhancement network, and a decoder. The speech enhancement network includes a first speech enhancement module, a second speech enhancement module, and a feature fusion module.

[0116] like Figure 4 As shown, the speech enhancement model consists of three parts: an encoder, a speech enhancement network, and a decoder. The encoder comprises five sequentially connected convolutional blocks: the first, second, third, fourth, and fifth convolutional blocks. The original speech data is first input to the first convolutional block, which encodes the data sequentially. The fifth convolutional block outputs the encoded speech features. The decoder comprises five sequentially connected convolutional blocks: the sixth, seventh, eighth, ninth, and tenth convolutional blocks. The target speech enhancement features are first input to the sixth convolutional block, which decodes them sequentially. The tenth convolutional block outputs the enhanced speech data. The speech enhancement network connects the fifth and sixth convolutional blocks respectively. The speech enhancement network enhances the encoded speech features output by the fifth convolutional block and outputs the target enhanced speech features to the sixth convolutional block.

[0117] It should be noted that the encoder and the decoder in the speech enhancement model are connected in a skip connection manner. Specifically, the first convolutional block and the tenth convolutional block are connected in a skip connection manner, the second convolutional block and the ninth convolutional block are connected in a skip connection manner, the third convolutional block and the eighth convolutional block are connected in a skip connection manner, the fourth convolutional block and the seventh convolutional block are connected in a skip connection manner, and the fifth convolutional block and the sixth convolutional block are connected in a skip connection manner. The skip connection between the encoder and the decoder can effectively solve the problem of gradient disappearance, thereby improving the speech enhancement effect and model stability of the speech enhancement model.

[0118] As shown in Figure 5 The speech enhancement network of the speech enhancement model includes a plurality of first speech enhancement modules, a plurality of second speech enhancement modules, and a plurality of feature fusion modules. According to the connection order of the first speech enhancement modules, the second speech enhancement modules, and the feature fusion modules, the speech enhancement network can be divided into a plurality of speech enhancement groups, each of which is composed of a first speech enhancement module, a second speech enhancement module, and a feature fusion module. The encoded speech features are first input into the first speech enhancement module of the first speech enhancement group, and sequentially undergo speech enhancement processing of the first speech enhancement module, the second speech enhancement module, and the feature fusion module of the first speech enhancement group to obtain an output feature. The output feature is taken as the input of the second speech enhancement group, and the last feature fusion module of the last speech enhancement group is taken as the target enhanced speech feature.

[0119] In some specific embodiments, the speech enhancement network can be composed of 6 speech enhancement groups, each of which is composed of a first speech enhancement module, a second speech enhancement module, and a feature fusion module. The network structures of the first speech enhancement module and the second speech enhancement module are the same.

[0120] It should be noted that the first speech enhancement module and the second speech enhancement module of the speech enhancement network are connected in a recursive connection manner. Specifically, when the recursive connection is adopted, each speech enhancement group of the speech enhancement network is connected in sequence. For each speech enhancement group of the speech enhancement network, the first speech enhancement module, the second speech enhancement module, and the feature fusion module are connected in sequence, and the encoded speech features output by the encoder are also taken as the input of the feature fusion module, which are fused with the output of the second speech enhancement module. This manner can realize parameter sharing between the first speech enhancement module and the second speech enhancement module. Compared with the chain stacking manner in the related art, the feature extraction capability of the speech enhancement module can be improved, and the computational complexity can be greatly reduced, thereby improving the real-time performance of the speech enhancement model.

[0121] As shown in Figure 6As shown, in the first speech enhancement module, firstly, the input encoded speech features are received by a convolutional layer, and then the output features of the convolutional layer are taken as inputs of the first branch network and the second branch network, and after speech enhancement processing of the first branch network and the second branch network, the output features of the first branch network and the output features of the second branch network are obtained. Then, the output features of the first branch network and the output features of the second branch network are multiplied to obtain a product result, and the product result is sequentially input into a Relu function, a normalization layer and a convolutional layer to obtain an output result. Finally, the output result is added with the encoded speech features to obtain third enhanced speech features.

[0122] It should be noted that the first branch network is composed of a Relu function, a normalization layer and a dilated convolutional layer, and the second branch network is composed of a Relu function, a normalization layer, a dilated convolutional layer and a sigmoid function. In a specific embodiment, the convolutional kernel of the dilated convolutional layer of the first branch network and the dilated convolutional layer of the second branch network is 5, and in other embodiments, the convolutional kernel of the dilated convolutional layer of the first branch network and the dilated convolutional layer of the second branch network can also be other numerical values. The convolutional kernel of the dilated convolutional layer of the first branch network and the dilated convolutional layer of the second branch network can be equal or not equal, and is not limited.

[0123] It should be noted that since the network structures of the first speech enhancement module and the second speech enhancement module are the same, the specific network structure of the second speech enhancement module is referred to the network structure of the first speech enhancement module, and will not be described in detail.

[0124] Through the above steps S201 to S206, the embodiment of the present application can conveniently train a speech enhancement model with good speech enhancement quality. The encoder and the decoder in the speech enhancement model use the sampling skip connection mode, which can effectively solve the problem of gradient disappearance, make the speech enhancement effect of the speech enhancement model better, and also make the model stability of the speech enhancement model higher. In addition, the speech enhancement network of the speech enhancement model uses the recursive connection mode, which can effectively reduce the calculation amount of the model and improve the real-time performance of the model. At the same time, the idea of contrast learning is introduced in the model training of the embodiment of the present application, so that the sample enhanced speech data is as close as possible to the reference speech data while effectively weakening the noise signal in the sample speech data, so as to improve the speech enhancement ability of the speech enhancement model. The noise signal in the sample speech data is effectively utilized in the model training process, which can improve the utilization rate of the noise signal in the sample speech data. Compared with the model training mode in the related art which does not consider the noise signal, the speech enhancement model trained by this model training mode has better speech enhancement performance.

[0125] In step S102 of some embodiments, the original speech data can be automatically input into the preset speech enhancement model by using a preset computer program, thereby improving the data processing efficiency and the automation degree of the speech enhancement process.

[0126] Referring to Figure 7 In some embodiments, the encoder includes a plurality of convolutional blocks, each of which includes a first convolutional layer, a first normalization layer, and a first activation function. Step S103 can include, but is not limited to, steps S701-S703:

[0127] In step S701, the original speech data is feature-extracted by the first convolutional layer to obtain preliminary speech features.

[0128] In step S702, the preliminary speech features are normalized by the first normalization layer to obtain preliminary normalized features.

[0129] In step S703, the preliminary normalized features are activated by the first activation function to obtain encoded speech features.

[0130] In step S701 of some embodiments, the original speech data is feature-extracted by the first convolutional layer to extract speech feature information in the original speech data, thereby obtaining preliminary speech features.

[0131] In step S702 of some embodiments, the preliminary speech features are normalized by the first normalization layer to make the preliminary speech features satisfy a preset feature distribution requirement, thereby obtaining preliminary normalized features. This approach can effectively solve the problem of feature overfitting.

[0132] In step S703 of some embodiments, the preliminary normalized features are activated by the first activation function to realize the de-linearization of the preliminary normalized features, thereby obtaining non-linear encoded speech features.

[0133] As shown in Figure 4 The encoder includes five convolutional blocks, each of which includes a first convolutional layer, a first normalization layer, and a first activation function. The original speech data is sequentially subjected to encoding processing by the first convolutional block, the second convolutional block, the third convolutional block, the fourth convolutional block, and the fifth convolutional block in the encoder, and finally, the encoded speech features are output by the fifth convolutional block.

[0134] Through steps S701-S703 described above, the embodiments of the present application can conveniently extract speech feature information in the original speech data, normalize and de-linearize the speech feature information, and obtain encoded speech features with good feature quality.

[0135] Referring to Figure 8In some embodiments, the first speech enhancement module comprises a first branch network and a second branch network, and step S104 can comprise but is not limited to steps S801-S805:

[0136] Step S801: performing dimension reduction processing on the encoded speech feature to obtain a low-dimensional speech feature.

[0137] Step S802: performing speech enhancement on the low-dimensional speech feature through the first branch network to obtain a third enhanced speech feature.

[0138] Step S803: performing speech enhancement on the low-dimensional speech feature through the second branch network to obtain a fourth enhanced speech feature.

[0139] Step S804: performing splicing processing on the third enhanced speech feature and the fourth enhanced speech feature to obtain a spliced speech feature.

[0140] Step S805: performing dimension elevation processing on the spliced speech feature to obtain the first enhanced speech feature.

[0141] In step S801 of some embodiments, the convolutional layer in the first speech enhancement module is first used to perform dimension reduction processing on the encoded speech feature to obtain a low-dimensional speech feature.

[0142] In step S802 of some embodiments, the Relu function of the first branch network is first used to activate the low-dimensional speech feature to obtain a first activated speech feature. Then, the normalization layer of the first branch network is used to normalize the first activated speech feature to obtain a first normalized speech feature. Finally, the first normalized speech feature is convoluted through the dilated convolutional layer of the first branch network to obtain the third enhanced speech feature.

[0143] In step S803 of some embodiments, the Relu function of the second branch network is first used to activate the low-dimensional speech feature to obtain a second activated speech feature. Then, the normalization layer is used to normalize the second activated speech feature to obtain a second normalized speech feature. Then, the second normalized speech feature is convoluted through the dilated convolutional layer of the second branch network to obtain a convoluted speech feature. Finally, the convoluted speech feature is adjusted in feature distribution through the preset function of the second branch network to obtain the fourth enhanced speech feature.

[0144] In step S804 of some embodiments, when the third enhanced speech feature and the fourth enhanced speech feature are spliced, since the third enhanced speech feature and the fourth enhanced speech feature are both matrix feature vectors, the third enhanced speech feature and the fourth enhanced speech feature can be multiplied in matrix to obtain a spliced speech feature, which contains speech enhancement feature information corresponding to two speech enhancement processes.

[0145] In step S805 of some embodiments, the spliced speech feature is first activated by using a Relu function to obtain an activated feature, the activated feature is normalized, and the normalized result is dimensionally upgraded by using a convolution layer to obtain an upgraded speech feature. The upgraded speech feature and the encoded speech feature are added in feature to obtain the first enhanced speech feature. The first enhanced speech feature and the encoded speech feature have the same feature dimension.

[0146] Through steps S801 to S805, speech enhancement can be performed by using the reduced dimension encoded speech feature, and the accuracy of speech enhancement is improved. Further, the low-dimensional speech feature is respectively enhanced by using the first branch network and the second branch network, and the diversity of speech enhancement is improved. The third enhanced speech feature and the fourth enhanced speech feature are integrated into the first enhanced speech feature by using feature splicing, feature activation, feature normalization, and feature dimension upgrading, so that the first enhanced speech feature contains relatively rich speech enhancement feature information, and the feature accuracy and feature richness of the first enhanced speech feature are improved.

[0147] Please refer to Figure 9 In some embodiments, step S802 includes but is not limited to steps S901 to S903:

[0148] In step S901, the low-dimensional speech feature is activated by using the first branch network to obtain a first activated speech feature.

[0149] In step S902, the first activated speech feature is normalized to obtain a first normalized speech feature.

[0150] In step S903, the first normalized speech feature is convoluted by using the atrous convolution layer of the first branch network to obtain the third enhanced speech feature.

[0151] In step S901 of some embodiments, the low-dimensional speech feature is activated by using the Relu function of the first branch network to realize the de-linearization of the low-dimensional speech feature, so as to obtain the non-linear first activated speech feature.

[0152] In step S902 of some embodiments, the first activated speech feature is normalized by a normalization layer of the first branch network, so that the first activated speech feature satisfies a preset feature distribution requirement, and a first normalized speech feature is obtained. This approach can effectively solve the problem of feature overfitting.

[0153] In step S903 of some embodiments, the first normalized speech feature is convoluted by a dilated convolution layer of the first branch network, which can effectively expand the receptive field and capture more multi-scale speech feature information, and a third enhanced speech feature containing more rich speech feature information is obtained.

[0154] Through steps S901 to S903, the low-dimensional speech feature can be enhanced by feature activation, feature normalization, and feature convolution, so as to reduce noise feature information in the sample coding feature and enhance non-noise feature information in the sample coding feature, thereby improving the quality of speech enhancement.

[0155] Please refer to Figure 10 In some embodiments, step S803 can include but is not limited to steps S1001 to S1004:

[0156] In step S1001, the low-dimensional speech feature is activated by the second branch network to obtain a second activated speech feature.

[0157] In step S1002, the second activated speech feature is normalized to obtain a second normalized speech feature.

[0158] In step S1003, the second normalized speech feature is convoluted by a dilated convolution layer of the second branch network to obtain a convoluted speech feature.

[0159] In step S1004, the convoluted speech feature is adjusted in feature distribution by a preset function of the second branch network to obtain a fourth enhanced speech feature.

[0160] In some embodiments, the specific implementation process of steps S1001 to S1003 is similar to the specific implementation process of steps S901 to S903 described above. To save space, no longer tedious.

[0161] In step S1004 of some embodiments, the preset function can be a sigmoid function or the like. Taking the preset function as a sigmoid function as an example, the sigmoid function is used to adjust the feature distribution of the convoluted speech feature in a nonlinear manner, so that the speech feature information in the convoluted speech feature is more uniformly distributed, and a fourth enhanced speech feature is obtained.

[0162] The low-dimensional speech feature can be enhanced by feature activation, feature normalization, feature convolution, etc. through steps S1001 to S1004, so as to reduce the noise feature information in the sample coding feature, enhance the non-noise feature information in the sample coding feature, and improve the speech enhancement quality. Meanwhile, the preset function is introduced to adjust the nonlinear feature distribution of the convolution speech feature, so as to effectively improve the feature distribution accuracy of the fourth enhanced speech feature.

[0163] In step S105 of some embodiments, the first enhanced speech feature is subjected to second speech enhancement by the second speech enhancement module, and the specific implementation process of obtaining the second enhanced speech feature is similar to that of step S104. The difference is that step S105 is based on the second speech enhancement module located after the first speech enhancement module to perform second speech enhancement on the first enhanced speech feature output by the second speech enhancement module to obtain the second enhanced speech feature, while step S104 is to perform first speech enhancement on the coding speech feature output by the encoder to obtain the first enhanced speech feature. For the sake of brevity, no further description is given.

[0164] In step S106 of some embodiments, when the feature fusion module performs feature fusion on the first enhanced speech feature, the second enhanced speech feature, and the coding speech feature, the first enhanced speech feature and the second enhanced speech feature can be subjected to feature splicing or feature fusion to obtain the target enhanced speech feature.

[0165] In step S107 of some embodiments, the decoder includes a plurality of convolution blocks, each of which includes a deconvolution layer, a normalization layer, and an activation function, wherein the activation function can be a Relu function, etc. When the target enhanced speech feature is subjected to decoding processing by the decoder to obtain the enhanced speech data, the target enhanced speech feature is sequentially input into each convolution block of the decoder for decoding processing, the output of the previous convolution block is taken as the input of the next convolution block, and the output of the last convolution block is taken as the enhanced speech data. Taking the first convolution block of the decoder as an example, the target enhanced speech feature is subjected to deconvolution processing by the deconvolution layer of the convolution block to obtain a deconvolution speech feature, the deconvolution speech feature is subjected to normalization processing by the normalization layer to obtain a normalized enhanced speech feature, then the normalized enhanced speech feature is subjected to activation processing based on the Relu function to obtain the output of the first convolution block, and the output of the first convolution block is taken as the input of the second convolution block.

[0166] As Figure 4As shown, the decoder includes five convolutional blocks, each of which includes a deconvolutional layer, a normalization layer, and an activation function. The target enhanced speech feature is sequentially subjected to decoding processing by the sixth convolutional block, the seventh convolutional block, the eighth convolutional block, the ninth convolutional block, and the tenth convolutional block in the decoder, and finally, the enhanced speech data is output by the tenth convolutional block.

[0167] It should be noted that, since the enhanced speech data is formed by performing speech enhancement on the non-noisy speech signal by the speech enhancement model on the original speech data, the noise contained in the enhanced speech data is less than that in the original speech data.

[0168] It should be noted that, since the enhanced speech data is formed by performing speech enhancement on the non-noisy speech signal by the speech enhancement model on the original speech data, the noise contained in the enhanced speech data is less than that in the original speech data. Figure 11 The embodiments of the present application also provide a speech enhancement device, which can implement the speech enhancement method described above, and the device comprises:

[0169] The data acquisition module 1101 is configured to acquire original speech data, wherein the original speech data is speech data containing noise.

[0170] The input module 1102 is configured to input the original speech data to a preset speech enhancement model, wherein the speech enhancement model comprises an encoder, a speech enhancement network, and a decoder, and the speech enhancement network comprises a first speech enhancement module, a second speech enhancement module, and a feature fusion module.

[0171] The encoding module 1103 is configured to perform encoding processing on the original speech data by the encoder to obtain encoded speech features.

[0172] The first enhancement module 1104 is configured to perform first speech enhancement on the encoded speech features by the first speech enhancement module to obtain first enhanced speech features.

[0173] The second enhancement module 1105 is configured to perform second speech enhancement on the first enhanced speech features by the second speech enhancement module to obtain second enhanced speech features.

[0174] The feature fusion module 1106 is configured to perform feature fusion on the first enhanced speech features, the second enhanced speech features, and the encoded speech features by the feature fusion module to obtain target enhanced speech features.

[0175] The decoding module 1107 is configured to perform decoding processing on the target enhanced speech features by the decoder to obtain enhanced speech data, wherein the noise contained in the enhanced speech data is less than that in the original speech data.

[0176] The specific implementation of the speech enhancement device is basically the same as that of the above-described speech enhancement method, and will not be described here again.

[0177] The electronic device provided by the embodiment of the present application comprises a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection communication between the processor and the memory, and the program is executed by the processor to realize the voice enhancement method. The electronic device can be any smart terminal including a tablet computer, a vehicle-mounted computer, etc.

[0178] Please refer to Figure 12 , Figure 12 The hardware structure of the electronic device of another embodiment is illustrated, and the electronic device comprises:

[0179] The processor 1201 can be implemented in the form of a general CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, etc., and is used to execute related programs to realize the technical solutions provided by the embodiments of the present application.

[0180] The memory 1202 can be implemented in the form of a ROM (ReadOnly Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory), etc. The memory 1202 can store an operating system and other application programs, and when the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 1202 and are called and executed by the processor 1201 to execute the voice enhancement method of the embodiments of the present application.

[0181] The input / output interface 1203 is used to realize information input and output.

[0182] The communication interface 1204 is used to realize the communication interaction between the device and other devices, and the communication can be realized by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0183] The bus 1205 is used to transmit information between various components (for example, the processor 1201, the memory 1202, the input / output interface 1203, and the communication interface 1204) of the device.

[0184] The processor 1201, the memory 1202, the input / output interface 1203, and the communication interface 1204 realize the communication connection between each other in the device through the bus 1205.

[0185] The embodiment of the present application further provides a computer readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the voice enhancement method.

[0186] The memory, as a non-transitory computer readable storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0187] The voice enhancement method, the voice enhancement device, the electronic equipment and the computer readable storage medium provided by the embodiments of the present application can realize the extraction of the voice feature information in the original voice data by obtaining the original voice data, inputting the original voice data into a preset voice enhancement model, and performing encoding processing on the original voice data by an encoder to obtain encoded voice features. Then, the first voice enhancement module is used to perform first voice enhancement on the encoded voice features to obtain first enhanced voice features, the second voice enhancement module is used to perform second voice enhancement on the first enhanced voice features to obtain second enhanced voice features, and the feature fusion module is used to perform feature fusion on the first enhanced voice features, the second enhanced voice features and the encoded voice features to obtain target enhanced voice features. The target enhanced voice features can contain voice enhancement information generated based on the first voice enhancement module and voice enhancement information generated based on the second voice enhancement module, and the feature richness and the feature comprehensiveness of the target enhanced voice features are improved. Finally, the decoder is used to perform decoding processing on the target enhanced voice features to obtain enhanced voice data, which can effectively reduce the noise signal in the original voice data and improve the voice quality of the enhanced voice data. In addition, the encoder and the decoder in the voice enhancement model of the embodiments of the present application are connected in a sampling skip connection mode, which can effectively solve the problem of gradient disappearance, make the voice enhancement effect of the voice enhancement model better, and also make the model stability of the voice enhancement model higher. In addition, the voice enhancement network of the voice enhancement model adopts a recursive connection mode, which can effectively reduce the calculation amount of the model and improve the real-time performance of the model. At the same time, the idea of contrast learning is introduced in the model training of the embodiments of the present application, so that the sample enhanced voice data is as close as possible to the reference voice data while effectively reducing the noise signal in the sample voice data, so as to improve the voice enhancement capability of the voice enhancement model. The noise signal in the sample voice data is effectively utilized in the model training process, the utilization rate of the noise signal in the sample voice data can be improved, and the voice enhancement model trained has better voice enhancement performance.

[0188] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0189] Those skilled in the art can understand that, Figures 1-10 The technical solutions shown in the foregoing embodiments do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than those shown in the figures, or combine certain steps or different steps.

[0190] The apparatus embodiments described above are merely exemplary, and the units described as separate units can or can not be physically separate, i.e., can be located in one place, or can be distributed over multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purposes of the embodiments.

[0191] Those skilled in the art can understand that all or some of the steps in the method disclosed above, the functional modules / units in the system and the device can be implemented as software, firmware, hardware and appropriate combinations thereof.

[0192] The terms "first", "second", "third", "fourth" and the like in the description of the application and in the claims of the foregoing drawings, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so construed can be interchanged, such that the embodiments of the application described herein can be carried out in other than the order discussed herein without departing from the scope of the application. Further, the terms "comprise" and "comprising" and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, product or apparatus that comprises a list of steps or units does not necessarily comprise only those steps or units but can include other not expressly listed steps or units.

[0193] It should be understood that in the present application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the relationship between associated objects, which means that there can be three relationships, for example, "A and / or B" can mean that there are three cases: only A, only B, and A and B at the same time, where A and B can be singular or plural. The character " / " generally represents that the associated objects before and after are in an "or" relationship. "At least one of the following" or the like means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0194] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented by other manners. For example, the apparatus embodiments described above are merely illustrative, for example, the division of the above units is merely a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or components shown or discussed can be indirect coupling or communication connection through some interfaces, apparatuses or units, and can be electrical, mechanical or other forms.

[0195] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they can be located in one place or distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0196] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0197] When the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0198] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, but this does not limit the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A speech enhancement method, characterized by, The method comprises: acquiring original voice data, wherein the original voice data is voice data containing noise; inputting the original voice data into a preset voice enhancement model, wherein the voice enhancement model comprises an encoder, a voice enhancement network and a decoder, and the voice enhancement network comprises a first voice enhancement module, a second voice enhancement module and a feature fusion module; encoding the original voice data through the encoder to obtain encoded voice features; performing first voice enhancement on the encoded voice features through the first voice enhancement module to obtain first enhanced voice features; performing second voice enhancement on the first enhanced voice features through the second voice enhancement module to obtain second enhanced voice features; performing feature fusion on the first enhanced voice features, the second enhanced voice features and the encoded voice features through the feature fusion module to obtain target enhanced voice features; decoding the target enhanced voice features through the decoder to obtain enhanced voice data, wherein the enhanced voice data contains less noise than the original voice data.

2. The speech enhancement method of claim 1, wherein, The encoder comprises a first convolutional layer, a first normalization layer and a first activation function, and the encoding of the original voice data through the encoder to obtain encoded voice features comprises: extracting features of the original voice data through the first convolutional layer to obtain preliminary voice features; normalizing the preliminary voice features through the first normalization layer to obtain preliminary normalized features; activating the preliminary normalized features through the first activation function to obtain encoded voice features.

3. The speech enhancement method of claim 1, wherein, The first voice enhancement module comprises a first branch network and a second branch network, and the first voice enhancement on the encoded voice features through the first voice enhancement module to obtain first enhanced voice features comprises: performing dimension reduction processing on the encoded voice features to obtain low-dimensional voice features; performing voice enhancement on the low-dimensional voice features through the first branch network to obtain third enhanced voice features; performing voice enhancement on the low-dimensional voice features through the second branch network to obtain fourth enhanced voice features; performing splicing processing on the third enhanced voice features and the fourth enhanced voice features to obtain spliced voice features; performing dimension increasing processing on the spliced voice features to obtain the first enhanced voice features, wherein the feature dimensions of the first enhanced voice features and the encoded voice features are the same.

4. The speech enhancement method of claim 3, wherein, The voice enhancement on the low-dimensional voice features through the first branch network to obtain third enhanced voice features comprises: activating the low-dimensional voice features through the first branch network to obtain first activated voice features; normalizing the first activated voice features to obtain first normalized voice features; performing convolution processing on the first normalized voice features through a dilated convolutional layer of the first branch network to obtain the third enhanced voice features.

5. The speech enhancement method of claim 3, wherein, The speech enhancement on the low-dimensional speech feature through the second branch network comprises: activating the low-dimensional speech feature through the second branch network to obtain a second activated speech feature; normalizing the second activated speech feature to obtain a second normalized speech feature; performing convolution processing on the second normalized speech feature through a cavity convolution layer of the second branch network to obtain a convolution speech feature; performing feature distribution adjustment on the convolution speech feature through a preset function of the second branch network to obtain the fourth enhanced speech feature.

6. The speech enhancement method according to any one of claims 1 to 5, characterized in that, Before the original speech data is input into the preset speech enhancement model, the method further comprises pre-training the speech enhancement model, specifically comprising: obtaining sample speech data, wherein the sample speech data is speech data containing noise; performing encoding processing on the sample speech data through a preset original model to obtain sample encoding features; performing speech enhancement on the sample encoding features through the original model to obtain sample enhanced speech features; performing decoding processing on the sample enhanced speech features through the original model to obtain sample enhanced speech data; performing loss calculation on the original model based on the sample enhanced speech data, the sample speech data and pre-acquired reference speech data to obtain a model loss value; performing parameter updating on the original model based on the model loss value to obtain the speech enhancement model.

7. The speech enhancement method of claim 6, wherein, The loss calculation on the original model based on the sample enhanced speech data, the sample speech data and pre-acquired reference speech data to obtain a model loss value comprises: performing first loss calculation based on the sample enhanced speech data and the sample speech data to obtain a mean square error loss value; performing second loss calculation based on the sample enhanced speech data, the sample speech data and the reference speech data to obtain a contrast loss value; obtaining the model loss value based on the mean square error loss value and the contrast loss value.

8. A speech enhancement device, characterized by The device comprises: a data acquisition module configured to acquire original speech data, wherein the original speech data is speech data containing noise; an input module configured to input the original speech data into a preset speech enhancement model, wherein the speech enhancement model comprises an encoder, a speech enhancement network and a decoder, and the speech enhancement network comprises a first speech enhancement module, a second speech enhancement module and a feature fusion module; an encoding module configured to perform encoding processing on the original speech data through the encoder to obtain encoding speech features; a first enhancement module configured to perform first speech enhancement on the encoding speech features through the first speech enhancement module to obtain first enhanced speech features; a second enhancement module configured to perform second speech enhancement on the first enhanced speech features through the second speech enhancement module to obtain second enhanced speech features; and a feature fusion module configured to perform feature fusion on the first enhanced speech features and the second enhanced speech features to obtain final enhanced speech features. The feature fusion module is configured to perform feature fusion on the first enhanced speech feature, the second enhanced speech feature and the encoded speech feature to obtain a target enhanced speech feature. The decoding module is configured to perform decoding processing on the target enhanced speech feature by using the decoder to obtain enhanced speech data, wherein the enhanced speech data contains less noise than the original speech data.

9. An electronic device, comprising: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the speech enhancement method in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to implement the speech enhancement method in any one of claims 1 to 7.

Images