Speech enhancement method, speech enhancement device, electronic equipment and storage medium

Abstract

The invention provides a speech enhancement method, a speech enhancement device, electronic equipment and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the steps of obtaining original voice data; inputting the original voice data into a voice enhancement model, and encoding the original voice data through an encoder to obtain encoded voice features; performing first speech enhancement on the coded speech feature through a first speech enhancement module to obtain a first enhanced speech feature; performing second speech enhancement on the first enhanced speech feature through a second speech enhancement module to obtain a second enhanced speech feature; performing feature fusion on the first enhanced speech feature, the second enhanced speech feature and the coded speech feature through a feature fusion module to obtain a target enhanced speech feature; and decoding the target enhanced voice features through a decoder to obtain enhanced voice data. The voice quality of the voice can be improved and enhanced. The method and the device can be used for scenes such as e-commerce and online transaction.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a speech enhancement method, a speech enhancement device, an electronic device and a storage medium. Background Art

[0002] Currently, in various financial scenarios such as e-commerce and online transactions, conversations between objects are often achieved through voice communication.

[0003] Take insurance recommendation as an example. When staff introduce insurance products to the target through online video, online audio and other channels, if the staff's description of the insurance product is directly transmitted to the target, the clarity of the description received by the target will be greatly affected by factors such as the environment in which the staff is located and the quality of the transmission equipment. For example, when the staff is talking to the target in a noisy environment, there will be a lot of environmental noise in the description of the insurance product, and the voice quality of the description will be poor. At this time, the target may not be able to clearly determine the specific content contained in the description, and cannot better judge whether the insurance product meets its own needs, which affects the normal recommendation of the insurance product.

[0004] In the related art, a deep learning-based speech enhancement method is often used to enhance the conversational speech to improve the speech quality. However, the speech enhancement method in the related art often ignores the influence of noise signals, which results in poor speech quality of the enhanced speech. Summary of the invention

[0005] The main purpose of the embodiments of the present application is to propose a speech enhancement method, a speech enhancement device, an electronic device and a storage medium, aiming to improve the speech quality of enhanced speech.

[0006] To achieve the above object, a first aspect of an embodiment of the present application provides a speech enhancement method, the method comprising:

[0007] Acquire original speech data, wherein the original speech data is speech data containing noise;

[0008] Inputting the original speech data into a preset speech enhancement model, wherein the speech enhancement model includes an encoder, a speech enhancement network and a decoder, and the speech enhancement network includes a first speech enhancement module, a second speech enhancement module and a feature fusion module;

[0009] The encoder encodes the original speech data to obtain encoded speech features;

[0010] Performing a first speech enhancement on the encoded speech feature by the first speech enhancement module to obtain a first enhanced speech feature;

[0011] Performing a second speech enhancement on the first enhanced speech feature by the second speech enhancement module to obtain a second enhanced speech feature;

[0012] The first enhanced speech feature, the second enhanced speech feature and the encoded speech feature are subjected to feature fusion by the feature fusion module to obtain a target enhanced speech feature;

[0013] The target enhanced speech feature is decoded by the decoder to obtain enhanced speech data, wherein the enhanced speech data contains less noise than the original speech data.

[0014] In some embodiments, the encoder includes a first convolutional layer, a first normalization layer, and a first activation function, and the encoding process of the original speech data by the encoder to obtain the encoded speech features includes:

[0015] Extracting features from the original speech data through the first convolutional layer to obtain preliminary speech features;

[0016] Normalizing the preliminary speech features by the first normalization layer to obtain preliminary normalized features;

[0017] The preliminary normalized features are activated by using the first activation function to obtain encoded speech features.

[0018] In some embodiments, the first speech enhancement module includes a first branch network and a second branch network, and performing a first speech enhancement on the encoded speech feature by the first speech enhancement module to obtain a first enhanced speech feature includes:

[0019] Performing dimensionality reduction processing on the encoded speech features to obtain low-dimensional speech features;

[0020] Performing speech enhancement on the low-dimensional speech feature through the first branch network to obtain a third enhanced speech feature;

[0021] Performing speech enhancement on the low-dimensional speech feature through the second branch network to obtain a fourth enhanced speech feature;

[0022] Performing concatenation processing on the third enhanced speech feature and the fourth enhanced speech feature to obtain a concatenated speech feature;

[0023] The concatenated speech feature is subjected to dimensionality increase processing to obtain the first enhanced speech feature, wherein the first enhanced speech feature has the same feature dimension as the encoded speech feature.

[0024] In some embodiments, performing speech enhancement on the low-dimensional speech feature through the first branch network to obtain a third enhanced speech feature includes:

[0025] Performing activation processing on the low-dimensional speech feature through the first branch network to obtain a first activated speech feature;

[0026] Normalizing the first activated speech feature to obtain a first normalized speech feature;

[0027] The first normalized speech feature is convolved through the hole convolution layer of the first branch network to obtain the third enhanced speech feature.

[0028] In some embodiments, performing speech enhancement on the low-dimensional speech feature through the second branch network to obtain a fourth enhanced speech feature includes:

[0029] Performing activation processing on the low-dimensional speech feature through the second branch network to obtain a second activated speech feature;

[0030] Normalizing the second activated speech feature to obtain a second normalized speech feature;

[0031] Performing convolution processing on the second normalized speech feature through the hole convolution layer of the second branch network to obtain a convolution speech feature;

[0032] The feature distribution of the convolutional speech feature is adjusted through a preset function of the second branch network to obtain the fourth enhanced speech feature.

[0033] In some embodiments, before inputting the original speech data into a preset speech enhancement model, the method further includes pre-training the speech enhancement model, specifically including:

[0034] Acquire sample speech data, wherein the sample speech data is speech data containing noise;

[0035] The sample speech data is encoded by using a preset original model to obtain sample encoding features;

[0036] Performing speech enhancement on the sample coding feature through the original model to obtain a sample enhanced speech feature;

[0037] Decoding the sample enhanced speech features through the original model to obtain sample enhanced speech data;

[0038] Performing loss calculation on the original model based on the sample enhanced speech data, the sample speech data and the pre-acquired reference speech data to obtain a model loss value;

[0039] The parameters of the original model are updated based on the model loss value to obtain the speech enhancement model.

[0040] In some embodiments, the performing loss calculation on the original model based on the sample enhanced speech data, the sample speech data and the pre-acquired reference speech data to obtain the model loss value includes:

[0041] Perform a first loss calculation based on the sample enhanced speech data and the sample speech data to obtain a mean square error loss value;

[0042] Perform a second loss calculation based on the sample enhanced speech data, the sample speech data, and the reference speech data to obtain a contrast loss value;

[0043] Based on the mean square error loss value and the contrast loss value, the model loss value is obtained.

[0044] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a speech enhancement device, the device comprising:

[0045] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0046] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0047] The speech enhancement method, speech enhancement device, electronic device and storage medium proposed in the present application obtain original speech data; input the original speech data into a preset speech enhancement model; encode the original speech data through an encoder to obtain encoded speech features, which can realize the extraction of speech feature information in the original speech data. Then, the encoded speech features are subjected to first speech enhancement by a first speech enhancement module to obtain first enhanced speech features, and the first enhanced speech features are subjected to second speech enhancement by a second speech enhancement module to obtain second enhanced speech features; and the first enhanced speech features, the second enhanced speech features and the encoded speech features are subjected to feature fusion by a feature fusion module to obtain target enhanced speech features, so that the target enhanced speech features can include speech enhancement information generated based on the first speech enhancement module and speech enhancement information generated based on the second speech enhancement module, thereby improving the feature richness and feature comprehensiveness of the target enhanced speech features. Finally, the target enhanced speech features are decoded and processed through a decoder to obtain enhanced speech data, which can effectively reduce the noise signal in the original speech data and improve the speech quality of the enhanced speech data. In various financial scenarios such as e-commerce and online transactions, each object can more clearly receive the specific content described by another conversation object in the enhanced speech data, thereby improving the conversation efficiency of the voice conversation. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a flow chart of a speech enhancement method provided in an embodiment of the present application;

[0049] Figure 2 Another flow chart of the speech enhancement method provided by an embodiment of the present application;

[0050] Figure 3 yes Figure 2 Flow chart of step S205 in FIG.

[0051] Figure 4 It is a structural diagram of a speech enhancement model provided in an embodiment of the present application;

[0052] Figure 5 yes Figure 4 A schematic diagram of the structure of the speech enhancement network of the speech enhancement model in FIG.

[0053] Figure 6 yes Figure 5 Schematic diagram of the structure of the first speech enhancement module in

[0054] Figure 7 yes Figure 1 Flow chart of step S103 in FIG.

[0055] Figure 8 yes Figure 1 Flow chart of step S104 in FIG.

[0056] Fig. 9 yes Figure 8 Flowchart of step S802 in FIG.

[0057] Fig.10 yes Figure 8 Flow chart of step S803 in FIG.

[0058] Fig.11 is a structural schematic diagram of a speech enhancement device provided in an embodiment of the present application;

[0059] Fig.12 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0061] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0063] First, some nouns involved in this application are analyzed:

[0064] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0065] Natural language processing (NLP): NLP uses computers to process, understand and apply human languages ​​(such as Chinese, English, etc.). NLP is a branch of artificial intelligence and an interdisciplinary subject between computer science and linguistics. It is often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and text understanding. Natural language processing is often used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent recognition, information extraction and filtering, text classification and clustering, public opinion analysis and opinion mining. It involves data mining related to language processing, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.

[0066] Information Extraction (NER): A text processing technology that extracts specified types of entity, relationship, event and other factual information from natural language text and forms structured data output. Information extraction is a technology that extracts specific information from text data. Text data is composed of some specific units, such as sentences, paragraphs, and chapters. Text information is composed of some small specific units, such as characters, words, phrases, sentences, paragraphs, or a combination of these specific units. Extracting noun phrases, names, place names, etc. from text data are all text information extraction. Of course, the information extracted by text information extraction technology can be various types of information.

[0067] Currently, conversations between objects in scenarios such as e-commerce and online transactions are often carried out through voice communication.

[0068] Taking the recommendation of insurance and financial products as an example, when the staff introduces insurance and financial products to the subjects through online video, online audio and other means, if the staff's voice description of the insurance and financial products is directly transmitted to the subjects, the clarity of the voice description received by the subjects will be greatly affected by factors such as the environment in which the staff is located and the quality of the transmission equipment.

[0069] For example, when a staff member is talking to a subject in a noisy environment, there will be a lot of environmental noise in the description of insurance and financial products, and the voice quality of the description will be poor. At this time, the subject may not be able to clearly determine the specific content contained in the description, and cannot better judge whether the insurance and financial products meet their needs, which affects the normal recommendation of insurance and financial products.

[0070] In the related art, a deep learning-based speech enhancement method is often used to enhance the conversational speech to improve the speech quality. However, the speech enhancement method in the related art often ignores the influence of noise signals, which results in poor speech quality of the enhanced speech.

[0071] Based on this, the embodiments of the present application provide a speech enhancement method, a speech enhancement device, an electronic device and a storage medium, aiming to improve the speech quality of enhanced speech.

[0072] The speech enhancement method and device, electronic device and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the speech enhancement method in the embodiments of the present application is described.

[0073] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0074] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0075] The speech enhancement method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The speech enhancement method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the speech enhancement method, etc., but is not limited to the above forms.

[0076] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0077] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to the identity or characteristics of the object, such as object information, object behavior data, object historical data, and object location information, the permission or consent of the object will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the personal information of the object, the separate permission or separate consent of the object will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the separate permission or separate consent of the object, the necessary object-related data for the normal operation of the embodiment of the present application will be obtained.

[0078] Figure 1 is an optional flow chart of the speech enhancement method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S107.

[0079] Step S101, obtaining original voice data;

[0080] Step S102, inputting the original speech data into a preset speech enhancement model;

[0081] Step S103, encoding the original speech data through an encoder to obtain encoded speech features;

[0082] Step S104, performing a first speech enhancement on the encoded speech feature by a first speech enhancement module to obtain a first enhanced speech feature;

[0083] Step S105, performing second speech enhancement on the first enhanced speech feature by a second speech enhancement module to obtain a second enhanced speech feature;

[0084] Step S106, fusing the first enhanced speech feature, the second enhanced speech feature and the encoded speech feature through a feature fusion module to obtain a target enhanced speech feature;

[0085] Step S107: decoding the target enhanced speech features through a decoder to obtain enhanced speech data.

[0086] Steps S101 to S107 shown in the embodiment of the present application are obtained by obtaining original voice data; inputting the original voice data into a preset voice enhancement model; encoding the original voice data through an encoder to obtain encoded voice features, which can realize the extraction of voice feature information in the original voice data. Then, the first voice enhancement module performs a first voice enhancement on the encoded voice features to obtain a first enhanced voice feature, and the second voice enhancement module performs a second voice enhancement on the first enhanced voice feature to obtain a second enhanced voice feature; and the feature fusion module performs feature fusion on the first enhanced voice feature, the second enhanced voice feature and the encoded voice feature to obtain a target enhanced voice feature, which can make the target enhanced voice feature include the voice enhancement information generated based on the first voice enhancement module and the voice enhancement information generated based on the second voice enhancement module, thereby improving the feature richness and comprehensiveness of the target enhanced voice feature. Finally, the decoder decodes the target enhanced voice feature to obtain enhanced voice data, which can effectively reduce the noise signal in the original voice data and improve the voice quality of the enhanced voice data, so that each object in various financial scenarios such as e-commerce and online transactions can clearly receive the specific content described by another conversation object in the enhanced voice data, thereby improving the conversation efficiency of the voice conversation. In addition, this method can effectively improve the subject's conversation experience, increase the subject's trust in the staff, and help improve the transaction rate of business recommendations.

[0087] In step S101 of some embodiments, the original voice data may be conversation data generated in online transactions or the like, or voice data generated by a conversation service between an intelligent customer service robot commonly used in the digital financial field and various objects. Objects are people who have certain business needs in business scenarios such as insurance business, online shopping business, and financial transactions.

[0088] When obtaining the original voice data, a web crawler can be used to crawl the voice data on a preset data source to obtain the original voice data, wherein the preset data source can be an audio collector on the object terminal, etc.; the audio collection module on the object terminal or the intelligent customer service robot can also be used to collect the voice data generated during the conversation to obtain the original voice data. It should be noted that the original voice data can also be collected by other methods, not limited to this.

[0089] Since the positions, scenes, and environments of the objects may be different when they are conversing, and various scenes in daily life may have various environmental noises or interfering sounds produced by other objects, the original voice data obtained is often voice data containing noise.

[0090] See also Figure 2 In some embodiments, before step S102, the speech enhancement method further includes pre-training a speech enhancement model, which may specifically include but is not limited to steps S201 to S206:

[0091] Step S201, obtaining sample voice data;

[0092] Step S202, encoding the sample speech data using a preset original model to obtain sample encoding features;

[0093] Step S203, performing speech enhancement on the sample coding features through the original model to obtain sample enhanced speech features;

[0094] Step S204, decoding the sample enhanced speech features through the original model to obtain sample enhanced speech data;

[0095] Step S205, calculating the loss of the original model based on the sample enhanced speech data, the sample speech data and the pre-acquired reference speech data to obtain a model loss value;

[0096] Step S206, updating the parameters of the original model based on the model loss value to obtain a speech enhancement model.

[0097] In step S201 of some embodiments, the sample speech data is speech data containing noise. The specific implementation process of obtaining the sample speech data is similar to the above steps. The difference is that step S201 obtains the sample speech data, while step S101 obtains the original speech data. To save space, it is not repeated here.

[0098] In step S202 of some embodiments, the sample speech data is encoded using the original model, and speech feature information in the sample speech data is feature encoded to obtain sample encoding features.

[0099] In step S203 of some embodiments, speech enhancement is performed on the sample coding features through the original model to reduce noise feature information in the sample coding features and enhance non-noise feature information in the sample coding features to obtain sample enhanced speech features.

[0100] In step S204 of some embodiments, the sample enhanced speech features are decoded by the original model, and the sample enhanced speech features are converted from a spectrum form to a waveform form to obtain sample enhanced speech data.

[0101] See also Figure 3 In some embodiments, step S205 may include but is not limited to steps S301 to S303:

[0102] Step S301, performing a first loss calculation based on the sample enhanced speech data and the sample speech data to obtain a mean square error loss value;

[0103] Step S302, performing a second loss calculation based on the sample enhanced speech data, the sample speech data, and the reference speech data to obtain a contrast loss value;

[0104] Step S303: Obtain a model loss value based on the mean square error loss value and the contrast loss value.

[0105] In step S301 of some embodiments, a specific process of performing a first loss calculation based on the sample enhanced speech data and the sample speech data to obtain a mean square error loss value can be expressed as shown in formula (1).

[0106] L2=|||g|-|X||| 2 Formula (1)

[0107] Wherein, L2 is the mean square error loss value, |g| is the amplitude of the sample enhanced speech data g, and |X| is the amplitude of the sample speech data X.

[0108] In step S302 of some embodiments, the specific process of performing a second loss calculation based on the sample enhanced speech data, the sample speech data, and the reference speech data to obtain the contrast loss value can be expressed as shown in formula (2).

[0109]

[0110] Among them, L cr is the contrast loss value; f(|g|) is the feature vector of the amplitude of the sample enhanced speech data g, f(|X|) is the feature vector of the amplitude of the sample speech data X, and f(|c|) is the feature vector of the amplitude of the reference speech data c.

[0111] In step S303 of some embodiments, the specific process of obtaining the model loss value based on the mean square error loss value and the contrast loss value can be expressed as shown in formula (3).

[0112] L=a*L2+b*L cr Formula (3)

[0113] Among them, L is the model loss value, L2 is the mean square error loss value, and L cr is the comparison loss value, a is the weight of the average error loss value, b is the weight of the comparison loss value, a+b=1, and the specific values ​​of a and b can be set according to actual needs without restriction.

[0114] It should be noted that the reference voice data can be directly called from a preset voice database. The reference voice data is voice data without noise signals. The reference voice data and the sample voice data are derived from the same sample object, and the reference voice data and the sample voice data represent the same semantic information.

[0115] In step S206 of some embodiments, when the parameters of the original model are updated based on the model loss value to obtain the speech enhancement model, the model parameters of the original model are continuously adjusted according to the model loss value and the size of the preset loss threshold, and the original model is iteratively trained until the model loss value is less than the loss threshold, and the updating of the original model is stopped, and the model parameters of the original model at this time are used as the final model parameters, and the original model at this time is used as the speech enhancement model. The speech enhancement model includes an encoder, a speech enhancement network and a decoder, and the speech enhancement network includes a first speech enhancement module, a second speech enhancement module and a feature fusion module.

[0116] like Figure 4 As shown, the speech enhancement model includes three parts: an encoder, a speech enhancement network, and a decoder. The encoder includes five sequentially connected convolution blocks, namely, a first convolution block, a second convolution block, a third convolution block, a fourth convolution block, and a fifth convolution block, wherein the original speech data is first input into the first convolution block, and the first convolution block, the second convolution block, the third convolution block, the fourth convolution block, and the fifth convolution block sequentially encode the original speech data, and finally the fifth convolution block outputs the encoded speech features. The decoder includes five sequentially connected convolution blocks, namely, a sixth convolution block, a seventh convolution block, an eighth convolution block, a ninth convolution block, and a tenth convolution block, wherein the target enhanced speech features are first input into the sixth convolution block, and the sixth convolution block, the seventh convolution block, the eighth convolution block, the ninth convolution block, and the tenth convolution block sequentially decode the target enhanced speech features, and finally the tenth convolution block outputs the enhanced speech data. The speech enhancement network is connected to the fifth convolution block and the sixth convolution block respectively. The speech enhancement network performs speech enhancement on the encoded speech features output by the fifth convolution block and outputs the target enhanced speech features to the sixth convolution block.

[0117] It should be noted that the encoder and decoder in the speech enhancement model are connected by jump connection. Specifically, the first convolution block and the tenth convolution block are jump connected, the second convolution block and the ninth convolution block are jump connected, the third convolution block and the eighth convolution block are jump connected, the fourth convolution block and the seventh convolution block are jump connected, and the fifth convolution block and the sixth convolution block are jump connected. The jump connection between the encoder and the decoder can effectively solve the problem of gradient disappearance, thereby improving the speech enhancement effect and model stability of the speech enhancement model.

[0118] like Figure 5 As shown, the speech enhancement network of the speech enhancement model includes multiple first speech enhancement modules, multiple second speech enhancement modules and multiple feature fusion modules, wherein the speech enhancement network can be divided into multiple speech enhancement groups according to the connection sequence of the first speech enhancement module, the second speech enhancement module and the feature fusion module, and each speech enhancement group is composed of a first speech enhancement module, a second speech enhancement module and a feature fusion module. The encoded speech feature will first be input into the first speech enhancement module of the first speech enhancement group, and then be processed by the first speech enhancement module, the second speech enhancement module and the feature fusion module of the first speech enhancement group in sequence to obtain an output feature. This output feature is used as the input of the second speech enhancement group, and so on, and the output of the feature fusion module of the last speech enhancement group is used as the target enhanced speech feature.

[0119] In some specific embodiments, the speech enhancement network may be composed of 6 speech enhancement groups, each of which is composed of a first speech enhancement module, a second speech enhancement module, and a feature fusion module, wherein the network structures of the first speech enhancement module and the second speech enhancement module are the same.

[0120] It should be noted that the first speech enhancement module and the second speech enhancement module of the speech enhancement network are connected in a recursive connection manner. Specifically, when recursive connection is adopted, each speech enhancement group of the speech enhancement network is connected in sequence. For each speech enhancement group of the speech enhancement network, the first speech enhancement module, the second speech enhancement module, and the feature fusion module are connected in sequence. At the same time, the encoded speech features output by the encoder will also be used as the input of the feature fusion module, and the features will be fused together with the output of the second speech enhancement module. This method can realize parameter sharing between the first speech enhancement module and the second speech enhancement module. Compared with the chain stacking method used in the related art, it can improve the feature extraction capability of the speech enhancement module, and can also greatly reduce the amount of calculation, thereby improving the real-time performance of the speech enhancement model.

[0121] like Figure 6As shown, in the first speech enhancement module, a convolution layer first receives the input coded speech features, and then the first branch network and the second branch network take the output features of the convolution layer as input, and after the speech enhancement processing of the first branch network and the second branch network, the output features of the first branch network and the output features of the second branch network are obtained. Next, the output features of the first branch network and the output features of the second branch network are feature multiplied to obtain a product result, and the product result is sequentially input into the Relu function, the normalization layer and the convolution layer to obtain an output result. Finally, the output result is feature added with the coded speech feature to obtain the third enhanced speech feature.

[0122] It should be noted that the first branch network is composed of a Relu function, a normalization layer, and a dilated convolution layer, and the second branch network is composed of a Relu function, a normalization layer, a dilated convolution layer, and a sigmoid function. In a specific embodiment, the convolution kernels of the dilated convolution layer of the first branch network and the dilated convolution layer of the second branch network are both 5. In other embodiments, the convolution kernels of the dilated convolution layer of the first branch network and the dilated convolution layer of the second branch network can also be other values. The convolution kernels of the dilated convolution layer of the first branch network and the dilated convolution layer of the second branch network can be equal or unequal without limitation.

[0123] It should be noted that, since the network structures of the first voice enhancement module and the second voice enhancement module are the same, the specific network structure of the second voice enhancement module refers to the network structure of the first voice enhancement module and will not be repeated here.

[0124] Through the above steps S201 to S206, the embodiment of the present application can be trained to obtain a speech enhancement model with good speech enhancement quality more conveniently. The sampling jump connection mode of the encoder and decoder in the speech enhancement model can effectively solve the problem of gradient disappearance, so that the speech enhancement effect of the speech enhancement model is better, and the model stability of the speech enhancement model is also higher. In addition, the speech enhancement network of the speech enhancement model adopts a recursive connection mode, which can effectively reduce the calculation amount of the model and improve the real-time performance of the model. At the same time, the embodiment of the present application also introduces the idea of ​​contrastive learning during model training, so that the sample enhanced speech data is as close to the reference speech data as possible, and the noise signal in the sample speech data is effectively weakened to improve the speech enhancement ability of the speech enhancement model. The noise signal in the sample speech data is effectively utilized in the model training process, and the utilization rate of the noise signal in the sample speech data can be improved. Compared with the model training mode that does not consider the noise signal in the related art, this model training mode can make the trained speech enhancement model have better speech enhancement performance.

[0125] In step S102 of some embodiments, a preset computer program may be used to automatically input the original speech data into a preset speech enhancement model, thereby improving data processing efficiency and the degree of automation of the speech enhancement process.

[0126] See also Figure 7 In some embodiments, the encoder includes a plurality of convolution blocks, each convolution block includes a first convolution layer, a first normalization layer, and a first activation function, and step S103 may include but is not limited to steps S701 to S703:

[0127] Step S701, extracting features from the original speech data through the first convolutional layer to obtain preliminary speech features;

[0128] Step S702, normalizing the preliminary speech features through a first normalization layer to obtain preliminary normalized features;

[0129] Step S703: Activate the preliminary normalized features through a first activation function to obtain encoded speech features.

[0130] In step S701 of some embodiments, feature extraction is performed on the original speech data through the first convolutional layer to extract speech feature information in the original speech data to obtain preliminary speech features.

[0131] In step S702 of some embodiments, the preliminary speech features are normalized by the first normalization layer so that the preliminary speech features meet the preset feature distribution requirements, thereby obtaining preliminary normalized features. This method can effectively solve the problem of feature overfitting.

[0132] In step S703 of some embodiments, the preliminary normalized features are activated by a first activation function to achieve delinearization of the preliminary normalized features, thereby obtaining nonlinear encoded speech features.

[0133] like Figure 4 As shown in the figure, the encoder includes five convolution blocks, each of which includes a first convolution layer, a first normalization layer, and a first activation function. The original speech data will be encoded by the first convolution block, the second convolution block, the third convolution block, the fourth convolution block, and the fifth convolution block in the encoder in sequence, and finally, the fifth convolution block will output the encoded speech features.

[0134] Through the above steps S701 to S703, the embodiment of the present application can more conveniently extract the speech feature information in the original speech data, and normalize and delinearize the speech feature information to obtain encoded speech features with better feature quality.

[0135] See also Figure 8In some embodiments, the first voice enhancement module includes a first branch network and a second branch network, and step S104 may include but is not limited to steps S801 to S805:

[0136] Step S801, performing dimensionality reduction processing on the encoded speech features to obtain low-dimensional speech features;

[0137] Step S802, performing speech enhancement on the low-dimensional speech feature through the first branch network to obtain a third enhanced speech feature;

[0138] Step S803, performing speech enhancement on the low-dimensional speech feature through the second branch network to obtain a fourth enhanced speech feature;

[0139] Step S804, concatenating the third enhanced speech feature and the fourth enhanced speech feature to obtain a concatenated speech feature;

[0140] Step S805: Perform dimensionality upgrading processing on the concatenated speech features to obtain first enhanced speech features.

[0141] In step S801 of some embodiments, the dimensionality reduction process is first performed on the encoded speech features using the convolutional layer in the first speech enhancement module to obtain low-dimensional speech features.

[0142] In step S802 of some embodiments, the low-dimensional speech feature is first activated by the Relu function of the first branch network to obtain a first activated speech feature. Then, the first activated speech feature is normalized by the normalization layer of the first branch network to obtain a first normalized speech feature. Finally, the first normalized speech feature is convolved by the hole convolution layer of the first branch network to obtain a third enhanced speech feature.

[0143] In step S803 of some embodiments, the low-dimensional speech feature is first activated by the Relu function of the second branch network to obtain a second activated speech feature. Then, the normalization layer of the second branch network performs normalization processing on the second activated speech feature to obtain a second normalized speech feature. Then, the second normalized speech feature is convolved by the hole convolution layer of the second branch network to obtain a convolution speech feature. Finally, the convolution speech feature is feature-distributed by the preset function of the second branch network to obtain a fourth enhanced speech feature.

[0144] In step S804 of some embodiments, when the third enhanced speech feature and the fourth enhanced speech feature are concatenated, since the third enhanced speech feature and the fourth enhanced speech feature are both matrix feature vectors, the third enhanced speech feature and the fourth enhanced speech feature can be matrix multiplied to obtain a concatenated speech feature, which contains speech enhancement feature information corresponding to the two speech enhancement processing processes.

[0145] In step S805 of some embodiments, the concatenated speech features are first activated using a Relu function to obtain activated features, the activated features are normalized, and the normalized results are dimensionally increased using a convolutional layer to obtain dimensionally increased speech features, and the dimensionally increased speech features and the encoded speech features are feature added to obtain a first enhanced speech feature. The first enhanced speech feature and the encoded speech feature have the same feature dimension.

[0146] Through steps S801 to S805, the coded speech features after dimensionality reduction can be used for speech enhancement, thereby improving the accuracy of speech enhancement. Furthermore, the first branch network and the second branch network are used to perform speech enhancement on low-dimensional speech features respectively, thereby improving the diversity of speech enhancement. By integrating the third enhanced speech feature and the fourth enhanced speech feature into the first enhanced speech feature by means of feature concatenation, feature activation, feature normalization, and feature dimensionality increase, the first enhanced speech feature can contain relatively rich speech enhancement feature information, thereby improving the feature accuracy and feature richness of the first enhanced speech feature.

[0147] See also Fig. 9 In some embodiments, step S802 includes but is not limited to steps S901 to S903:

[0148] Step S901, activating the low-dimensional speech feature through the first branch network to obtain a first activated speech feature;

[0149] Step S902, normalizing the first activated speech feature to obtain a first normalized speech feature;

[0150] Step S903, performing convolution processing on the first normalized speech feature through the hole convolution layer of the first branch network to obtain a third enhanced speech feature.

[0151] In step S901 of some embodiments, the low-dimensional speech feature is activated by the Relu function of the first branch network to achieve delinearization of the low-dimensional speech feature, thereby making a nonlinear first activated speech feature.

[0152] In step S902 of some embodiments, the first activated speech feature is normalized by the normalization layer of the first branch network so that the first activated speech feature meets the preset feature distribution requirements, thereby obtaining a first normalized speech feature. This method can effectively solve the problem of feature overfitting.

[0153] In step S903 of some embodiments, convolution processing is performed on the first normalized speech feature through the hole convolution layer of the first branch network, which can effectively expand the receptive field, capture more multi-scale speech feature information, and obtain a third enhanced speech feature containing richer speech feature information.

[0154] Through steps S901 to S903, low-dimensional speech features can be enhanced by using feature activation, feature normalization, feature convolution, etc., reducing noise feature information in sample coding features and enhancing non-noise feature information in sample coding features, thereby improving the quality of speech enhancement.

[0155] See also Fig.10 In some embodiments, step S803 may include but is not limited to steps S1001 to S1004:

[0156] Step S1001, activating the low-dimensional speech feature through the second branch network to obtain a second activated speech feature;

[0157] Step S1002, normalizing the second activated speech feature to obtain a second normalized speech feature;

[0158] Step S1003, performing convolution processing on the second normalized speech feature through the hole convolution layer of the second branch network to obtain a convolution speech feature;

[0159] Step S1004, adjusting the feature distribution of the convolution speech feature through a preset function of the second branch network to obtain a fourth enhanced speech feature.

[0160] In some embodiments, the specific implementation process of step S1001 to step S1003 is similar to the specific implementation process of step S901 to step S903 described above, which will not be described in detail to save space.

[0161] In step S1004 of some embodiments, the preset function may be a sigmoid function, etc. Taking the preset function being a sigmoid function as an example, the sigmoid function is used to perform nonlinear feature distribution adjustment on the convolution speech feature, so that the speech feature information in the convolution speech feature is more evenly distributed, and the fourth enhanced speech feature is obtained.

[0162] Through steps S1001 to S1004, the low-dimensional speech features can be enhanced by means of feature activation, feature normalization, feature convolution, etc., thereby reducing the noise feature information in the sample coding features, enhancing the non-noise feature information in the sample coding features, and improving the quality of speech enhancement. At the same time, this method also introduces a preset function to perform nonlinear feature distribution adjustment on the convolution speech features, which can effectively improve the feature distribution accuracy of the fourth enhanced speech feature.

[0163] In step S105 of some embodiments, the specific implementation process of performing second speech enhancement on the first enhanced speech feature by the second speech enhancement module to obtain the second enhanced speech feature is similar to the above step S104. The difference is that step S105 is based on the second speech enhancement module located after the first speech enhancement module to perform second speech enhancement on the first enhanced speech feature output by the second speech enhancement module to obtain the second enhanced speech feature, while step S104 is to perform first speech enhancement on the encoded speech feature output by the encoder to obtain the first enhanced speech feature. In order to save space, it will not be repeated.

[0164] In step S106 of some embodiments, when the first enhanced speech feature, the second enhanced speech feature and the encoded speech feature are feature fused by the feature fusion module, the first enhanced speech feature and the second enhanced speech feature can be feature concatenated or feature fused to obtain the target enhanced speech feature.

[0165] In step S107 of some embodiments, the decoder includes several convolution blocks, each of which includes a deconvolution layer, a normalization layer, and an activation function, wherein the activation function may be a Relu function, etc. When the target enhanced speech feature is decoded by the decoder to obtain enhanced speech data, the target enhanced speech feature is sequentially input into each convolution block of the decoder for decoding, the output of the previous convolution block is used as the input of the next convolution block, and the output of the last convolution block is used as the enhanced speech data. Taking the first convolution block of the decoder as an example, the target enhanced speech feature is deconvoluted by the deconvolution layer of the convolution block to obtain a deconvolution speech feature, and then the deconvolution speech feature is normalized by the normalization layer to obtain a normalized enhanced speech feature, and then the normalized enhanced speech feature is activated based on the Relu function to obtain the output of the first convolution block, and the output of the first convolution block is used as the input of the second convolution block.

[0166] like Figure 4As shown in the figure, the decoder contains five convolution blocks, each of which includes a deconvolution layer, a normalization layer, and an activation function. The target enhanced speech features will be decoded by the sixth convolution block, the seventh convolution block, the eighth convolution block, the ninth convolution block, and the tenth convolution block in the decoder. Finally, the tenth convolution block outputs the enhanced speech data.

[0167] It should be noted that, since the enhanced speech data is formed by performing speech enhancement on the non-noise speech signal through the speech enhancement model on the original speech data, the noise contained in the enhanced speech data is less than that in the original speech data.

[0168] See also Fig.11 The embodiment of the present application also provides a speech enhancement device, which can implement the above-mentioned speech enhancement method, and the device includes:

[0169] The data acquisition module 1101 is used to acquire original speech data, wherein the original speech data is speech data containing noise;

[0170] An input module 1102 is used to input the original speech data into a preset speech enhancement model, wherein the speech enhancement model includes an encoder, a speech enhancement network and a decoder, and the speech enhancement network includes a first speech enhancement module, a second speech enhancement module and a feature fusion module;

[0171] The encoding module 1103 is used to encode the original speech data through an encoder to obtain encoded speech features;

[0172] A first enhancement module 1104, configured to perform a first speech enhancement on the encoded speech feature through a first speech enhancement module to obtain a first enhanced speech feature;

[0173] The second enhancement module 1105 is used to perform a second speech enhancement on the first enhanced speech feature through a second speech enhancement module to obtain a second enhanced speech feature;

[0174] A feature fusion module 1106 is used to perform feature fusion on the first enhanced speech feature, the second enhanced speech feature and the encoded speech feature through a feature fusion module to obtain a target enhanced speech feature;

[0175] The decoding module 1107 is used to decode the target enhanced speech feature through a decoder to obtain enhanced speech data, wherein the enhanced speech data contains less noise than the original speech data.

[0176] The specific implementation of the speech enhancement device is substantially the same as the specific implementation of the speech enhancement method described above, and will not be described in detail herein.

[0177] The embodiment of the present application also provides an electronic device, the electronic device comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program is executed by the processor to realize the above-mentioned voice enhancement method. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0178] See also Fig.12 , Fig.12 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0179] The processor 1201 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0180] The memory 1202 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1202 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1202, and the processor 1201 calls and executes the speech enhancement method of the embodiment of this application;

[0181] Input / output interface 1203, used to implement information input and output;

[0182] The communication interface 1204 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0183] A bus 1205 that transmits information between the various components of the device (e.g., the processor 1201, the memory 1202, the input / output interface 1203, and the communication interface 1204);

[0184] The processor 1201 , the memory 1202 , the input / output interface 1203 and the communication interface 1204 are connected to each other in communication within the device via the bus 1205 .

[0185] An embodiment of the present application also provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the above-mentioned speech enhancement method.

[0186] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0187] The speech enhancement method, speech enhancement device, electronic device and computer-readable storage medium provided by the embodiment of the present application obtain original speech data; input the original speech data into a preset speech enhancement model; encode the original speech data through an encoder to obtain encoded speech features, which can realize the extraction of speech feature information in the original speech data. Then, the encoded speech features are subjected to first speech enhancement by a first speech enhancement module to obtain a first enhanced speech feature, and the first enhanced speech features are subjected to second speech enhancement by a second speech enhancement module to obtain a second enhanced speech feature; and the first enhanced speech features, the second enhanced speech features and the encoded speech features are subjected to feature fusion by a feature fusion module to obtain a target enhanced speech feature, so that the target enhanced speech feature can contain speech enhancement information generated based on the first speech enhancement module and speech enhancement information generated based on the second speech enhancement module, thereby improving the feature richness and comprehensiveness of the target enhanced speech feature. Finally, the target enhanced speech features are decoded by a decoder to obtain enhanced speech data, which can effectively reduce the noise signal in the original speech data and improve the speech quality of the enhanced speech data. In addition, the sampling jump connection method of the encoder and decoder in the speech enhancement model of the embodiment of the present disclosure can effectively solve the problem of gradient disappearance, so that the speech enhancement effect of the speech enhancement model is better, and the model stability of the speech enhancement model is also higher. In addition, the speech enhancement network of the speech enhancement model adopts a recursive connection method, which can effectively reduce the calculation amount of the model and improve the real-time performance of the model. At the same time, the embodiment of the present application also introduces the idea of ​​contrastive learning during model training, so that the sample enhanced speech data can be as close to the reference speech data as possible, while effectively reducing the noise signal in the sample speech data to improve the speech enhancement ability of the speech enhancement model. The noise signal in the sample speech data is effectively utilized in the model training process, which can improve the utilization rate of the noise signal in the sample speech data, and enable the trained speech enhancement model to have better speech enhancement performance.

[0188] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0189] It can be understood by those skilled in the art that Figure 1-10 The technical solutions shown in the figure do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown in the figure, or a combination of certain steps, or different steps.

[0190] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0191] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0192] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0193] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0194] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0195] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0196] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0197] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0198] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A speech enhancement method, characterized in that: The method comprises: Acquire original speech data, wherein the original speech data is speech data containing noise; Inputting the original speech data into a preset speech enhancement model, wherein the speech enhancement model includes an encoder, a speech enhancement network and a decoder, and the speech enhancement network includes a first speech enhancement module, a second speech enhancement module and a feature fusion module; The encoder encodes the original speech data to obtain encoded speech features; Performing a first speech enhancement on the encoded speech feature by the first speech enhancement module to obtain a first enhanced speech feature; Performing a second speech enhancement on the first enhanced speech feature by the second speech enhancement module to obtain a second enhanced speech feature; The feature fusion module performs feature fusion on the first enhanced speech feature, the second enhanced speech feature and the encoded speech feature to obtain a target enhanced speech feature; The target enhanced speech feature is decoded by the decoder to obtain enhanced speech data, wherein the enhanced speech data contains less noise than the original speech data.

2. The speech enhancement method according to claim 1, characterized in that: The encoder includes a first convolutional layer, a first normalization layer and a first activation function, and the encoding process of the original speech data by the encoder to obtain the encoded speech features includes: Extracting features from the original speech data through the first convolutional layer to obtain preliminary speech features; Normalizing the preliminary speech features by the first normalization layer to obtain preliminary normalized features; The preliminary normalized features are activated by the first activation function to obtain encoded speech features.

3. The speech enhancement method according to claim 1, characterized in that: The first speech enhancement module includes a first branch network and a second branch network, and performing a first speech enhancement on the encoded speech feature by the first speech enhancement module to obtain a first enhanced speech feature includes: Performing dimensionality reduction processing on the encoded speech features to obtain low-dimensional speech features; Performing speech enhancement on the low-dimensional speech feature through the first branch network to obtain a third enhanced speech feature; Performing speech enhancement on the low-dimensional speech feature through the second branch network to obtain a fourth enhanced speech feature; Performing concatenation processing on the third enhanced speech feature and the fourth enhanced speech feature to obtain a concatenated speech feature; The concatenated speech feature is subjected to dimensionality increase processing to obtain the first enhanced speech feature, wherein the first enhanced speech feature has the same feature dimension as the encoded speech feature.

4. The speech enhancement method according to claim 3, characterized in that: The step of performing speech enhancement on the low-dimensional speech feature through the first branch network to obtain a third enhanced speech feature includes: Performing activation processing on the low-dimensional speech feature through the first branch network to obtain a first activated speech feature; Normalizing the first activated speech feature to obtain a first normalized speech feature; The first normalized speech feature is convolved through the hole convolution layer of the first branch network to obtain the third enhanced speech feature.

5. The speech enhancement method according to claim 3, characterized in that: The step of performing speech enhancement on the low-dimensional speech feature through the second branch network to obtain a fourth enhanced speech feature includes: Performing activation processing on the low-dimensional speech feature through the second branch network to obtain a second activated speech feature; Normalizing the second activated speech feature to obtain a second normalized speech feature; Performing convolution processing on the second normalized speech feature through the hole convolution layer of the second branch network to obtain a convolution speech feature; The feature distribution of the convolutional speech feature is adjusted through a preset function of the second branch network to obtain the fourth enhanced speech feature.

6. The speech enhancement method according to any one of claims 1 to 5, characterized in that: Before inputting the original speech data into a preset speech enhancement model, the method further includes pre-training the speech enhancement model, specifically including: Acquire sample speech data, wherein the sample speech data is speech data containing noise; The sample speech data is encoded by using a preset original model to obtain sample encoding features; Performing speech enhancement on the sample coding feature through the original model to obtain a sample enhanced speech feature; Decoding the sample enhanced speech features through the original model to obtain sample enhanced speech data; Performing loss calculation on the original model based on the sample enhanced speech data, the sample speech data and the pre-acquired reference speech data to obtain a model loss value; The parameters of the original model are updated based on the model loss value to obtain the speech enhancement model.

7. The speech enhancement method according to claim 6, characterized in that: The step of performing loss calculation on the original model based on the sample enhanced speech data, the sample speech data and the pre-acquired reference speech data to obtain a model loss value includes: Perform a first loss calculation based on the sample enhanced speech data and the sample speech data to obtain a mean square error loss value; Perform a second loss calculation based on the sample enhanced speech data, the sample speech data, and the reference speech data to obtain a contrast loss value; The model loss value is obtained based on the mean square error loss value and the contrast loss value.

8. A speech enhancement device, characterized in that: The device comprises: A data acquisition module, used to acquire original speech data, wherein the original speech data is speech data containing noise; An input module, used for inputting the original speech data into a preset speech enhancement model, wherein the speech enhancement model includes an encoder, a speech enhancement network and a decoder, and the speech enhancement network includes a first speech enhancement module, a second speech enhancement module and a feature fusion module; An encoding module, used for encoding the original speech data through the encoder to obtain encoded speech features; A first enhancement module, configured to perform a first speech enhancement on the encoded speech feature through the first speech enhancement module to obtain a first enhanced speech feature; A second enhancement module, configured to perform a second speech enhancement on the first enhanced speech feature through the second speech enhancement module to obtain a second enhanced speech feature; A feature fusion module, configured to perform feature fusion on the first enhanced speech feature, the second enhanced speech feature and the encoded speech feature through the feature fusion module to obtain a target enhanced speech feature; A decoding module is used to decode the target enhanced speech feature through the decoder to obtain enhanced speech data, wherein the enhanced speech data contains less noise than the original speech data.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the speech enhancement method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the speech enhancement method according to any one of claims 1 to 7 is implemented.

Images