Speech enhancement model training method and apparatus, device, medium and program product

Abstract

The present application relates to the technical field of artificial intelligence, and in particular to a speech enhancement model training method and apparatus, an electronic device and a storage medium. The method comprises: in each iterative training of an initial speech enhancement model, by means of an input layer, extracting a first audio feature of a speech signal to be enhanced; performing band compression on the first audio feature by means of a band compression layer to obtain a second audio feature having undergone dimension reduction; performing feature mapping on the second audio feature in a loop iteration mode by means of a feature mapping layer to obtain a third audio feature, wherein in the loop iteration process, the number of output channels of the feature mapping layer is gradually increased; and inputting the third audio feature into an output layer to obtain estimated gain information, and incorporating real gain information, performing parameter adjustment on the initial speech enhancement model. The present application can ensure the processing effect of a trained model, and reduce the operation complexity to increase the operation speed, thereby meeting the real-time operation requirement, and improving the communication experience.

Description

Training method, device, equipment, medium and program product for speech enhancement model

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on August 17, 2023, with application number 2023110420658 and application name “Training method, device, electronic device and storage medium for speech enhancement model”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of artificial intelligence technology, and in particular to speech enhancement. Background Art

[0003] With the rapid development of deep learning technology, speech enhancement models are widely used in speech denoising scenarios. Using speech enhancement models, we can extract effective speech signals from the original speech signal, thereby suppressing and reducing the interference of noise signals.

[0004] In related technologies, speech enhancement models usually include multi-layer neural networks, such as convolutional neural networks (CNN), long short-term memory networks (LSTM), gated recurrent units (GRU), etc., which makes the speech enhancement models have high computational complexity.

[0005] Therefore, in real-time communication scenarios (such as online meetings, etc.), using speech enhancement models for speech enhancement is difficult to meet real-time computing requirements, resulting in communication delays and affecting the communication experience.

[0006] Summary of the Invention

[0007] The embodiments of the present application provide a training method, device, electronic device, storage medium, and program product for a speech enhancement model, which are used to reduce the computational complexity of the speech enhancement model while ensuring the processing effect of the speech enhancement model, so as to increase the computational speed, thereby meeting real-time computing requirements and improving the communication experience.

[0008] In one aspect, an embodiment of the present application provides a method for training a speech enhancement model, the method comprising:

[0009] Obtaining a training sample set, each training sample including: a sample speech signal and a corresponding noisy speech signal;

[0010] Iteratively training the initial speech enhancement model based on the training sample set to obtain a trained speech enhancement model, wherein the initial speech enhancement model includes an input layer, a frequency band compression layer, a feature mapping layer, and an output layer; wherein each training process includes:

[0011] Through the input layer, feature extraction is performed on the sample speech signal and the noisy speech signal in the selected training sample respectively, and then fusion is performed to obtain a first audio feature;

[0012] The first audio feature is subjected to frequency band compression by the frequency band compression layer to obtain a second audio feature; the feature dimension number of the second audio feature is smaller than the feature dimension number of the first audio feature;

[0013] Performing feature mapping on the second audio feature through the feature mapping layer in a loop iterative manner until the number of iterations reaches a set number of mappings, thereby obtaining a third audio feature; wherein, during the loop iterative process, the number of output channels of the feature mapping layer increases successively;

[0014] The third audio feature is input into the output layer to obtain estimated gain information, and the initial speech enhancement model is adjusted based on the difference between the estimated gain information and the corresponding true gain information.

[0015] In one aspect, an embodiment of the present application provides a method for speech enhancement, the method comprising:

[0016] Inputting the speech signal to be enhanced into the speech enhancement model obtained by the training method of the speech enhancement model to obtain estimated gain information;

[0017] Speech enhancement is performed on the speech signal to be enhanced based on the estimated gain information.

[0018] In one aspect, an embodiment of the present application provides a training device for a speech enhancement model, the device comprising:

[0019] An acquisition unit is used to acquire a training sample set, each training sample including: a sample speech signal and a corresponding noisy speech signal;

[0020] A training unit is configured to iteratively train an initial speech enhancement model based on the training sample set to obtain a trained speech enhancement model, wherein the initial speech enhancement model includes an input layer, a frequency band compression layer, a feature mapping layer, and an output layer; wherein each training process includes:

[0021] Through the input layer, feature extraction is performed on the sample speech signal and the noisy speech signal in the selected training sample respectively, and then fusion is performed to obtain a first audio feature;

[0022] The first audio feature is subjected to frequency band compression by the frequency band compression layer to obtain a second audio feature; the feature dimension number of the second audio feature is smaller than the feature dimension number of the first audio feature;

[0023] Performing feature mapping on the second audio feature through the feature mapping layer in a loop iterative manner until the number of iterations reaches the set number of mappings, thereby obtaining a third audio feature; wherein, during the loop iterative process, the number of output channels of the feature mapping layer increases successively;

[0024] The third audio feature is input into the output layer to obtain estimated gain information, and the initial speech enhancement model is adjusted based on the difference between the estimated gain information and the corresponding true gain information.

[0025] In one aspect, an embodiment of the present application provides a speech enhancement device, comprising:

[0026] An input unit, configured to input the speech signal to be enhanced into the speech enhancement model obtained by the training method of the speech enhancement model to obtain estimated gain information;

[0027] A speech enhancement unit is configured to perform speech enhancement on the speech signal to be enhanced based on the estimated gain information.

[0028] On the one hand, an embodiment of the present application provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of any one of the above-mentioned speech enhancement methods or speech enhancement model training methods.

[0029] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which includes a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to perform the steps of any one of the above-mentioned speech enhancement methods or speech enhancement model training methods.

[0030] On the one hand, an embodiment of the present application provides a computer program product, which includes a computer program, and the computer program is stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device performs the steps of any one of the above-mentioned speech enhancement methods or speech enhancement model training methods.

[0031] The above solution of the embodiment of the present application has at least the following beneficial effects:

[0032] In each round of training of the initial speech enhancement model, the embodiment of the present application first extracts the first audio feature of the training sample. Then, in order to reduce the complexity of subsequent operations, the first audio feature is subjected to frequency band compression through the frequency band compression layer to obtain the second audio feature after dimensionality reduction. The second audio feature is cyclically iteratively input into the feature mapping layer, and the number of output channels of multiple iterations is gradually increased. In this way, the depth of the feature mapping and the number of parameters can be increased without increasing the model structure to improve the training effect of the model. Finally, the estimated gain information is obtained based on the third audio feature output by the feature mapping layer. Therefore, the embodiment of the present application performs feature dimensionality reduction through the frequency band compression layer, which greatly reduces the complexity of subsequent operations. Moreover, the cyclic iterative processing of the feature mapping layer improves the training effect of the model, so that the trained speech enhancement model reduces the computational complexity while ensuring the processing effect, thereby improving the computational speed, thereby meeting the real-time computing requirements and improving the communication experience.

[0033] Other features and advantages of the present application will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present application. The purposes and other advantages of the present application can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] FIG1 is a schematic diagram of an application scenario of a speech enhancement method in an embodiment of the present application;

[0035] FIG2 is a schematic diagram of the structure of an initial speech enhancement model in an embodiment of the present application;

[0036] FIG3 is a flow chart of a method for training a speech enhancement model in an embodiment of the present application;

[0037] FIG4 is a schematic diagram of the structure of an input layer of an initial speech enhancement model in an embodiment of the present application;

[0038] FIG5A is a schematic diagram of the number of output channels of a feature mapping layer of an initial speech enhancement model in an embodiment of the present application;

[0039] FIG5B is a schematic diagram of the structure of a feature mapping layer of an initial speech enhancement model in an embodiment of the present application;

[0040] FIG6 is a schematic diagram of the structure of an output layer of an initial speech enhancement model in an embodiment of the present application;

[0041] FIG7A is a schematic diagram of a training process of a speech enhancement model in an embodiment of the present application;

[0042] FIG7B is a schematic diagram of the training process of another speech enhancement model in an embodiment of the present application;

[0043] FIG8 is a flow chart of a method for speech enhancement according to an embodiment of the present application;

[0044] FIG9 is a logic diagram of a speech enhancement method in an embodiment of the present application;

[0045] FIG10 is a schematic diagram of an original speech signal and an enhanced speech signal in an embodiment of the present application;

[0046] FIG11 is a schematic diagram of the structure of a training device for a speech enhancement model according to an embodiment of the present application;

[0047] FIG12 is a schematic structural diagram of a speech enhancement device according to an embodiment of the present application;

[0048] FIG13 is a schematic diagram of a hardware structure of an electronic device to which an embodiment of the present application is applied;

[0049] FIG14 is a schematic diagram of a hardware structure of another electronic device to which an embodiment of the present application is applied. DETAILED DESCRIPTION

[0050] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of the technical solutions of this application, but not all of them. Based on the embodiments described in this application document, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the technical solutions of this application.

[0051] The following is an introduction to some concepts involved in the embodiments of this application.

[0052] Speech enhancement: When the speech signal is interfered with or even drowned by various noises, the technology is used to extract useful speech signals from the noisy background and suppress and reduce noise interference.

[0053] Convolutional Neural Networks (CNNs) are a type of feedforward neural network with a deep structure that includes convolutional calculations. They consist of convolutional layers, pooling layers, and fully connected layers. Convolutional layers are responsible for extracting local features of input data, pooling layers are used to significantly reduce the number of parameters (dimensionality reduction), and fully connected layers are used to output the desired results.

[0054] The Attention Mechanism (AIM): It originates from the study of human vision. In cognitive science, due to information processing bottlenecks, humans selectively focus on a subset of all information while ignoring other visible information. This mechanism is often referred to as the attention mechanism. The attention mechanism enables neural networks to focus on a subset of their inputs (or features), selecting specific inputs. Given limited computing power, the attention mechanism is a primary means of addressing information overload, a resource allocation solution that allocates computing resources to more important tasks.

[0055] Acoustic Perception Scale: The sensitivity of the human ear to speech varies with frequency, and the relationship between sensitivity and frequency is not a simple linear proportional relationship, but rather an approximately logarithmic relationship. In order to better approximate the sound pickup characteristics of the human ear, the frequency of speech is usually nonlinearly transformed into an acoustic perception scale to extract speech features. Acoustic perception scales include the Equivalent Rectangular Bandwidth (ERB) scale, the Mel scale, the Bark scale, etc. They are all psychoacoustic measurement methods used to describe the nonlinear transformation of the human ear's frequency perception. Psychoacoustics is an interdisciplinary subject that studies the relationship between objective sound parameters (such as frequency, amplitude, phase, etc.) and subjective perception (loudness, pitch, timbre, etc.).

[0056] The word “exemplary” is used hereinafter to mean “serving as an example, example, or illustration.” Any embodiment described as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0057] The terms "first" and "second" are used for descriptive purposes only and should not be construed as explicitly or implicitly indicating relative importance or the number of the technical features indicated. Therefore, features specified as "first" or "second" may explicitly or implicitly include one or more of such features. In the description of the embodiments of this application, unless otherwise specified, "plurality" means two or more.

[0058] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0059] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, interactive systems, and mechatronics. Software technologies typically encompass computer vision, natural language processing, and machine learning / deep learning. With the development and advancement of AI, research and application are being conducted in numerous fields, including smart homes, smart customer service, virtual assistants, smart speakers, smart marketing, driverless and autonomous driving, robotics, and smart healthcare. With further technological advancements, AI will be applied in even more areas, playing an increasingly important role.

[0060] Machine Learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance.

[0061] Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Deep learning, on the other hand, is the core of machine learning and a technology that enables it. Machine learning generally includes techniques such as deep learning, reinforcement learning, transfer learning, and inductive learning. Deep learning includes technologies such as MobileNet, Convolutional Neural Networks (CNNs), Deep Belief Networks, Recurrent Neural Networks, Autoencoders, and Generative Adversarial Networks.

[0062] In an embodiment of the present application, the initial speech enhancement model can be trained based on deep learning technology in machine learning to obtain a trained speech enhancement model, and then the speech enhancement model can be used to perform speech enhancement.

[0063] With the development and progress of artificial intelligence, artificial intelligence is being researched and applied in many fields, such as common smart homes, smart customer service, virtual assistants, smart speakers, smart marketing, unmanned driving, autonomous driving, robots, smart medical care, etc. It is believed that with the further development of future technology, artificial intelligence will be applied in more fields and play an increasingly important role.

[0064] Cloud technology refers to a hosting technology that unifies hardware, software, network and other resources within a wide area network or local area network to achieve data computing, storage, processing and sharing.

[0065] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool that can be used on demand and is flexible and convenient. Cloud computing technology will become an important support. The backend services of technical network systems require a large amount of computing and storage resources, such as video websites, image websites and more portal websites. With the rapid development and application of the Internet industry, each item may have its own identification mark in the future, and all need to be transmitted to the backend system for logical processing. Data of different levels will be processed separately. All types of industry data require strong system backend support, which can only be achieved through cloud computing. In the embodiment of the present application, the training of the speech enhancement model can be achieved through cloud computing.

[0066] The following is a brief introduction to the design concept of the embodiments of this application.

[0067] In related technologies, speech enhancement models typically include multi-layer neural networks, such as convolutional neural networks (CNNs), long short-term memory networks (LSTMs), and gated recurrent units (GRUs). This results in high computational complexity. Consequently, in real-time communication scenarios (such as online conferencing), using these models for speech enhancement struggles to meet real-time computational requirements, leading to communication delays and a negative impact on the communication experience.

[0068] In view of this, the embodiments of the present application provide a training method, device, electronic device and storage medium for a speech enhancement model. When training the initial speech enhancement model, in order to reduce the computational complexity, the feature dimension is reduced by frequency band compression, and the input features are feature mapped in a cyclic iteration manner through the feature mapping layer of the initial speech enhancement model, and the number of output channels of the feature mapping layer is gradually increased. In this way, the depth of the feature mapping can be increased and the number of parameters can be increased without increasing the model structure, thereby ensuring the training effect of the model. Therefore, the speech enhancement model obtained by training in the embodiments of the present application can reduce the computational complexity while ensuring the processing effect, so as to increase the computational speed, thereby meeting the real-time computing requirements and improving the communication experience.

[0069] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application can be combined with each other if there is no conflict.

[0070] As shown in Figure 1, which is a schematic diagram of an application scenario of an embodiment of the present application, the application scenario diagram includes a terminal device 110 and a server 120.

[0071] In the embodiments of the present application, the terminal device 110 includes but is not limited to smart phones, tablet computers, laptop computers, desktop computers, smart speakers, smart watches, e-book readers, intelligent voice interaction devices, smart home appliances, car terminals and other devices; a client related to real-time communication can be installed on the terminal device, which can be software (such as conference software, social software, etc.), or a web page, applet, etc. The server 120 can be a background server corresponding to the software or web page, applet, etc., or a server specifically used for voice enhancement, which is not specifically limited in this application. The server 120 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0072] In an optional implementation, the terminal device 110 and the server 120 may be directly or indirectly connected via wired or wireless communication, which is not limited in this application.

[0073] It should be noted that the speech enhancement method or the training method of the speech enhancement model in each embodiment of the present application can be executed by an electronic device, which can be a terminal device 110 or a server 120, that is, the speech enhancement method can be executed by the terminal device 110 or the server 120, and the training method of the speech enhancement model can also be executed by the terminal device 110 or the server 120.

[0074] In some embodiments, in a real-time communication scenario, the terminal device 120 executes a speech enhancement method and the server 120 executes a speech enhancement model training method as an example for description.

[0075] The server 120 uses the training method of the speech enhancement model of the embodiment of the present application to train the initial speech enhancement model and obtain a trained speech enhancement model. During the process of user A using the terminal device 110 to communicate with the terminal device 110 of user B in real time, when the terminal device 110 of user A sends the voice signal on the user A side to the terminal device 110 of user B through the server 120, the terminal device 110 of user B can use the speech enhancement method of the embodiment of the present application to process the received speech signal using the trained speech enhancement model to obtain estimated gain information, and then perform speech enhancement on the speech signal based on the estimated gain information, that is, perform denoising on the speech signal, and play the enhanced speech signal after obtaining it.

[0076] It should be noted that what is shown in FIG1 is only an example. In fact, the number of terminal devices and servers is not limited and is not specifically limited in the embodiments of the present application.

[0077] The following describes the speech enhancement method and speech enhancement model training method provided by the exemplary embodiment of the present application in combination with the application scenarios described above and with reference to the accompanying drawings. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of the present application, and the implementation of the present application is not limited in this respect.

[0078] In an embodiment of the present application, an initial speech enhancement model is trained to obtain a trained speech enhancement model. As shown in FIG2 , the initial speech enhancement model includes an input layer 201, a frequency band compression layer 202, a feature mapping layer 203, and an output layer 204. The following embodiment introduces the training process of the speech enhancement model.

[0079] As shown in FIG3 , a flow chart of a method for training a speech enhancement model provided in an embodiment of the present application is shown. Taking server execution as an example, the method specifically includes the following steps S31-S32:

[0080] S31, obtaining a training sample set, each training sample including: a sample speech signal and a corresponding noisy speech signal.

[0081] Among them, the sample speech signal and the noisy speech signal in each training sample are both frequency domain signals; in one possible implementation, the sample speech signal is obtained by transforming the original sample speech signal from the time domain to the frequency domain, and the noisy speech signal is obtained by transforming the original noisy speech signal from the time domain to the frequency domain.

[0082] The sample speech signal can be obtained through on-site collection. In order to improve the quality of model training, the collection environment needs to minimize the occurrence of noise, such as background noise. However, this application does not limit the sample speech signal itself to be completely noise-free. For example, it can carry some unavoidable noise.

[0083] The noisy speech signal is obtained by adding noise to the sample speech signal. In one possible implementation, the original noisy speech signal is obtained by adding noise to the corresponding original sample speech signal. The added noise can be common noise in a real environment.

[0084] Optionally, the original sample speech signal and the original noisy speech signal may be ultra-wideband speech signals (with a sampling rate of 32000 Hz and a bandwidth of 16000 Hz). Of course, they may also be ordinary wideband speech signals, which is not limited.

[0085] Specifically, the original sample speech signal can be transformed from the time domain to the frequency domain using discrete Fourier transform, which includes fast Fourier transform (FFT) and the like. For example, the original sample speech signal is subjected to FFT transform to obtain eigenvalues ​​of multiple frequency points. The eigenvalues ​​include the real part and imaginary part of the amplitude, where the real part represents the amplitude of the speech signal and the imaginary part represents how the phase of the speech signal changes with frequency.

[0086] In addition, each training sample also includes true gain information, which can be calculated using the sample speech signal and the noisy speech signal in the training sample. Specifically, the true gain information can be obtained by dividing the energy spectrum of the sample speech signal by the sum of the energy spectra of the sample speech signal and the noisy speech signal. This true gain information can accurately identify the noise added to the noisy speech signal.

[0087] S32, iteratively training the initial speech enhancement model based on the training sample set to obtain a trained speech enhancement model; wherein each training process includes S321-S324.

[0088] S321 , performing feature extraction on the sample speech signal and the corresponding noisy speech signal in the selected training sample through the input layer and then fusing them to obtain a first audio feature.

[0089] In an embodiment of the present application, the input layer of the initial speech enhancement model can utilize a neural network, such as a convolutional neural network, which performs feature extraction on the sample speech signal and the noisy speech signal, and then fuses the extracted features to obtain a first audio feature. Assuming that both the sample speech signal and the noisy speech signal include eigenvalues ​​at 513 frequency points, after feature extraction at the input layer, the feature dimension of the obtained first audio feature can be 513.

[0090] The sample speech signal and the corresponding noisy speech signal used to fuse the first audio feature can be called a speech signal pair. Assume that speech signal pair 1 includes sample speech signal 1 and noisy speech signal 1, and noisy speech signal 1 is obtained by adding noise to sample speech signal 1.

[0091] In an optional embodiment, as shown in Figure 4, the input layer includes two parallel convolutional neural networks and a convolutional neural network connected to the two convolutional neural networks. The sample speech signal and the noisy speech signal are respectively input into the two parallel convolutional neural networks, and the output features of the two convolutional neural networks are spliced ​​and then input into the latter convolutional neural network for feature integration to obtain the first audio feature.

[0092] S322. Perform frequency band compression on the first audio feature through a frequency band compression layer to obtain a second audio feature; wherein the number of feature dimensions of the second audio feature is smaller than the number of feature dimensions of the first audio feature.

[0093] Among them, the frequency band compression layer can divide each frequency of the first audio feature into a set number of frequency bands. For example, the frequency band compression layer can include a bandpass filter. By converting each frequency of the first audio feature into an acoustic perception scale, each frequency can be converted into a set number of frequency bands to obtain a second audio feature containing a set number of frequency bands. Assume that the first audio feature is 513-dimensional, that is, the characteristic values ​​of 513 frequency points, and the second audio feature is obtained by converting the 513 frequency points into a set number of frequency bands, including the characteristic values ​​of a set number of frequency bands, and the characteristic value of each frequency band includes amplitude. The set number is less than 513, for example, 128, then the second audio feature is 128-dimensional. Therefore, the first audio feature can be reduced in dimension through the frequency band compression layer to obtain the second audio feature after dimensionality reduction.

[0094] In some embodiments, performing frequency band compression on the first audio feature to obtain the second audio feature in S32 may include the following steps A1-A2:

[0095] A1. Transform the first audio feature from the frequency domain into the acoustic perception scale domain, and divide the transformed first audio feature into a set number of sub-audio features according to the acoustic perception scale domain.

[0096] The acoustic perception scale may be an ERB scale, a Mel scale, or a Bark scale. Specifically, the lowest frequency and the cutoff frequency of the first audio feature are transformed into the acoustic perception scale through a nonlinear function; for example, taking the acoustic perception scale as the ERB scale as an example, the lowest frequency and the cutoff frequency are transformed into the acoustic perception scale through the nonlinear function shown in the following formula (1) to obtain the lowest ERB scale and the highest ERB scale:

[0097] Assuming the number is set to M, the lowest ERB scale to the highest ERB scale is divided into M equal parts, and the sub-audio features corresponding to the M scale segments are obtained; for example, the lowest ERB scale is 0 and the highest ERB scale is 40, then 0 to 40 are divided into M equally spaced scale segments.

[0098] A2. Inversely transform a set number of sub-audio features from the acoustic perception scale domain to the frequency domain, and filter the inversely transformed sub-audio features of each frequency band through a bandpass filter to obtain a second audio feature containing a set number of frequency bands.

[0099] Among them, the scale segment of each sub-audio feature is inversely transformed from the acoustic perception scale back to the frequency. The inverse transformation process is the inverse process of the transformation process in the above step A1. For example, taking the ERB scale as an example, the inverse transformation process is the inverse process of the above formula (1). Each scale segment is inversely transformed into a frequency band to obtain a second audio feature containing a set number of frequency bands, specifically in the form of a feature vector.

[0100] In an embodiment of the present application, by transforming the first audio feature from the frequency domain to the acoustic perception scale domain, the frequencies of the first audio feature can be divided into a set number of frequency bands in the acoustic perception scale domain to obtain the second audio feature, thereby achieving feature dimensionality reduction. In subsequent steps, the complexity of the calculation can be reduced and the calculation speed can be improved.

[0101] S323, feature mapping is performed on the second audio feature through the feature mapping layer in a loop iterative manner until the number of iterations reaches the set mapping number, thereby obtaining a third audio feature; wherein, during the loop iterative process, the number of output channels of the feature mapping layer increases successively.

[0102] Among them, the number of mappings can be set as needed, for example, 3 times, and there is no limitation on this. In each iteration, the number of output channels of the feature mapping layer can also be set as needed, which can be achieved by setting the network structure of the feature mapping layer. For example, the feature mapping layer includes a convolutional neural network, and its output channel number can be determined according to the number of filters of the convolution layer. The filter can be regarded as a collection of convolution kernels, that is, each filter can include one or more convolution kernels, and each filter can output a feature map. The number of feature maps is the number of output channels. For example, as shown in Figure 5A, assuming that the feature of 1 channel is input, the input feature is convolved through n filters (including 1 convolution kernel) respectively, and each filter outputs a feature map of one channel, thereby obtaining a feature map of n channels, that is, the number of output channels is n. In multiple loop iterations, the number of filters used in each iteration can be increased successively, so that the number of output channels increases successively.

[0103] Optionally, during the loop iteration process, the number of output channels of the feature mapping layer may be increased exponentially. For example, taking three iterations as an example, the number of output channels is 32, 64, and 128, respectively. There is no limitation on this.

[0104] In an embodiment of the present application, the input second audio feature is iteratively processed through the feature mapping layer, and the number of output channels is gradually increased. This can increase the depth of the feature mapping and the number of parameters without increasing the model structure, and can more effectively learn the characteristics of the input signal to ensure the processing effect of the trained model.

[0105] In some embodiments, the feature mapping layer may include at least one layer of a convolutional neural network and an attention mechanism network. S33 in the above embodiment may include the following steps B1-B2:

[0106] The loop iterates and executes the following steps until the number of iterations reaches the set mapping number:

[0107] B1. Input the second audio features into at least one layer of convolutional neural network in sequence for convolution processing to obtain intermediate convolution features; input the intermediate convolution features into the attention mechanism network for feature interaction to obtain new second audio features.

[0108] Among them, the number of convolutional neural network layers of the feature mapping layer can be set as needed, and the feature mapping layer can also include an activation function, such as a PReLU function, a ReLU function, a tanh function, or a sigmoid function, etc. The activation function can be located between at least one layer of convolutional neural network, or can also be located between at least one layer of convolutional neural network and an attention mechanism network. After the second audio feature is sequentially input into at least one layer of convolutional neural network, a multi-channel convolution feature (i.e., an intermediate convolution feature) is obtained. Then, for the convolution feature of each channel, different weights are assigned to different parts of the convolution feature based on the attention mechanism network, so as to focus on important information more accurately, thereby obtaining a new multi-channel convolution feature as a new second audio feature.

[0109] B2. Use the new second audio feature as the third audio feature.

[0110] In an optional embodiment, as shown in Figure 5B, the feature mapping layer includes two convolutional neural networks, an activation function, a convolutional neural network, and an attention mechanism network connected in sequence. If the mapping number is set to 3 times, the second audio feature is input into the two convolutional neural networks, the activation function, the convolutional neural network, and the attention mechanism network in sequence, and the output result of the attention mechanism network is input into the above-mentioned networks again, and so on, until the loop iterates 3 times, and the third audio feature is output. The third audio feature is in the form of a feature vector, specifically including the feature values ​​of a set number of frequency bands.

[0111] In an embodiment of the present application, the feature mapping layer first convolves the input second audio feature, and then performs feature interaction on the convolution feature through the attention mechanism network, which can pay more attention to the important information in the convolution feature, thereby improving the processing performance and efficiency of the speech enhancement model.

[0112] S324: Input the third audio feature into the output layer to obtain estimated gain information, and adjust the parameters of the initial speech enhancement model based on the difference between the estimated gain information and the corresponding actual gain information.

[0113] Wherein, the output layer may include a neural network, for example, a convolutional neural network, and the third audio feature output by the feature mapping layer is input into the convolutional neural network to obtain estimated gain information, which includes the estimated gains of a set number of frequency bands, and the number of dimensions thereof is consistent with the number of feature dimensions of the second audio feature output by the frequency band compression layer, for example, 128 dimensions. Before determining the difference between the estimated gain information and the actual gain information, the estimated gain information may be subjected to frequency band decompression, that is, the inverse process of the frequency band compression in S32 above, so as to upgrade the dimension of the estimated gain information so that its number of dimensions is consistent with the number of feature dimensions of the input noisy speech signal (for example, 513 dimensions), and then the difference between the estimated gain information after dimensional upgrade and the actual gain information is determined, and then the parameters of the initial speech enhancement model are adjusted.

[0114] Since the estimated gain information can reflect the noise recognized by the initial speech enhancement model based on the two input speech signals, and the true gain information accurately identifies the noise actually added to the sample speech signal, the difference between the two can accurately identify the noise recognition error of the initial speech enhancement model, thereby guiding the adjustment of model parameters.

[0115] In some embodiments, adjusting parameters of the initial speech enhancement model based on the difference between the estimated gain information and the corresponding actual gain information in S324 may include the following steps C1-C2:

[0116] C1. Decompressing the estimated gains of a set number of frequency bands to obtain decompressed estimated gain information, where the decompressed estimated gain information includes the estimated gain of each frequency of the noisy speech signal;

[0117] Frequency band decompression can be understood as the inverse process of frequency band compression in S322 in the above-mentioned embodiment of the present application. Optionally, a set number of frequency bands are transformed from the frequency domain to the acoustic perception scale domain to obtain acoustic perception scale ranges of a set number of segments, and the acoustic perception scale ranges of the set number of segments are merged into one acoustic perception scale segment, and then the acoustic perception scale segment is inversely transformed to the frequency domain to obtain the estimated gain of each frequency.

[0118] C2. Adjust the parameters of the initial speech enhancement model based on the difference between the decompressed estimated gain information and the corresponding true gain information.

[0119] In an embodiment of the present application, a loss value between the estimated gain information and the actual gain information can be calculated based on a loss function. For example, the loss function can be a mean-square error (MSE) loss function. The loss value is back-propagated, that is, the parameters of each network layer of the initial speech enhancement model are passed in reverse to obtain the gradient of the parameters of each network layer. Then, the parameters of each network layer of the initial speech enhancement model can be updated by gradient descent. For example, the parameters of each network layer of the initial speech enhancement model can include convolution kernel parameters (such as weights, etc.), bias parameters, etc., and can also include activation function parameters, etc.

[0120] When the gain information (estimated gain information and actual gain information) is represented in the frequency domain dimension (a set number of frequency bands), the decompressed estimated gain information can be obtained through frequency band decompression. Based on the decompressed estimated gain information and the actual gain information, the difference between the two in the frequency domain dimension can be determined. Since the frequency domain dimension is more suitable for expressing speech signals, the difference between the two gain information can be identified more intuitively and accurately, which helps the initial speech enhancement model learn the recognition knowledge of noise in the frequency domain dimension and improve the quality of model training.

[0121] In an embodiment of the present application, in each round of training of the initial speech enhancement model, feature dimensionality reduction is performed through the frequency band compression layer, which greatly reduces the complexity of subsequent operations. Moreover, through the cyclic iterative processing of the feature mapping layer, the depth of the feature mapping is increased and the number of parameters is increased without increasing the model structure, thereby improving the training effect of the model. The trained speech enhancement model reduces the computational complexity while ensuring the processing effect, thereby increasing the computational speed, thereby meeting the real-time computing requirements and improving the communication experience.

[0122] In some embodiments, the output layer in the above S324 may include an intermediate convolutional network and an output convolutional network. Then, the above S324 obtains the estimated gain information based on the third audio feature through the output layer of the initial speech enhancement model, and specifically may include the following steps D1-D2:

[0123] D1. Perform convolution processing on the third audio feature through the intermediate convolutional network in a loop iterative manner until the number of iterations reaches the set number of convolutions to obtain the intermediate audio feature; wherein, during the loop iterative process, the number of output channels of the intermediate convolutional network remains unchanged.

[0124] Among them, the intermediate convolutional network can include at least one layer of convolutional neural network, the number of layers can be set as needed, and the number of convolutions can be set as needed. For example, the intermediate convolutional network includes two layers of convolutional neural networks, and the number of convolutions is set to 2 times, which is not limited to this.

[0125] The number of output channels of the intermediate convolutional network can be set as needed. For example, it can be consistent with the number of output channels of the last iteration of the feature mapping layer, such as 128, and there is no limit on this.

[0126] D2. Perform convolution processing on the intermediate audio features through the output convolutional network in a loop iteration manner until the number of iterations reaches the set output number to obtain estimated gain information; wherein, during the loop iteration process, the number of output channels of the output convolutional network decreases each time.

[0127] Among them, the output convolutional network can include at least one layer of convolutional neural network, the number of layers can be set as needed, and the number of outputs can be set as needed. For example, the output convolutional network includes a three-layer convolutional neural network, and the number of convolutions is set to 3 times, which is not limited.

[0128] The number of output channels of multiple iterations of the output convolutional network can be set as needed. For example, the number of output channels of multiple iterations can be reduced exponentially. Assume that the feature mapping layer is iterated three times, and the number of output channels is 32, 64, and 128 respectively. The output convolutional network is also iterated three times, and the number of output channels is 128, 64, and 32 respectively. There is no limitation on this.

[0129] In an optional embodiment, as shown in FIG6 , the intermediate convolutional network in the output layer includes two connected layers of convolutional neural networks, and the output convolutional network includes three layers of convolutional neural networks connected in sequence. Assuming that the number of iterations of the intermediate convolutional network is 2 and the number of iterations of the output convolutional network is 3, the third audio feature output by the feature mapping layer is sequentially input into the two layers of convolutional neural networks of the intermediate convolutional network, and the loop is iterated twice to output the intermediate audio feature. The intermediate audio feature is then sequentially input into the three layers of convolutional neural networks of the output convolutional network. After three iterations, the estimated gain information is output.

[0130] In an embodiment of the present application, the output of the feature mapping layer is subjected to cyclic iterative convolution through the intermediate convolutional network, which can further improve the feature extraction effect. Then, the output of the intermediate convolutional network is subjected to cyclic iterative convolution through the output convolutional network to gradually reduce the number of output channels, thereby outputting the final estimated gain information.

[0131] The following is an exemplary introduction to the training process of the initial speech enhancement model of an embodiment of the present application in conjunction with Figures 7A and 7B.

[0132] In an embodiment of the present application, the training samples input into the initial speech enhancement model include a pair of sample speech signals and noisy speech signals. As shown in FIG7A , before inputting the initial speech enhancement model, the sample speech signal and the noisy speech signal are first subjected to FFT transformation respectively. For example, using an FFT operation of 1024 sampling points, the sample speech signal and the noisy speech signal are transformed respectively to obtain frequency domain signals of the two signals. The frequency domain signal includes the amplitude at each frequency, and the amplitude includes a real part and an imaginary part. The real part is 513 features, and the imaginary part is also 513 eigenvalues. Therefore, there are a total of 513*4 eigenvalues.

[0133] Then, the frequency domain signals of the sample speech signal and the noisy speech signal are input into the input layer and the frequency band compression layer of the initial speech enhancement model in turn; the output of the frequency band compression layer is iteratively looped and input into the feature mapping layer, and the number of output channels increases gradually. The number of iterations is set as needed, and the output of the feature mapping layer is iteratively looped and input into the output layer; the output layer includes an intermediate convolutional network and an output convolutional network. These two convolutional networks are iteratively looped respectively, and their number of iterations is set as needed. The number of output channels of the intermediate convolutional network remains unchanged, and the number of output channels of the output convolutional network increases gradually. Finally, the output layer outputs the estimated gain information, calculates the loss value between the estimated gain information and the actual gain information (calculated by the sample speech signal and the noisy speech signal), and adjusts the parameters of the initial speech enhancement model based on the loss value.

[0134] Exemplarily, as shown in FIG7B , each training process of the initial speech enhancement model is as follows:

[0135] 1. Obtain selected training samples, including frequency domain signals of the sample speech signal and the noisy speech signal. For example, the frequency domain signal includes 513*2 eigenvalues.

[0136] 2. The frequency domain signals of the sample speech signal and the noisy speech signal in the training sample are respectively passed through two parallel convolutional neural networks in the input layer. The outputs of these two convolutional neural networks are fused and spliced, and then passed through another convolutional neural network to integrate the feature data to obtain the first audio feature.

[0137] 3. Input the first audio feature into a frequency band compression layer, such as an ERB module. The frequency band compression layer is used to perform frequency band compression on the first audio feature, thereby further reducing computational complexity.

[0138] For example, the dimension of the first audio feature is reduced from 513 dimensions to 128 dimensions (assuming that the frequency band compression layer divides the frequency bands into 128 frequency bands) to obtain the second audio feature, thereby reducing the feature dimension.

[0139] 4. The output of the band compression layer is fed into the feature mapping layer, where it passes through a series of sub-networks: a convolutional neural network, batch normalization, an activation function, two convolutional neural networks, and an attention mechanism network. The loop is iterated three times, with the number of output channels increasing in each iteration.

[0140] Batch normalization normalizes the output of the first convolutional neural network to address numerical instability in deep neural networks. This ensures that the feature distributions of training samples in the same batch are similar, making training easier. The activation function can be PReLU, ReLU, tanh, or sigmoid.

[0141] For example, the number of output channels of the three iterations of the feature mapping layer are 32, 64, and 128 respectively. The purpose is to increase the network depth and the number of parameters so that the signal features can be learned more effectively.

[0142] 5. The third audio feature output from the feature mapping layer after three iterations is input to the intermediate convolutional network of the output layer. The intermediate convolutional network contains two convolutional neural networks and iterates twice. The number of output channels remains unchanged, for example, 128 and 128 respectively.

[0143] 6. The output results of the intermediate convolutional network after two cycles of iteration are input into the output convolutional network of the output layer. The output convolutional network contains three layers of convolutional neural networks and iterates three times. The number of output channels decreases each time.

[0144] For example, the number of output channels of the three layers is 128, 64, and 32 respectively. The purpose is to reduce the number of channels of the output features and keep them consistent with the dimension of the input features of the feature mapping layer.

[0145] 7. Finally, the output layer outputs the estimated gain information, then calculates the loss value between the estimated gain information and the actual gain information (calculated using the sample speech signal and the noisy speech signal), and adjusts the parameters of the initial speech enhancement model based on the loss value.

[0146] Specifically, the estimated gain information can be subjected to frequency band decompression, i.e., the inverse process of the frequency band compression of the frequency band compression layer in step 3 above, to obtain the estimated gain information after dimensionality increase, for example, from 128 dimensions to 513 dimensions. When calculating the true gain information using the sample speech signal and the noisy speech signal, the energy spectrum of the sample speech signal (obtained based on the frequency domain signal) can be divided by the sum of the energy spectra of the sample speech signal and the noisy speech signal to obtain the true gain information. Finally, the loss value between the estimated gain information and the true gain information can be calculated based on the MSE loss function.

[0147] After the initial speech enhancement model is trained, a trained speech enhancement model is obtained. This model is then used for online enhancement processing. Simply perform a discrete Fourier transform (e.g., FFT) on the noisy speech signal to obtain frequency domain features, such as 513*2 real and imaginary features (which remain consistent with the input features from the training process). This frequency domain feature is then input into the trained speech enhancement model, which adaptively adjusts its parameters to obtain the optimal estimated gain information. Finally, the estimated gain information is decompressed and multiplied with the power spectrum of the noisy speech signal. An inverse FFT transform is then used to obtain the final enhanced speech signal.

[0148] The present application also provides a method for speech enhancement, which is implemented by training a speech enhancement model obtained by the above-mentioned embodiment. The speech enhancement method is described below.

[0149] 8 is a flowchart of a speech enhancement method provided in an embodiment of the present application. Taking a terminal device as an example, the specific implementation process of the method includes the following steps S81-S82:

[0150] S81: Input the speech signal to be enhanced into a speech enhancement model to obtain estimated gain information.

[0151] During real-time communication, the original voice signal received by the terminal device is a time-domain signal, with the horizontal axis representing time points and the vertical axis representing amplitude. To facilitate feature analysis of the original voice signal, the original voice signal is transformed from the time domain to the frequency domain to obtain a voice signal to be enhanced, with the horizontal axis representing frequency points and the vertical axis representing amplitude. Optionally, the original voice signal can be an ultra-wideband voice signal or a standard wideband voice signal, without limitation.

[0152] Specifically, the discrete Fourier transform can be used to transform the original speech signal from the time domain to the frequency domain; for example, the original speech signal is subjected to FFT transformation to obtain the eigenvalues ​​of multiple frequency points. The eigenvalues ​​include the real part and imaginary part of the amplitude, the real part represents the amplitude of the signal, and the imaginary part represents the change of the phase of the signal with frequency.

[0153] Based on the above embodiments of the present application, it can be seen that the speech enhancement model includes an input layer, a frequency band compression layer, a feature mapping layer and an output layer. The speech signal to be enhanced is input into each of the above network layers in sequence to obtain estimated gain information.

[0154] The specific implementation process of the above S81 is introduced below.

[0155] In some embodiments, the speech signal to be enhanced is input into the speech enhancement model to obtain estimated gain information, which specifically includes the following steps E1-E4:

[0156] E1. Extract the first audio feature of the speech signal to be enhanced through the input layer of the speech enhancement model.

[0157] The feature dimension of the first audio feature obtained by extracting features of the speech signal to be enhanced through the input layer of the speech enhancement model can be 513, that is, the number of feature dimensions of the first audio feature can be consistent with the number of feature dimensions of the speech signal to be enhanced.

[0158] E2. Perform frequency band compression on the first audio feature through a frequency band compression layer of a speech enhancement model to obtain a second audio feature; the feature dimension of the second audio feature is smaller than the feature dimension of the first audio feature.

[0159] Among them, the specific implementation process of this step is similar to the specific implementation process of S322 in the above embodiment of this application, and will not be repeated here.

[0160] E3. Perform feature mapping on the second audio feature through the feature mapping layer of the speech enhancement model in a loop iterative manner until the number of iterations reaches the set mapping number, thereby obtaining a third audio feature; wherein, during the loop iterative process, the number of output channels of the feature mapping layer increases successively.

[0161] Among them, the specific implementation process of this step is similar to the specific implementation process of S323 in the above embodiment of this application, and will not be repeated here.

[0162] E4. Input the third audio feature into the output layer of the speech enhancement model, obtain estimated gain information based on the third audio feature, and use the estimated gain information to perform speech enhancement on the speech signal to be enhanced.

[0163] Among them, the specific implementation process of this step is similar to the specific implementation process of S324 in the above embodiment of this application, and will not be repeated here.

[0164] In the embodiment of the present application, when using the speech enhancement model for speech enhancement, the first audio feature of the speech signal to be enhanced is first extracted. Then, in order to reduce the complexity of subsequent operations, the first audio feature is band-compressed to obtain the second audio feature after dimensionality reduction. Furthermore, the second audio feature is feature mapped in a cyclic iterative manner through the feature mapping layer of the speech enhancement model, and the number of output channels of the feature mapping layer is gradually increased. In this way, the depth of the feature mapping can be increased without increasing the model structure, thereby ensuring the model processing effect. Finally, based on the third audio feature output by the feature mapping layer, estimated gain information is obtained to perform speech enhancement on the speech signal to be enhanced. Therefore, the embodiment of the present application can reduce the computational complexity while ensuring the processing effect, thereby increasing the computational speed, thereby meeting the real-time computing requirements and improving the communication experience.

[0165] S82: Perform speech enhancement on the speech signal to be enhanced based on the estimated gain information to remove noise from the speech signal to be enhanced to obtain an enhanced speech signal.

[0166] The estimated gain information includes the estimated gains for each of a set number of frequency bands. Before performing speech enhancement on the speech signal to be enhanced based on the estimated gain information, the estimated gain information can be subjected to frequency band decompression to obtain the estimated gains for each frequency of the speech signal to be enhanced, and then speech enhancement is performed on the speech signal to be enhanced. The above-mentioned frequency band decompression can be understood as the inverse process of the frequency band compression in step E2 of the present application.

[0167] In the embodiment of the present application, when using the speech enhancement model for speech enhancement, the first audio feature of the speech signal to be enhanced is first extracted. Then, in order to reduce the complexity of subsequent operations, the first audio feature is band-compressed to obtain the second audio feature after dimensionality reduction. Furthermore, the second audio feature is feature mapped in a cyclic iterative manner through the feature mapping layer of the speech enhancement model, and the number of output channels of the feature mapping layer is gradually increased. In this way, the depth of the feature mapping can be increased without increasing the model structure, thereby ensuring the model processing effect. Finally, based on the third audio feature output by the feature mapping layer, estimated gain information is obtained to perform speech enhancement on the speech signal to be enhanced. Therefore, the embodiment of the present application can reduce the computational complexity while ensuring the processing effect, thereby increasing the computational speed, thereby meeting the real-time computing requirements and improving the communication experience.

[0168] In some embodiments, when performing speech enhancement on the speech signal to be enhanced based on the estimated gain information in S82, the following steps F1-F3 may be performed:

[0169] F1. Decompress the estimated gains of a set number of frequency bands to obtain the estimated gains of each frequency of the speech signal to be enhanced.

[0170] This step is the inverse of the frequency band compression in step E2 above. Specifically, a set number of frequency bands are transformed from the frequency domain to the acoustic perception scale domain to obtain a set number of acoustic perception scale ranges. These acoustic perception scale ranges are then combined into a single acoustic perception scale segment, which is then inversely transformed back to the frequency domain to obtain the estimated gain for each frequency.

[0171] F2. Obtain an initial enhanced speech signal based on the estimated gain of each frequency and the speech signal to be enhanced.

[0172] Specifically, the estimated gain of each frequency is multiplied by the power spectrum of the speech signal to be enhanced. The power spectrum includes the power at each frequency, that is, the estimated gain of each frequency is multiplied by the power at the frequency to obtain the initial enhanced speech signal.

[0173] F3. Transform the initial enhanced speech signal from the frequency domain to the time domain to obtain the final enhanced speech signal.

[0174] Among them, the initial enhanced speech signal can be transformed from the frequency domain to the time domain using the inverse discrete Fourier transform. For example, in the above embodiment of the present application, the original speech signal is subjected to FFT transformation to obtain the speech signal to be enhanced. Then, in step F3, the initial enhanced speech signal is transformed from the frequency domain to the time domain using the inverse FFT transform, thereby obtaining the final enhanced speech signal.

[0175] In an embodiment of the present application, after using a speech enhancement model to obtain estimated gain information of the speech signal to be enhanced, speech enhancement is performed on the speech signal to be enhanced based on the estimated gain information to obtain an enhanced speech signal. The enhanced speech signal is a signal after removing noise from the original speech signal, which can improve the quality and intelligibility of the speech signal, thereby improving the communication experience.

[0176] The overall implementation process of the speech enhancement method according to the embodiment of the present application is exemplarily introduced below with reference to FIG9 .

[0177] As shown in Figure 9, during real-time communication, after the terminal device obtains the original voice signal, it first transforms the original voice signal from the time domain to the frequency domain, for example, by using FFT transformation, to obtain the voice signal to be enhanced; then, the voice signal to be enhanced is sequentially input into the input layer and the frequency band compression layer of the voice enhancement model; the output of the frequency band compression layer is cyclically iteratively input into the feature mapping layer, and the number of output channels increases successively, and the number of iterations is set as needed, and the output of the feature mapping layer is cyclically iteratively input into the output layer; the output layer includes an intermediate convolutional network and an output convolutional network, and these two convolutional networks are cyclically iterated respectively, and their number of iterations is set as needed, and the number of output channels of the intermediate convolutional network remains unchanged, and the number of output channels of the output convolutional network increases successively. Finally, the output layer outputs estimated gain information; the gain module performs voice enhancement on the voice signal to be enhanced based on the estimated gain information to obtain an enhanced voice signal.

[0178] For example, the original speech signal is shown in (a) of FIG10 . After the original speech signal is processed through the speech enhancement process of an embodiment of the present application, the enhanced speech signal obtained is shown in (b) of FIG10 . It can be seen that the noise signal in the original speech signal is removed from the enhanced speech signal.

[0179] The speech enhancement method of the embodiment of the present application can be applied to real-time communication scenarios, such as real-time conference scenarios. When the call environment is relatively noisy, the speech enhancement method of the embodiment of the present application can effectively remove background environmental noise, thereby ensuring the quality and intelligibility of the voice call.

[0180] For example, during real-time communication, when the microphone of the terminal device is turned on, a voice signal is collected through the microphone. Then, the terminal device uses the voice enhancement method of the embodiment of the present application to perform voice enhancement on the voice signal, and a relatively pure voice signal, that is, an enhanced voice signal, can be obtained, which effectively retains the voice of the main speaker and removes excess background noise.

[0181] Based on the same inventive concept as the above-mentioned method embodiment, the embodiment of the present application also provides a training device for a speech enhancement model. The principle of solving the problem by this device is similar to the training method for the speech enhancement model of the above-mentioned embodiment. Therefore, the implementation of this device can refer to the implementation of the above-mentioned method, and the repeated parts will not be repeated.

[0182] FIG11 is a block diagram of a device for training a speech enhancement model according to an embodiment of the present invention. The device 1100 includes:

[0183] An acquiring unit 1101 is configured to acquire a training sample set, each training sample including: a sample speech signal and a corresponding noisy speech signal, wherein the noisy speech signal is obtained by adding noise to the sample speech signal;

[0184] The training unit 1102 is configured to iteratively train the initial speech enhancement model based on the training sample set to obtain a trained speech enhancement model, wherein the initial speech enhancement model includes an input layer, a frequency band compression layer, a feature mapping layer, and an output layer; wherein each training process includes:

[0185] Through the input layer, feature extraction is performed on the sample speech signal and the corresponding noisy speech signal in the selected training sample, and then fusion is performed to obtain the first audio feature;

[0186] The first audio feature is subjected to frequency band compression by a frequency band compression layer to obtain a second audio feature; the feature dimension number of the second audio feature is smaller than the feature dimension number of the first audio feature;

[0187] Performing feature mapping on the second audio feature in a loop iterative manner through the feature mapping layer until the number of iterations reaches a set number of mappings, thereby obtaining a third audio feature; wherein, during the loop iterative process, the number of output channels of the feature mapping layer increases successively;

[0188] The third audio feature is input into the output layer to obtain estimated gain information, and the initial speech enhancement model is adjusted based on the difference between the estimated gain information and the corresponding true gain information.

[0189] In an embodiment of the present application, in each round of training of the initial speech enhancement model, feature dimensionality reduction is performed through the frequency band compression layer, which greatly reduces the complexity of subsequent operations. Moreover, through the cyclic iterative processing of the feature mapping layer, the depth of the feature mapping is increased and the number of parameters is increased without increasing the model structure, thereby improving the training effect of the model. The trained speech enhancement model reduces the computational complexity while ensuring the processing effect, thereby increasing the computational speed, thereby meeting the real-time computing requirements and improving the communication experience.

[0190] Optionally, the output layer includes an intermediate convolutional network and an output convolutional network;

[0191] When the third audio feature is input into the output layer to obtain the estimated gain information, the training unit 1102 is specifically configured to:

[0192] The third audio feature is convolved through the intermediate convolutional network in a loop iterative manner until the number of iterations reaches the set number of convolutions, thereby obtaining the intermediate audio feature; wherein, during the loop iterative process, the number of output channels of the intermediate convolutional network remains unchanged;

[0193] Through the output convolutional network, the intermediate audio features are convolved in a cyclic iterative manner until the number of iterations reaches the set output number to obtain estimated gain information; wherein, during the cyclic iterative process, the number of output channels of the output convolutional network decreases each time.

[0194] Optionally, the feature mapping layer includes at least one layer of convolutional neural network and attention mechanism network;

[0195] By performing feature mapping on the second audio feature in a cyclic iterative manner through the feature mapping layer until the number of iterations reaches a set mapping number, and when the third audio feature is obtained, the training unit 1102 is specifically configured to:

[0196] The loop iterates and executes the following steps until the number of iterations reaches the set mapping number:

[0197] Inputting the second audio features into at least one layer of convolutional neural network in sequence for convolution processing to obtain intermediate convolution features;

[0198] Input the intermediate convolutional features into the attention mechanism network for feature interaction to obtain new second audio features;

[0199] The new second audio feature is used as the third audio feature.

[0200] Optionally, the sample speech signal and the noisy speech signal are both frequency domain signals, and the first audio feature is a frequency domain feature;

[0201] When performing frequency band compression on the first audio feature to obtain the second audio feature, the training unit 1102 is specifically configured to:

[0202] transforming the first audio feature from the frequency domain into an acoustic perception scale domain, and dividing the transformed first audio feature into a set number of sub-audio features according to the acoustic perception scale domain;

[0203] A set number of sub-audio features are inversely transformed from the acoustic perception scale domain to the frequency domain, and the sub-audio features of each frequency band after the inverse transformation are filtered by a bandpass filter to obtain a second audio feature containing a set number of frequency bands.

[0204] Optionally, the estimated gain information includes estimated gains of respective set number of frequency bands;

[0205] When adjusting parameters of the initial speech enhancement model based on the difference between the estimated gain information and the corresponding actual gain information, the training unit 1102 is specifically configured to:

[0206] decompressing the estimated gains of the respective frequency bands of a set number of frequency bands to obtain decompressed estimated gain information, wherein the decompressed estimated gain information includes the estimated gain of each frequency of the noisy speech signal;

[0207] Based on the difference between the decompressed estimated gain information and the corresponding true gain information, the parameters of the initial speech enhancement model are adjusted.

[0208] Based on the same inventive concept as the above-mentioned method embodiment, a speech enhancement device is also provided in the embodiment of the present application. The principle of solving the problem by this device is similar to that of the speech enhancement method in the above-mentioned embodiment. Therefore, the implementation of this device can refer to the implementation of the above-mentioned method, and the repeated parts will not be repeated.

[0209] Referring to FIG. 12 , an embodiment of the present application provides a speech enhancement device 1200 , including:

[0210] An input unit 1201 is configured to input a speech signal to be enhanced into a speech enhancement model to obtain estimated gain information;

[0211] The speech enhancement unit 1202 is configured to perform speech enhancement on the speech signal to be enhanced based on the estimated gain information, so as to remove noise from the speech signal to be enhanced to obtain an enhanced speech signal.

[0212] Optionally, the speech signal to be enhanced is obtained by transforming the original speech signal from the time domain to the frequency domain, and the estimated gain information includes the estimated gain of each of a set number of frequency bands;

[0213] The speech enhancement unit 1202 is specifically used for:

[0214] Decompress the estimated gains of the set number of frequency bands to obtain the estimated gains of the respective frequencies of the speech signal to be enhanced;

[0215] Obtaining an initial enhanced speech signal based on the estimated gain of each frequency and the speech signal to be enhanced;

[0216] The initial enhanced speech signal is transformed from the frequency domain to the time domain to obtain the final enhanced speech signal.

[0217] For the convenience of description, the above parts are divided into units (or modules) according to their functions and described separately. Of course, when implementing this application, the functions of each unit (or module) can be implemented in the same or multiple software or hardware.

[0218] After introducing the speech enhancement method and apparatus according to an exemplary embodiment of the present application, an electronic device according to another exemplary embodiment of the present application is introduced next.

[0219] Based on the same inventive concept as the above-mentioned method embodiment, an electronic device is also provided in an embodiment of the present application. In one embodiment, the electronic device may be a terminal device, such as the terminal device 110 shown in Figure 1 . In this embodiment, the structure of the electronic device may be as shown in Figure 13 , including: a communication component 1310, a memory 1320, a display unit 1330, a camera 1340, a sensor 1350, an audio circuit 1360, a Bluetooth module 1370, a processor 1380, and other components.

[0220] The communication component 1310 is used to communicate with the server. In some embodiments, it may include a wireless fidelity (WiFi) module. The WiFi module is a short-range wireless transmission technology. Electronic devices can help users send and receive information through the WiFi module.

[0221] The memory 1320 may be used to store software programs and data. The processor 1380 executes the software programs or data stored in the memory 1320 to perform various functions of the terminal device 110 and process data.

[0222] The display unit 1330 may also be used to display information input by a user or information provided to the user, as well as a graphical user interface (GUI) of various menus of the terminal device 110 .

[0223] The camera 1340 may be used to capture static images, and the user may publish the images captured by the camera 1340 through an application.

[0224] The terminal device may further include at least one sensor 1350 .

[0225] The audio circuit 1360 , the speaker 1361 , and the microphone 1362 may provide an audio interface between the user and the terminal device 110 .

[0226] The Bluetooth module 1370 is used to exchange information with other Bluetooth devices having a Bluetooth module through the Bluetooth protocol.

[0227] The processor 1380 is the control center of the terminal device. It uses various interfaces and lines to connect various parts of the entire terminal. It performs various functions of the terminal device and processes data by running or executing software programs stored in the memory 1320 and calling data stored in the memory 1320.

[0228] In another embodiment, the electronic device may also be a server, such as the server 120 shown in FIG1 . In this embodiment, the structure of the electronic device may be as shown in FIG14 , including a memory 1401 , a communication module 1403 and one or more processors 1402 .

[0229] Memory 1401 is used to store computer programs executed by processor 1402. Memory 1401 may primarily include a program storage area and a data storage area. The program storage area may store an operating system and programs required for running instant messaging functions, while the data storage area may store various instant messaging messages and operating instruction sets.

[0230] Memory 1401 may be a volatile memory, such as random-access memory (RAM); a non-volatile memory, such as read-only memory, flash memory, a hard disk drive (HDD), or a solid-state drive (SSD); or any other medium capable of carrying or storing a desired computer program in the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 1401 may be a combination of the aforementioned memories.

[0231] The processor 1402 may include one or more central processing units (CPUs) or digital processing units, etc. The processor 1402 is configured to implement the above-mentioned speech enhancement method or speech enhancement model training method when calling the computer program stored in the memory 1401 .

[0232] The communication module 1403 is used to communicate with terminal devices and other servers.

[0233] The specific connection medium between the memory 1401, communication module 1403, and processor 1402 is not limited in the embodiments of the present application. In Figure 14, the embodiment of the present application shows that the memory 1401 and the processor 1402 are connected via a bus 1404. The bus 1404 is depicted as a bold line in Figure 14. The connection methods between other components are merely schematic and are not intended to be limiting. The bus 1404 can be divided into an address bus, a data bus, a control bus, etc. For ease of description, Figure 14 only uses a single bold line, but does not represent a single bus or a single type of bus.

[0234] Memory 1401 stores a computer storage medium, which stores computer-executable instructions for implementing the speech enhancement method or speech enhancement model training method of the embodiments of the present application. Processor 1402 is configured to execute the aforementioned speech enhancement method or speech enhancement model training method, as shown in FIG3 or FIG8.

[0235] In some possible implementations, various aspects of the speech enhancement method or speech enhancement model training method provided in the present application can also be implemented in the form of a program product, which includes a computer program. When the program product is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of the speech enhancement method or speech enhancement model training method according to the various exemplary embodiments of the present application described above in this specification. For example, the electronic device can execute the steps shown in Figure 3 or Figure 8.

[0236] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0237] The program product of the embodiment of the present application may be a portable compact disc read-only memory (CD-ROM) and include a computer program, and can be run on an electronic device. However, the program product of the present application is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with a command execution system, apparatus, or device.

[0238] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries a readable computer program. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with a command execution system, apparatus, or device.

[0239] The computer program embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0240] The computer program for performing the operations of the present application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The computer program can be executed entirely on the user electronic device, partially on the user electronic device, as a separate software package, partially on the user electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server. In cases involving remote electronic devices, the remote electronic device can be connected to the user electronic device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external electronic device (for example, using an Internet service provider to connect through the Internet).

[0241] It should be noted that although several units or subunits of the device are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, depending on the embodiment of the application, the features and functions of two or more units described above can be embodied in a single unit. Conversely, the features and functions of a single unit described above can be further divided and embodied by multiple units.

[0242] Furthermore, although the operations of the method of the present application are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all illustrated operations must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0243] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain a computer-usable computer program.

[0244] The present application is described with reference to the flow chart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present application. It should be understood that each flow process and / or box in the flow chart and / or block diagram and the combination of the flow process and / or box in the flow chart and / or block diagram can be realized by computer program commands. These computer program commands can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processing machine or other programmable data processing device to produce a machine, so that the command executed by the processor of the computer or other programmable data processing device produces a device for realizing the function specified in one flow chart flow or multiple flows and / or one box or multiple boxes of the block diagram.

[0245] These computer program commands may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the commands stored in the computer-readable memory produce a manufactured product including a command device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0246] These computer program commands can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the commands executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0247] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0248] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A method for training a speech enhancement model, the method being performed by an electronic device, the method comprising: Acquire a training sample set, each training sample comprising: a sample speech signal and a corresponding noisy speech signal, wherein the noisy speech signal is obtained by adding noise to the sample speech signal; Iteratively training the initial speech enhancement model based on the training sample set to obtain a trained speech enhancement model, wherein the initial speech enhancement model includes an input layer, a frequency band compression layer, a feature mapping layer and an output layer; wherein each training process includes: Through the input layer, feature extraction is performed on the sample speech signal and the corresponding noisy speech signal in the selected training sample respectively, and then fusion is performed to obtain the first audio feature; The first audio feature is subjected to frequency band compression by the frequency band compression layer to obtain a second audio feature; the feature dimension number of the second audio feature is smaller than the feature dimension number of the first audio feature; Through the feature mapping layer, the second audio feature is feature mapped in a loop iterative manner until the number of iterations reaches a set mapping number, so as to obtain a third audio feature; wherein, during the loop iterative process, the number of output channels of the feature mapping layer increases gradually; The third audio feature is input into the output layer to obtain estimated gain information, and based on the difference between the estimated gain information and the corresponding true gain information, the initial speech enhancement model is adjusted.

2. The method according to claim 1, wherein the output layer comprises an intermediate convolutional network and an output convolutional network; The step of inputting the third audio feature into the output layer to obtain estimated gain information includes: Through the intermediate convolutional network, the third audio feature is convolved in a loop iterative manner until the number of iterations reaches a set number of convolutions, so as to obtain the intermediate audio feature; wherein, during the loop iterative process, the number of output channels of the intermediate convolutional network remains unchanged; Through the output convolution network, the intermediate audio features are convolved in a loop iterative manner until the number of iterations reaches the set output number, so as to obtain the estimated gain information; wherein, during the loop iterative process, the number of output channels of the output convolution network decreases successively.

3. The method according to claim 1, wherein the feature mapping layer comprises at least one layer of convolutional neural network and an attention mechanism network; The step of performing feature mapping on the second audio feature in a loop iteration manner through the feature mapping layer until the number of iterations reaches a set number of mappings to obtain a third audio feature includes: The following steps are executed in a loop iteratively until the number of iterations reaches the set mapping number: Inputting the second audio features into the at least one layer of convolutional neural network in sequence for convolution processing to obtain intermediate convolution features; Inputting the intermediate convolutional features into the attention mechanism network for feature interaction to obtain new second audio features; The new second audio feature is used as the third audio feature.

4. The method according to claim 1, wherein the sample speech signal and the noisy speech signal are both frequency domain signals, and the first audio feature is a frequency domain feature; The step of performing frequency band compression on the first audio feature to obtain a second audio feature comprises: The first audio feature is transformed from the frequency domain to the acoustic perception scale domain, and the transformed first audio feature is converted according to The acoustic perception scale domain is divided into a set number of sub-audio features; The set number of sub-audio features are inversely transformed from the acoustic perception scale domain to the frequency domain, and the sub-audio features of each frequency band after the inverse transformation are filtered by a bandpass filter to obtain a second audio feature containing the set number of frequency bands.

5. The method according to claim 4, wherein the estimated gain information includes the estimated gain of each of the set number of frequency bands; The adjusting parameters of the initial speech enhancement model based on the difference between the estimated gain information and the corresponding real gain information includes: Decompressing the estimated gains of the set number of frequency bands to obtain decompressed estimated gain information, wherein the decompressed estimated gain information includes the estimated gain of each frequency of the noisy speech signal; Based on the difference between the decompressed estimated gain information and the corresponding true gain information, the initial speech enhancement model is adjusted.

6. A speech enhancement method, the method being performed by an electronic device, the method comprising: Inputting the speech signal to be enhanced into the speech enhancement model trained by the method according to any one of claims 1 to 5 to obtain estimated gain information; Speech enhancement is performed on the speech signal to be enhanced based on the estimated gain information to remove noise in the speech signal to be enhanced to obtain an enhanced speech signal.

7. The method according to claim 6, wherein the speech signal to be enhanced is obtained by transforming the original speech signal from the time domain to the frequency domain, and the estimated gain information includes the estimated gain of each of a set number of frequency bands; The performing speech enhancement on the speech signal to be enhanced based on the estimated gain information to remove noise in the speech signal to be enhanced to obtain an enhanced speech signal includes: Decompressing the estimated gains of the set number of frequency bands to obtain the estimated gains of the frequencies of the speech signal to be enhanced; Based on the estimated gain of each frequency and the speech signal to be enhanced, obtaining an initial enhanced speech signal; The initial enhanced speech signal is transformed from the frequency domain to the time domain to obtain a final enhanced speech signal.

8. A training device for a speech enhancement model, the device comprising: An acquisition unit is used to acquire a training sample set, each training sample includes: a sample speech signal and a corresponding noisy speech signal, wherein the noisy speech signal is obtained by adding noise to the sample speech signal; A training unit is used to iteratively train the initial speech enhancement model based on the training sample set to obtain a trained speech enhancement model, wherein the initial speech enhancement model includes an input layer, a frequency band compression layer, a feature mapping layer and an output layer; wherein each training process includes: Through the input layer, feature extraction is performed on the sample speech signal and the corresponding noisy speech signal in the selected training sample respectively, and then fusion is performed to obtain the first audio feature; The first audio feature is subjected to frequency band compression by the frequency band compression layer to obtain a second audio feature; the feature dimension number of the second audio feature is smaller than the feature dimension number of the first audio feature; Through the feature mapping layer, the second audio feature is feature mapped in a loop iterative manner until the number of iterations reaches the set mapping number, thereby obtaining a third audio feature; wherein, during the loop iterative process, the feature The number of output channels of the feature map layer increases gradually; The third audio feature is input into the output layer to obtain estimated gain information, and based on the difference between the estimated gain information and the corresponding true gain information, the initial speech enhancement model is adjusted.

9. The apparatus according to claim 8, wherein the output layer comprises an intermediate convolutional network and an output convolutional network; When the third audio feature is input into the output layer to obtain the estimated gain information, the training unit is specifically used for: Through the intermediate convolution network, the third audio feature is convolved in a loop iterative manner until the number of iterations reaches the set number of convolutions to obtain the intermediate audio feature; wherein, During the loop iteration process, the number of output channels of the intermediate convolutional network remains unchanged; Through the output convolution network, the intermediate audio features are convolved in a loop iterative manner until the number of iterations reaches the set output number, so as to obtain the estimated gain information; wherein, during the loop iterative process, the number of output channels of the output convolution network decreases successively.

10. The device according to claim 8, wherein the feature mapping layer comprises at least one layer of convolutional neural network and an attention mechanism network; The feature mapping layer is used to perform feature mapping on the second audio feature in a loop iteration manner until the number of iterations reaches a set number of mappings to obtain a third audio feature, and the training unit is specifically used to: The following steps are executed in a loop iteratively until the number of iterations reaches the set mapping number: Inputting the second audio features into the at least one layer of convolutional neural network in sequence for convolution processing to obtain intermediate convolution features; Inputting the intermediate convolutional features into the attention mechanism network for feature interaction to obtain new second audio features; The new second audio feature is used as the third audio feature.

11. The device according to claim 8, wherein the sample speech signal and the noisy speech signal are both frequency domain signals, and the first audio feature is a frequency domain feature; When the first audio feature is subjected to frequency band compression to obtain the second audio feature, the training unit is specifically used for: Transforming the first audio feature from the frequency domain to an acoustic perception scale domain, and dividing the transformed first audio feature into a set number of sub-audio features according to the acoustic perception scale domain; The set number of sub-audio features are inversely transformed from the acoustic perception scale domain to the frequency domain, and the sub-audio features of each frequency band after the inverse transformation are filtered by a bandpass filter to obtain a second audio feature containing the set number of frequency bands.

12. A speech enhancement device, comprising: An input unit, used for inputting a speech signal to be enhanced into a speech enhancement model trained by the method according to any one of claims 1 to 5 to obtain estimated gain information; The speech enhancement unit is used to perform speech enhancement on the speech signal to be enhanced based on the estimated gain information, so as to remove noise in the speech signal to be enhanced to obtain an enhanced speech signal.

13. An electronic device comprising a processor and a memory, wherein: The memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 5 or any one of claims 6 to 7.

14. A computer-readable storage medium comprising a computer program, wherein when the computer program is executed on an electronic device, the computer program is used to cause the electronic device to execute the steps of the method according to any one of claims 1 to 5 or any one of claims 6 to 7.

15. A computer program product, comprising a computer program, wherein the computer program is stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device performs the steps of the method described in any one of claims 1 to 5 or any one of 6 to 7.

Images