Intercom voice signal processing method, device, related equipment and program product

Abstract

The present application discloses a method, apparatus, related equipment and computer program product for processing voice signals of a walkie-talkie. For an uplink full-band voice signal, a noise reduction process is performed through a configured noise reduction network to obtain a clean full-band voice signal after noise reduction, and the clean full-band voice signal is transmitted externally; for a received downlink narrow-band voice signal, a noise reduction process is performed through the noise reduction network to obtain a clean narrow-band voice signal after noise reduction, and the noise reduction network includes a hybrid normalization layer composed of batch normalization and layer normalization; a target voice signal is output based on the clean narrow-band voice signal. Aiming at the scenario where the bandwidths of uplink and downlink voice signals of a walkie-talkie are different, the present application designs a noise reduction network, which can adapt to the noise reduction process of voice signals of different bandwidths through a hybrid normalization layer, ensuring that the same noise reduction network can be applied to the processing of uplink and downlink voice signals at the same time, thereby ensuring the quality of the voice signal transmitted by the walkie-talkie.

Description

Technical Field

[0001] The present application relates to the technical field of voice signal processing, and more specifically, to a method, apparatus, related equipment and computer program product for processing voice signals of an intercom. Background Art

[0002] In the fields of private network communication and civil use, walkie-talkies are an important tool for instant communication. They are widely used in various scenarios with unique communication needs, such as construction sites, the hotel industry, security, and outdoor activities. Effective voice communication is crucial to the efficiency and success of mission execution. However, walkie-talkie systems usually operate within a limited bandwidth, and voice signals are often limited by the bandwidth during transmission, resulting in a decrease in sound quality, especially in noisy environments, where noise and signal distortion can seriously affect the clarity and intelligibility of voice. Summary of the invention

[0003] In view of the above problems, the present application is proposed to provide a method, apparatus, related equipment and computer program product for processing voice signals of an intercom, so as to improve the quality of the voice signals transmitted by the intercom.

[0004] The specific plan is as follows:

[0005] In a first aspect, a method for processing a voice signal of an intercom is provided, comprising:

[0006] For the uplink full-band voice signal, a noise reduction process is performed through the configured noise reduction network to obtain a clean full-band voice signal after noise reduction, and the clean full-band voice signal is transmitted externally;

[0007] For the received downlink narrowband speech signal, the noise reduction network is used to perform noise reduction processing to obtain a clean narrowband speech signal after noise reduction, wherein the noise reduction network includes a hybrid normalization layer composed of a batch normalization BatchNorm and a layer normalization LayerNorm;

[0008] A target speech signal to be output is obtained based on the clean narrow-band speech signal, and the target speech signal is output.

[0009] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the denoising network further includes a dilated convolutional layer and a dual-path gated recurrent unit GRU;

[0010] The atrous convolution layer, the BatchNorm, the dual-path GRU and the LayerNorm are sequentially connected to form a feature extraction network unit in the denoising network.

[0011] In one possible design, in another implementation of the first aspect of the embodiments of the present application, the training data set used in the noise reduction network training stage includes uplink training data and downlink training data, the uplink training data is a full-band speech signal, and the downlink training data is a narrowband speech signal, at least part of the downlink training data is homologous data of the uplink training data, and the homologous data is obtained after the uplink training data is processed by a bandpass filter.

[0012] In one possible design, in another implementation of the first aspect of the embodiment of the present application, the denoising network training stage adopts a random Mask strategy to randomly select a period of time or a frequency band on the input and corresponding output speech signals for zeroing processing.

[0013] In a possible design, in another implementation of the first aspect of the embodiments of the present application, the denoising network is a network structure obtained by quantizing and compressing the original denoising network after training.

[0014] In a possible design, in another implementation of the first aspect of the embodiment of the present application, a process of obtaining a target speech signal to be output based on the clean narrow-band speech signal includes:

[0015] The clean narrowband speech signal is band-extended using a configured frequency band extension network to obtain a clean full-band speech signal, and the clean full-band speech signal is used as a target speech signal to be output.

[0016] In one possible design, in another implementation of the first aspect of the embodiment of the present application, the frequency band extension network includes a plurality of deconvolution networks connected in series, which are used to gradually upsample the input narrow-band speech audio domain signal until a restored full-band speech time domain signal is finally obtained.

[0017] In a possible design, in another implementation of the first aspect of the embodiments of the present application, the training process of the frequency band extension network includes:

[0018] Acquire a training data set, wherein the training data set includes narrowband speech signal samples and their corresponding full-band speech signal labels;

[0019] Extracting the short-time Fourier transform feature of the narrowband speech signal sample and sending it to the frequency band extension network to obtain the time domain signal output by the last deconvolution network, and performing inverse Fourier transform on the output signal of each deconvolution network except the last deconvolution network to obtain the time domain signal corresponding to each deconvolution network;

[0020] According to the sampling multiple of each deconvolution network, frame shift sampling is performed on the full-band speech signal label to obtain a supervision time domain signal of each deconvolution network;

[0021] For each deconvolution network, calculate the Mel spectrum distance loss value of its corresponding time domain signal and the supervision time domain signal;

[0022] The frequency band extension network is trained based on the total loss function value to obtain a trained frequency band extension network, wherein the total loss function value at least includes the Mel-spectrogram distance loss value.

[0023] In one possible design, in another implementation of the first aspect of the embodiment of the present application, the training process of the frequency band extension network adopts an adversarial training method, and the total loss function value also includes an adversarial loss value.

[0024] In one possible design, in another implementation of the first aspect of the embodiment of the present application, the total loss function value also includes a feature-level discriminator loss value, and the feature-level discriminator loss value is used to represent the distance loss value between the hidden features extracted by the discriminator for the time domain signal generated by the generator and the hidden features extracted by the discriminator for the full-band speech signal label.

[0025] In a possible design, in another implementation of the first aspect of the embodiments of the present application, the frequency band extension network is a network structure obtained by quantizing and compressing the trained original frequency band extension network.

[0026] In a second aspect, a walkie-talkie voice signal processing device is provided, comprising:

[0027] An uplink noise reduction unit, configured to perform noise reduction processing on the uplink full-band voice signal through a configured noise reduction network to obtain a clean full-band voice signal after noise reduction, and transmit the clean full-band voice signal externally;

[0028] A downlink noise reduction unit, configured to perform noise reduction processing on the received downlink narrowband speech signal through the noise reduction network to obtain a clean narrowband speech signal after noise reduction, wherein the noise reduction network includes a hybrid normalization layer composed of a batch normalization BatchNorm and a layer normalization LayerNorm;

[0029] The output signal determination unit is used to obtain a target speech signal to be output based on the clean narrow-band speech signal, and output the target speech signal.

[0030] In a third aspect, an electronic device is provided, comprising: a memory and a processor;

[0031] The memory is used to store programs;

[0032] The processor is used to execute the program to implement the intercom voice signal processing method described in any one of the first aspects of the present application.

[0033] In a fourth aspect, a readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the intercom voice signal processing method described in any one of the first aspects of the present application is implemented.

[0034] In a fifth aspect, a computer program product is provided, including a computer program, which, when executed by a processor, implements the intercom voice signal processing method described in any one of the first aspects of the present application.

[0035] By means of the above technical solution, considering that in actual applications, the uplink voice signal of the walkie-talkie is generally full-band, and due to the encoding and decoding processing of the transmission process, the downlink voice signal received by the walkie-talkie is narrow-band, in order to improve the quality of the voice signal transmitted by the walkie-talkie, the present application provides a network that can perform noise reduction on voice signals of different bandwidths, so that the uplink full-band voice signal can be processed for noise reduction to obtain a clean full-band voice signal and transmit it externally, and the received downlink narrow-band voice signal can be processed for noise reduction to obtain a clean narrow-band voice signal, and the target voice signal is determined based on this and output externally. The noise reduction network provided by the present application includes a hybrid normalization layer composed of batch normalization BatchNorm and layer normalization LayerNorm. The hybrid normalization layer can adapt to voice signals of different bandwidths, prevent the noise reduction network from overfitting to input data of a certain bandwidth, and avoid the decrease in noise reduction effect caused by the difference in distribution of narrow-band voice signals and full-band voice signals. It can be seen that the present application specially designs a noise reduction network for the scenario where the bandwidths of the uplink and downlink voice signals of the walkie-talkie are different. Through the mixed normalization layer, it can adapt to the noise reduction processing of voice signals with different bandwidths, ensuring that the same noise reduction network can be used for processing both uplink and downlink voice signals. While ensuring the quality of the voice signal transmitted by the walkie-talkie, it avoids the resource overhead brought about by designing different noise reduction networks for the uplink and downlink links respectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0037] Figure 1 A schematic diagram of an implementation system architecture of a method for processing intercom voice signals provided in an embodiment of the present application;

[0038] Figure 2 A schematic diagram of the structure of an intercom provided in an embodiment of the present application;

[0039] Figure 3 A schematic diagram of a flow chart of a method for processing intercom voice signals provided in an embodiment of the present application;

[0040] Figure 4 A schematic diagram of a scenario of an uplink voice signal processing process provided in an embodiment of the present application;

[0041] Figure 5 A schematic diagram of a downlink voice signal processing process provided in an embodiment of the present application;

[0042] Figure 6 A schematic diagram of a noise reduction network structure provided in an embodiment of the present application;

[0043] Figure 7 A schematic diagram of a frequency band extension network structure provided in an embodiment of the present application;

[0044] Figure 8 A schematic diagram of a frequency band extension network training process provided in an embodiment of the present application;

[0045] Fig. 9 A schematic diagram of the structure of a walkie-talkie voice signal processing device provided in an embodiment of the present application;

[0046] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0048] The present application provides a method for processing intercom voice signals, which can be applied to Figure 1 As shown in the system architecture, the system may include at least two intercoms 100 .

[0049] The intercom voice signal processing method of the present application can be applied to the intercom 100. In addition, the intercom voice signal processing method of the present application can also be applied to a system consisting of an intercom and a server. Exemplarily, the noise reduction network and the frequency band extension network used in the intercom voice signal processing method of the present application can be pre-trained in the server 200 and deployed to the intercom 100 after the training is completed.

[0050] Next describe Figure 1 The product form of the walkie-talkie 100;

[0051] The intercom 100 may be a device having a real-time intercom capability. Figure 2 An optional hardware structure diagram of the intercom 100 is shown.

[0052] refer to Figure 2 As shown, the intercom 100 may include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160, a speaker 161, a microphone 162, an earphone jack 163 (optional), a processor 170, an external interface 180, a power supply 190 and other components. Those skilled in the art will appreciate that Figure 2 These are merely examples of terminals or multi-function devices and do not constitute limitations on the terminals or multi-function devices, which may include more or fewer components than those shown in the figures, or combinations of certain components, or different components.

[0053] The input unit 130 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the portable multifunctional device. Specifically, the input unit 130 may include a touch screen 131 and / or other input devices 132. The touch screen 131 can collect the user's touch operations on or near it (such as the user's operation on or near the touch screen using any suitable object such as fingers, joints, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 170, and can receive and execute the command sent by the processor 170; the touch signal at least includes the touch point coordinate information. The touch screen 131 can provide an input interface and an output interface between the intercom 100 and the user. In addition, the touch screen can be implemented using multiple types such as resistive, capacitive, infrared and surface acoustic wave. In addition to the touch screen 131, the input unit 130 can also include other input devices. Specifically, other input devices 132 may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0054] Among them, other input devices 132 can receive input data and so on.

[0055] The display unit 140 may be used to display information input by a user or provided to a user, various menus of the terminal 100, an interactive interface, file display, and / or playback of any multimedia file.

[0056] The memory 120 can be used to store instructions and data. The memory 120 can mainly include an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files, texts, etc.; the instruction storage area can store software units such as operating systems, applications, instructions required for at least one function, or their subsets and extensions. It can also include a non-volatile random access memory; provide the processor 170 with hardware, software and data resources including management of computing and processing equipment, and support control software and applications. It is also used for the storage of multimedia files, and the storage of running programs and applications.

[0057] The processor 170 is the control center of the intercom 100. It uses various interfaces and lines to connect various parts of the entire terminal 100. By running or executing instructions stored in the memory 120 and calling data stored in the memory 120, it executes various functions of the intercom 100 and processes data, thereby controlling the intercom device as a whole. Optionally, the processor 170 may include one or more processing units; preferably, the processor 170 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 170. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips. The processor 170 may also be used to generate corresponding operation control signals, send them to corresponding components of the computing and processing device, read and process data in the software, especially read and process data and programs in the memory 120, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0058] Among them, the memory 120 can be used to store software codes related to the walkie-talkie voice signal processing method, the processor 170 can execute the steps of the walkie-talkie voice signal processing method, and can also schedule other units (such as the above-mentioned input unit 130 and display unit 140) to achieve corresponding functions.

[0059] The radio frequency unit 110 (optional) can be used to receive and send information or receive and send signals during a call, for example, after receiving downlink information, it is sent to the processor 170 for processing; in addition, the uplink data is sent out. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA), a duplexer, etc. In addition, the radio frequency unit 110 can also communicate with network devices and other devices through wireless communication.

[0060] In the embodiment of the present application, the radio frequency unit 110 can transmit uplink voice signals and receive downlink voice signals sent by other walkie-talkies.

[0061] The intercom 100 also includes a power supply 190 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 170 through a power management system, so that the power management system can manage charging, discharging, power consumption and other functions.

[0062] The intercom 100 further includes an external interface 180 , which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the intercom 100 to communicate with other devices, or to connect a charger to charge the terminal 100 .

[0063] Although not shown, the intercom 100 may also include a flashlight, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which will not be described in detail here. Some or all of the methods described below can be applied to Figure 2 In the intercom 100 shown.

[0064] Due to technical and design limitations, the current uplink voice signal of two-way radios is generally full-band, while the downlink voice signal is narrowband, which results in different bandwidths for the uplink and downlink voice signals. Traditional noise reduction networks do not take this feature of two-way radios into account. If the traditional noise reduction network is directly applied to the noise reduction process of both the uplink and downlink voice signals of a two-way radio, the noise reduction effect of one of the links will be poor, or even the noise reduction function cannot be achieved. In view of this, current two-way radio equipment generally only supports single-link noise reduction processing, that is, only the uplink voice signal is noise reduced, or only the downlink voice signal is noise reduced.

[0065] Since this type of walkie-talkie does not perform noise reduction processing on the entire link, the voice signal still contains a lot of noise and the quality is poor. In addition, there is also the compatibility issue between walkie-talkies of different brands.

[0066] One existing solution is to train two different noise reduction networks for uplink voice signals and downlink voice signals respectively, and deploy them in the intercom at the same time. In this way, although noise reduction can be performed on both uplink voice signals and downlink voice signals, it places higher requirements on the hardware performance of the intercom, increasing the hardware cost of the intercom.

[0067] In order to solve the above problems, the embodiment of the present application provides a method for processing voice signals of an intercom. Taking the method applied to an intercom as an example, referring to Figure 3 The intercom voice signal processing method specifically comprises the following steps:

[0068] Step S100: For the uplink full-band voice signal, a noise reduction process is performed through the configured noise reduction network to obtain a clean full-band voice signal after noise reduction, and the clean full-band voice signal is transmitted externally.

[0069] The method of the present application involves noise reduction processing of uplink voice signals and downlink voice signals respectively, wherein the uplink voice signal refers to the voice signal transmitted from the intercom, and the downlink voice signal refers to the voice signal received by the intercom.

[0070] Normally, the uplink voice signal to be transmitted by the intercom is full-band and generally carries noise. Therefore, the uplink noisy full-band voice signal (referred to as noisy full-band signal) can be subjected to noise reduction processing through a pre-configured noise reduction network to obtain a clean full-band voice signal after noise reduction (referred to as clean full-band signal), and then the clean full-band voice signal is transmitted externally.

[0071] Combination Figure 4 , which illustrates the processing process of the uplink voice signal of the walkie-talkie. For the uplink noisy full-band signal, a clean full-band signal is obtained after being processed by the noise reduction network.

[0072] Step S110: The received downlink narrow-band speech signal is subjected to noise reduction processing through the noise reduction network to obtain a clean narrow-band speech signal after noise reduction.

[0073] There is no necessary sequence between step S100 and step S110 described in this embodiment. The two are respectively the processing process of the uplink voice signal and the downlink voice signal. Figure 3 Only one optional process sequence is exemplified.

[0074] When the intercom transmits the uplink voice signal to the outside, it needs to encode the signal. The received downlink voice signal needs to be decoded. Due to the encoding and decoding process and other transmission environment influences, the downlink voice signal received by the intercom is narrowband, for example, limited to the frequency band range of 300-3400Hz. In addition, the downlink narrowband voice signal may carry noise. For example, when the other party's intercom does not have the ability to perform noise reduction processing on the uplink voice signal, the downlink narrowband voice signal received by this intercom carries noise. Alternatively, additional noise may be introduced during the signal transmission process and the encoding and decoding process.

[0075] In order to ensure the quality of the voice signal output by the intercom, in this embodiment, the downlink noisy narrowband voice signal (referred to as the noisy narrowband signal) is further subjected to noise reduction processing to obtain a clean narrowband voice signal after noise reduction (referred to as the clean narrowband signal). Figure 5, which illustrates the processing process of the downlink voice signal of the intercom. For the downlink noisy narrowband signal, a clean narrowband signal is obtained after being processed by the noise reduction network.

[0076] In this step, the noise reduction network used in the downlink voice signal noise reduction process is the same as the noise reduction network used in the uplink voice signal noise reduction process. This application can train the noise reduction network on the server in advance, and deploy the trained noise reduction network on the intercom.

[0077] Furthermore, in order to reduce the complexity of the noise reduction network and facilitate its deployment on the intercom on the end side, the noise reduction network trained on the server can be further quantized and compressed, for example, using a quantization strategy of 8Bit or other quantization scales for some structures to further reduce the structural complexity of the noise reduction network. The quantized and compressed noise reduction network can then be deployed on the intercom, meeting the requirements for the implementation of the noise reduction network on the intercom chip.

[0078] In order to adapt to the different bandwidths of uplink and downlink voice signals of walkie-talkies and to perform noise reduction processing on the uplink and downlink voice signals respectively through the same noise reduction network, the present application creatively proposes a noise reduction network structure. The noise reduction network includes a hybrid normalization layer composed of batch normalization BatchNorm and layer normalization LayerNorm.

[0079] Traditional noise reduction networks used in other scenarios generally only use a single BatchNorm or LayerNorm as a normalization layer. However, when applied to the intercom of this application, due to the different bandwidths of the uplink and downlink voice signals, the use of a single type of normalization layer can easily make the noise reduction network overfit to the input data of a certain bandwidth. The final effect may be that only the voice signal of one link of the uplink and downlink voice signals can be denoised, and the voice signal of the other link has a poor noise reduction effect or even cannot be denoised, which cannot meet the requirement of the intercom in this application to perform noise reduction on both the uplink and downlink voice signals at the same time.

[0080] The noise reduction network specially designed in this application can perform normalization processing from different directions through the mixed use of BatchNorm and LayerNorm, avoiding the degradation of the effect caused by the difference in distribution of narrowband signals and full-band signals, and can simultaneously ensure good noise reduction effects on both uplink and downlink voice signals.

[0081] It should be noted that the denoising network may include other network structures in addition to the above-mentioned hybrid normalization layer to achieve basic denoising capabilities. In this embodiment, only the structure of the normalization layer in the denoising network is introduced in detail, and an optional overall structure of the denoising network will be introduced in the following embodiments of the present application.

[0082] Step S120: obtaining a target speech signal to be output based on the clean narrow-band speech signal, and outputting the target speech signal.

[0083] Specifically, after the downlink voice signal is subjected to noise reduction processing, a clean narrowband voice signal is obtained. Based on the clean narrowband voice signal, a target voice signal to be outputted can be obtained and outputted.

[0084] In some possible implementations, the clean narrow-band speech signal may be directly used as the target speech signal to be output for output broadcasting.

[0085] In some other possible implementations, the clean narrowband speech signal may be further post-processed, and the post-processed speech signal may be output as the target speech signal. Figure 5 An optional post-processing method is to use a pre-configured frequency band extension network to perform frequency band extension on the clean narrow-band speech signal to obtain a clean full-band speech signal (referred to as the clean full-band signal for short), and use the clean full-band speech signal as the target speech signal to be output.

[0086] By adding the frequency band expansion process, the clarity and realism of the speech signal are enhanced.

[0087] The frequency band extension network may be pre-trained on a server, and the trained frequency band extension network may be deployed on the intercom.

[0088] Furthermore, in order to reduce the complexity of the frequency band extension network and facilitate its deployment on the intercom on the end side, the frequency band extension network trained on the server can be further quantized and compressed, for example, using a quantization strategy of 8Bit or other quantization scales for some structures to further reduce the structural complexity of the frequency band extension network. The quantized and compressed frequency band extension network can then be deployed on the intercom, meeting the requirements for the implementation of the frequency band extension network on the intercom chip.

[0089] The structure of the frequency band extension network and its training process are described in detail in the subsequent embodiments of this application.

[0090] The noise reduction network provided in the embodiment of the present application includes a hybrid normalization layer composed of batch normalization BatchNorm and layer normalization LayerNorm. The hybrid normalization layer can adapt to voice signals of different bandwidths, prevent the noise reduction network from overfitting to input data of a certain bandwidth, and avoid the decrease in noise reduction effect caused by the distribution difference between narrowband voice signals and full-band voice signals. It can be seen that the method of the present application can perform noise reduction processing on both the uplink voice signal and the downlink voice signal of the walkie-talkie through the same noise reduction network, thereby improving the quality of the voice signal played by the walkie-talkie. In addition, since noise reduction processing can be performed on both the uplink and downlink voice signals, the walkie-talkie using the method of the present application can seamlessly connect to walkie-talkies of the same brand or other brands, that is, brand compatibility is higher.

[0091] In some embodiments of the present application, the structure and training process of the above-mentioned noise reduction network are described.

[0092] In some possible implementations, the denoising network may further include a dilated convolution layer DilationConv and a dual-path gated recurrent unit DualPathGRU.

[0093] Among them, the dilated convolution layer DilationConv, BatchNorm, dual-path gated recurrent unit DualPathGRU and LayerNorm are connected in sequence to form a feature extraction network unit in the denoising network.

[0094] Combination Figure 6 The optional overall structure of the denoising network illustrated may include one or more of the above-mentioned feature extraction network units for extracting hidden features of speech signals.

[0095] Among them, DilationConv can achieve convolution with a larger field of view, thereby improving the noise reduction effect of the network. DualPathGRU is a bidirectional GRU layer in the time dimension and frequency dimension, which can fully mine the time-frequency characteristics of the data.

[0096] In this embodiment, by combining DilationConv and DualPathGRU, features of different scales in the input signal can be fully extracted.

[0097] Combination Figure 6 The overall structure of the denoising network illustrated in the example may adopt an encoder-decoder structure. Figure 6 The left branch can be regarded as an encoder, and the right branch can be regarded as a decoder.

[0098] further, Figure 6The denoising network shown in the example can adopt a Unet structure, which includes multiple layers (for example, two layers) of downsampling and upsampling, and cooperates with multiple feature extraction network units to extract shallow features and deep features, and then fuse the shallow features and deep features, so that the network has a deeper and more comprehensive understanding of the acoustic features of the speech signal, which can improve the denoising effect.

[0099] Figure 6 In the noise reduction network, the input is a speech domain signal (corresponding to the uplink full-band signal with noise). The speech domain signal can be a voice signal collected by the microphone of the intercom. In addition, for some intercoms with echo cancellation capabilities, the input includes not only the voice signal collected by the microphone, but also a reference signal (the echo signal of the sound played by the speaker).

[0100] Since the speech domain signal contains real and complex parts, the multi-channel signal feature interaction module can use complex convolution and modulus value addition to pre-process the input signal.

[0101] The complex signal estimation module is used to estimate the amplitude and phase of the clean signal based on the input features, and finally restore the clean signal through the estimated amplitude and phase, and the clean signal can be a speech domain signal.

[0102] Furthermore, the training process of the denoising network is introduced. An optional training strategy for the denoising network is provided as follows.

[0103] First, construct a training dataset.

[0104] The training data set includes uplink training data and downlink training data. The uplink training data is a full-band speech signal, and the downlink training data is a narrow-band speech signal.

[0105] In order to ensure that the processing capability of the noise reduction network for uplink and downlink voice signals is consistent, the ratio of uplink and downlink training data can be controlled when constructing the data set. Considering that the acquisition difficulty of downlink training data is higher than that of uplink training data, in order to match the richness of uplink and downlink training data, in this embodiment, the homologous data of the uplink training data can be used to construct at least part of the downlink training data. Among them, the homologous data is obtained after the uplink training data is processed by a bandpass filter.

[0106] Specifically, the uplink training data (full-band signal) is processed by a bandpass filter to obtain a narrowband signal, and the downlink training data is constructed using the narrowband signal.

[0107] For any piece of training data in the uplink or downlink training data, it may include a training sample and a sample label, wherein the training sample may be a noisy speech signal, and the sample label is the corresponding clean speech signal after denoising.

[0108] Taking the uplink training data as an example, we can arrange the use scenario of the walkie-talkie and collect the noise signal (full band) in the scenario. Then, we can superimpose the noise signal with the pre-collected clean voice signal to obtain the noisy speech signal sample, and use the clean voice signal as the sample label corresponding to the noisy speech signal sample.

[0109] In order to improve the robustness of the trained noise reduction network, the training data set constructed in this application can cover as many complex usage scenarios of walkie-talkies as possible, such as construction sites, fires, roads, and other scenarios.

[0110] After constructing the training data set, the denoising network is trained using the training data set.

[0111] In order to prevent the denoising network from overfitting to a fixed distribution of data, resulting in poor model robustness, the present application may further adopt a random Mask strategy during the training of the denoising network, that is, randomly selecting a period of time or a frequency band on the input and corresponding output speech signals for zeroing.

[0112] Furthermore, in order to improve the noise reduction effect of the noise reduction network on low signal-to-noise ratio signals, an asymmetric Loss function can be used. An example of an asymmetric Loss function is provided below:

[0113]

[0114] Among them, S represents the true value label, Represents the estimated spectrum of the network, t represents the time dimension, f represents the frequency dimension, T represents the maximum value of the time dimension, F represents the maximum value of the frequency dimension, x represents the value of the spectrum point, and P represents the exponential term, which is usually 2.

[0115] Through the above-mentioned asymmetric Loss function, when the denoising network over-estimates or targets non-speech segments, the network's Loss is not back-propagated. The Loss function pays more attention to the under-estimation of the speech segment. The asymmetric supervision of the speech segment and the noise segment can improve the denoising effect of the network on low signal-to-noise ratio signals.

[0116] In some embodiments of the present application, the structure and training process of the above-mentioned frequency band extension network are described.

[0117] In some possible implementations, combined with Figure 7 As shown, the frequency band extension network may include several deconvolution networks TransposeConv connected in series, which are used to gradually upsample the input narrow-band speech signal (voice domain signal) until a restored full-band speech signal (time domain signal) is finally obtained.

[0118] Figure 7In the network shown, the convolutional layer ConvBlock is used to extract hidden layer features.

[0119] The input of the band extension network can be the Mel spectrum of the narrowband speech signal. Figure 7 The example in the figure is to restore the STFT (Short Time Fourier Transform) feature with a window length of 512 and a frame shift of 256 to a time domain signal, so the time dimension needs to be upsampled by 256 times (frame shift). Figure 7 The network designed in the paper adopts 4 deconvolution networks with a step size of 4 (4×4×4×4=256), and finally obtains the full-band time domain signal.

[0120] Furthermore, the training process of the frequency band extension network is introduced. Figure 7 As shown, an optional training strategy for the band extension network is provided.

[0121] First, get the training dataset.

[0122] The training dataset includes narrowband speech signal samples and their corresponding full-band speech signal labels.

[0123] Specifically, in this embodiment, a clean full-band speech signal may be passed through a bandpass filter to obtain a narrow-band speech signal as a sample, and the clean full-band speech signal may be used as a label to construct a training data set.

[0124] In a possible implementation, in order to improve data richness, this embodiment can adjust the speaking speed of the training data to obtain training data with different speaking speeds to adapt to people with different speaking speeds.

[0125] Furthermore, the obtained training data set is used to train a frequency band extension network.

[0126] Specifically, the short-time Fourier transform features of the input narrowband speech signal sample are extracted and sent to the frequency band expansion network to obtain the time domain signal output by the last deconvolution network, and the output signal of each deconvolution network except the last deconvolution network is inverse Fourier transformed to obtain the time domain signal corresponding to each deconvolution network.

[0127] like Figure 7 As shown, during the training phase, a feature extraction unit can be spliced ​​after each deconvolution network except the last deconvolution network in the frequency band expansion network, such as Figure 7 As shown, a combination of a convolutional layer ConvBlock and a linear layer Linear is used as a feature extraction unit and is spliced ​​after each deconvolution network, so that the output signal of each deconvolution network can be obtained.

[0128] Except for the last deconvolution network which outputs a time domain signal, the output signals of each of the other deconvolution networks are frequency domain signals. Therefore, an inverse Fourier transform can be performed to transform them into time domain signals. At this point, the time domain signal corresponding to each deconvolution network can be obtained.

[0129] In order to obtain more supervision information for the signal in the upsampling process, the output of each deconvolution network can be supervised and trained separately during the training phase. The supervised time domain signal of each deconvolution network can be obtained by frame-shifting the full-band speech signal label according to the sampling multiple of each deconvolution network.

[0130] by Figure 7 As shown in the example, Figure 7 The example in the figure is to restore the STFT (Short Time Fourier Transform) feature with a window length of 512 and a frame shift of 256 to a time domain signal, so the time dimension needs to be upsampled by 256 times (frame shift). Figure 7 The network designed in the paper adopts 4 deconvolution networks with a step size of 4 (4×4×4×4=256), and finally obtains the full-band time domain signal.

[0131] but Figure 7 The supervised time domain signals of the first three deconvolution networks from left to right are obtained by Fourier transforming the full-band speech signal label with frame shifts of 64, 16, and 4. The supervised time domain signal of the last deconvolution network directly uses the full-band speech signal label.

[0132] For each deconvolution network in the band extension network, the Mel spectrum distance loss value of the corresponding time domain signal and the supervision time domain signal is calculated:

[0133]

[0134] Among them, G represents the generator, that is, the frequency band expansion network, φ represents the Mel transform, x represents the true value (that is, the supervised time domain signal), s represents the input narrowband speech signal sample, and E represents the expectation.

[0135] On this basis, the band extension network can be trained based on the total loss function value to obtain the trained band extension network.

[0136] The total loss function value at least includes the above-mentioned Mel spectrum distance loss value.

[0137] The frequency band extension network designed in the embodiment of the present application only needs to be connected in series through several deconvolution networks and convolution layers and linear layers. The network structure is simpler and more convenient to deploy on edge devices such as walkie-talkies, and the chip performance requirements are lower. At the same time, through the training strategy provided in the above embodiment, the output of each deconvolution network can be supervised during the training phase, ensuring that a simple network result can also achieve a good frequency band extension effect, achieving a win-win situation of simple network structure and performance.

[0138] In some possible implementations, the training process of the frequency band extension network in the embodiment of the present application can adopt an adversarial training strategy, using the frequency band extension network as the generator G, and designing one or more discriminators for adversarial training. On this basis, the total loss function value of the training can further include an adversarial loss value.

[0139] Furthermore, in order to improve the training effect, the total loss function can further increase the feature-level discriminator loss value, which is used to represent the distance loss value between the hidden features extracted by the discriminator for the time domain signal generated by the generator and the hidden features extracted by the discriminator for the full-band speech signal label. The feature-level discriminator loss value can achieve more fine-grained supervised training.

[0140] The total loss function of the frequency band extension network of the present application can be expressed as the following formula:

[0141]

[0142] Among them, G represents the generator, D k represents the kth discriminator, there are K discriminators of different scales, L G and L D Represent the total loss function of the generator and the total loss function of the discriminator, L Adv (G; D) represents the adversarial loss function of the generator, L Adv (G; D) represents the adversarial loss function of the discriminator, L FM (G; D) represents the feature-level discriminator loss function, λ fm and λ mel Represent different weight coefficients, N i represents the i-th hidden layer feature, there are T hidden layer features in total, D i (x) represents the i-th hidden feature extracted by the discriminator for the full-band speech signal label x, D i (G(s)) represents the i-th hidden layer feature extracted by the discriminator from the time domain signal s generated by the generator, || || 1 Indicates calculating the distance between features.

[0143] By adopting the training method provided in the above embodiment, the performance of the trained frequency band extension network can be effectively improved, and good results can be achieved while ensuring the lightweight of the frequency band extension network.

[0144] The above embodiments of the present application propose a two-stage noise reduction method and a real-time voice band extension method. A highly robust noise reduction network is designed based on both the uplink and downlink situations, and a frequency band extension scheme is designed for the downlink narrowband signal. This scheme has high versatility in intercom scenarios, and the lightweight network structure also makes the scheme very practical, and can be directly transplanted in intercom products. Compared with other frequency band extension schemes based on deep learning, it has a lighter structure. In addition, the noise reduction network performs targeted optimization on input signals of different bandwidths, and compared with other noise reduction schemes, it is more robust, especially in the downlink low signal-to-noise ratio scenario. Unlike the current situation where the brand and version binding of intercom scenarios is high, the scheme of this application has both uplink noise reduction and downlink noise reduction, which is more versatile. There is no need to design too many product-level constraints when it is applied, and it is a universal solution for intercom scenarios.

[0145] The intercom voice signal processing device provided in an embodiment of the present application is described below. The intercom voice signal processing device described below and the intercom voice signal processing method described above can be referred to each other.

[0146] See also Fig. 9 , Fig. 9 The present invention is a schematic diagram of the structure of a voice signal processing device for an intercom disclosed in an embodiment of the present application.

[0147] like Fig. 9 As shown, the intercom voice signal processing device 900 may include:

[0148] The uplink noise reduction unit 901 is used to perform noise reduction processing on the uplink full-band voice signal through the configured noise reduction network to obtain a clean full-band voice signal after noise reduction, and transmit the clean full-band voice signal externally;

[0149] A downlink noise reduction unit 902 is used to perform noise reduction processing on the received downlink narrowband speech signal through the noise reduction network to obtain a clean narrowband speech signal after noise reduction, wherein the noise reduction network includes a hybrid normalization layer composed of a batch normalization BatchNorm and a layer normalization LayerNorm;

[0150] The output signal determining unit 903 is configured to obtain a target speech signal to be output based on the clean narrow-band speech signal, and output the target speech signal.

[0151] In some possible implementations, the denoising network further includes a dilated convolutional layer and a dual-path gated recurrent unit GRU;

[0152] The atrous convolution layer, the BatchNorm, the dual-path GRU and the LayerNorm are sequentially connected to form a feature extraction network unit in the denoising network.

[0153] In some possible implementations, the training data set used in the noise reduction network training stage includes uplink training data and downlink training data, the uplink training data is a full-band speech signal, the downlink training data is a narrowband speech signal, at least part of the downlink training data is the same source data as the uplink training data, and the same source data is obtained after the uplink training data is processed by a bandpass filter.

[0154] In some possible implementations, the denoising network training phase adopts a random Mask strategy to randomly select a period of time or a frequency band on the input and corresponding output speech signals for zeroing processing.

[0155] In some possible implementations, the denoising network is a network structure obtained by quantizing and compressing the original denoising network after training.

[0156] In some possible implementations, the process of the output signal determining unit obtaining the target speech signal to be output based on the clean narrow-band speech signal includes:

[0157] The clean narrowband speech signal is band-extended using a configured frequency band extension network to obtain a clean full-band speech signal, and the clean full-band speech signal is used as a target speech signal to be output.

[0158] In some possible implementations, the frequency band extension network includes a plurality of deconvolution networks connected in series, which are used to gradually upsample the input narrow-band speech audio domain signal until a restored full-band speech time domain signal is finally obtained.

[0159] In some possible implementations, the apparatus of the present application may further include a model training unit for completing a training process for a frequency band extension network, the process including:

[0160] Acquire a training data set, wherein the training data set includes narrowband speech signal samples and their corresponding full-band speech signal labels;

[0161] Extracting the short-time Fourier transform feature of the narrowband speech signal sample and sending it to the frequency band extension network to obtain the time domain signal output by the last deconvolution network, and performing inverse Fourier transform on the output signal of each deconvolution network except the last deconvolution network to obtain the time domain signal corresponding to each deconvolution network;

[0162] According to the sampling multiple of each deconvolution network, frame shift sampling is performed on the full-band speech signal label to obtain a supervision time domain signal of each deconvolution network;

[0163] For each deconvolution network, calculate the Mel spectrum distance loss value of its corresponding time domain signal and the supervision time domain signal;

[0164] The frequency band extension network is trained based on the total loss function value to obtain a trained frequency band extension network, wherein the total loss function value at least includes the Mel-spectrogram distance loss value.

[0165] In some possible implementations, the training process of the frequency band extension network adopts an adversarial training method, and the total loss function value also includes an adversarial loss value.

[0166] In some possible implementations, the total loss function value also includes a feature-level discriminator loss value, which is used to represent the distance loss value between the hidden features extracted by the discriminator for the time domain signal generated by the generator and the hidden features extracted by the discriminator for the full-band speech signal label.

[0167] In some possible implementations, the frequency band extension network is a network structure obtained by quantizing and compressing the trained original frequency band extension network.

[0168] The present application also provides an electronic device in an embodiment. Fig.10 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application can be a device with a real-time intercom function, such as an intercom. Fig.10 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0169] like Fig.10 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603 to implement the intercom voice signal processing method of the aforementioned embodiment of the present application. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0170] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Fig.10 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0171] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any one of the walkie-talkie voice signal processing methods provided in the embodiments of the present application.

[0172] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any one of the walkie-talkie voice signal processing methods provided in the embodiments of the present application.

[0173] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0174] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0175] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0176] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0177] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

Claims

1. A method for processing intercom voice signals, characterized in that: include: For the uplink full-band voice signal, a noise reduction process is performed through the configured noise reduction network to obtain a clean full-band voice signal after noise reduction, and the clean full-band voice signal is transmitted externally; For the received downlink narrowband speech signal, the noise reduction network is used to perform noise reduction processing to obtain a clean narrowband speech signal after noise reduction, wherein the noise reduction network includes a hybrid normalization layer composed of a batch normalization BatchNorm and a layer normalization LayerNorm; A target speech signal to be output is obtained based on the clean narrow-band speech signal, and the target speech signal is output.

2. The method according to claim 1, characterized in that The denoising network also includes a hole convolution layer and a dual-path gated recurrent unit GRU; The atrous convolution layer, the BatchNorm, the dual-path GRU and the LayerNorm are sequentially connected to form a feature extraction network unit in the denoising network.

3. The method according to claim 1, characterized in that The training data set used in the noise reduction network training stage includes uplink training data and downlink training data, the uplink training data is a full-band speech signal, the downlink training data is a narrowband speech signal, at least part of the downlink training data is the same source data as the uplink training data, and the same source data is obtained by processing the uplink training data through a bandpass filter.

4. The method according to claim 1, characterized in that: The denoising network training phase adopts a random Mask strategy to randomly select a period of time or a frequency band on the input and corresponding output speech signals for zeroing.

5. The method according to claim 1, characterized in that: The denoising network is a network structure obtained by quantizing and compressing the original denoising network after training.

6. The method according to any one of claims 1 to 5, characterized in that: The process of obtaining a target speech signal to be output based on the clean narrow-band speech signal includes: The clean narrowband speech signal is band-extended using a configured frequency band extension network to obtain a clean full-band speech signal, and the clean full-band speech signal is used as a target speech signal to be output.

7. The method according to claim 6, characterized in that The frequency band extension network includes a plurality of deconvolution networks connected in series, which are used to gradually upsample the input narrow-band speech audio domain signal until a restored full-band speech time domain signal is finally obtained.

8. The method according to claim 7, characterized in that The training process of the frequency band extension network includes: Acquire a training data set, wherein the training data set includes narrowband speech signal samples and their corresponding full-band speech signal labels; Extracting the short-time Fourier transform feature of the narrowband speech signal sample and sending it to the frequency band extension network to obtain the time domain signal output by the last deconvolution network, and performing inverse Fourier transform on the output signal of each deconvolution network except the last deconvolution network to obtain the time domain signal corresponding to each deconvolution network; According to the sampling multiple of each deconvolution network, frame shift sampling is performed on the full-band speech signal label to obtain a supervision time domain signal of each deconvolution network; For each deconvolution network, calculate the Mel spectrum distance loss value of its corresponding time domain signal and the supervision time domain signal; The frequency band extension network is trained based on the total loss function value to obtain a trained frequency band extension network, wherein the total loss function value at least includes the Mel-spectrogram distance loss value.

9. The method according to claim 8, characterized in that The training process of the frequency band extension network adopts an adversarial training method, and the total loss function value also includes an adversarial loss value.

10. The method according to claim 9, characterized in that The total loss function value also includes a feature-level discriminator loss value, which is used to represent the distance loss value between the hidden features extracted by the discriminator for the time domain signal generated by the generator and the hidden features extracted by the discriminator for the full-band speech signal label.

11. The method according to claim 6, characterized in that The frequency band extension network is a network structure obtained by quantizing and compressing the trained original frequency band extension network.

12. A walkie-talkie voice signal processing device, characterized in that: include: An uplink noise reduction unit, configured to perform noise reduction processing on the uplink full-band voice signal through a configured noise reduction network to obtain a clean full-band voice signal after noise reduction, and transmit the clean full-band voice signal externally; A downlink noise reduction unit, configured to perform noise reduction processing on the received downlink narrowband speech signal through the noise reduction network to obtain a clean narrowband speech signal after noise reduction, wherein the noise reduction network includes a hybrid normalization layer composed of a batch normalization BatchNorm and a layer normalization LayerNorm; The output signal determination unit is used to obtain a target speech signal to be output based on the clean narrow-band speech signal, and output the target speech signal.

13. An electronic device, characterized in that: include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the intercom voice signal processing method according to any one of claims 1 to 11.

14. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the intercom voice signal processing method according to any one of claims 1 to 11 is implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, each step of the intercom voice signal processing method according to any one of claims 1 to 11 is implemented.

Images