Voice sending method and device, voice receiving method and device, and electronic equipment

Abstract

The invention provides a voice sending method and apparatus, a voice receiving method and apparatus, and an electronic device. The voice sending method comprises the steps of extracting low-frequency feature information from an input voice signal by using a first feature extraction network; and sending a voice signal corresponding to the low-frequency feature information to a receiving end.

Description

[0001] This application is a divisional application of the invention patent application with the application date of September 18, 2020, application number 202010988943.5, and invention name "Voice sending method and device, voice receiving method and device, electronic device". Technical Field

[0002] The present application generally relates to the field of voice calls, and more specifically, to a method and device for sending voice based on a neural network, a method and device for receiving voice, and an electronic device. Background Art

[0003] The current mobile communication network (2G / 3G / 4G) only supports narrowband (NB) / wideband (WB) voice sampling rate, but the current mainstream terminal devices already support ultra-wideband (SWB) voice signal collection and recording. Therefore, the current mobile communication network cannot give full play to the performance of the terminal equipment.

[0004] Since the bandwidth range of human speech is relatively wide, while the bandwidth supported by mobile communication networks is limited. Therefore, in mobile communications, the voice transmitter first needs to downsample the collected voice of the speaker to the sampling rate supported by the communication network, and then encode the downsampled voice and send it through the mobile communication network. At the same time, the receiving end can decode the received signal and play it directly to the recipient, but in order to improve the user experience, the received signal can also be extended to a higher bandwidth and sampling rate to make the sound more comfortable. Therefore, the prior art uses downsampling technology and bandwidth extension technology at the voice transmitter and receiver to convert the sampling rate of the voice signal.

[0005] However, in the prior art, the effect of sampling rate conversion when the terminal sends and receives voice is poor, and it is impossible to provide users with a high-quality auditory experience. Summary of the invention

[0006] In order to at least solve the above problems existing in the prior art, the present invention provides a method and device for sending speech based on a neural network and a method and device for receiving speech.

[0007] One aspect of the present invention is to provide a method for sending a voice, comprising: extracting low-frequency feature information from an input voice signal using a first feature extraction network; and sending a voice signal corresponding to the low-frequency feature information to a receiving end. The above operation can reduce the loss of low-frequency information in the AI-based downsampling process, and at the same time provide a voice receiving device with more information to obtain better upsampling performance in the voice receiving device.

[0008] The first feature extraction network includes at least one first feature extraction module and at least one second feature extraction module, wherein the step of extracting low-frequency feature information from the input speech signal using the first feature extraction network includes: using the at least one first feature extraction module to extract the speech feature information of the input speech signal; using the at least one second feature extraction module to obtain low-frequency feature information based on the extracted speech feature information.

[0009] At least one feature extraction module in the first feature extraction network extracts features from input information using at least two convolution processing parameters, and outputs the extracted feature information.

[0010] The at least two convolution processing parameters include at least two of the following: a convolution processing parameter corresponding to a receptive field between adjacent sampling points of a speech signal, a convolution processing parameter corresponding to a receptive field of one pitch length, and a convolution processing parameter corresponding to a receptive field of at least two pitch lengths.

[0011] The convolution processing parameters include a convolution kernel size corresponding to the convolution operation.

[0012] The step of sending a speech signal corresponding to the low-frequency feature information to a receiving end includes: using a first feature fusion network to fuse all low-frequency feature information output by a first feature extraction network to obtain a speech signal corresponding to the low-frequency feature information; and sending the speech signal corresponding to the low-frequency feature information to a receiving end.

[0013] The low-frequency feature information extracted by the first feature extraction network includes relevant information between high-frequency features and low-frequency features.

[0014] The voice sending method may also include: receiving a voice signal sent by the receiving end; extracting low-frequency feature information from the received voice signal and restoring high-frequency feature information using a second feature extraction network; and outputting a voice signal containing the low-frequency feature information and the high-frequency feature information.

[0015] Another aspect of the present invention is to provide a speech receiving method, which may include: receiving a speech signal sent by a transmitting end; using a second feature extraction network to extract low-frequency feature information from the received speech signal and restore high-frequency feature information; and outputting a speech signal containing the low-frequency feature information and the high-frequency feature information.

[0016] The speech receiving method may further include: before the step of extracting low-frequency feature information from the received speech signal and restoring high-frequency feature information using a second feature extraction network, expanding the data scale of the received speech signal by performing data replication on the received speech signal.

[0017] The step of using a second feature extraction network to extract low-frequency feature information from a received speech signal and restore high-frequency feature information may include: using a low-frequency feature extraction network in the second feature extraction network to extract low-frequency feature information from the received speech signal, wherein the low-frequency feature information includes relevant information between high-frequency features and low-frequency features; using a high-frequency feature extraction network in the second feature extraction network to restore high-frequency feature information based on the low-frequency feature information and fuse the high-frequency feature information with the low-frequency feature information to obtain feature information including high-frequency feature information and low-frequency feature information.

[0018] The low-frequency feature extraction network includes at least one third feature extraction module and at least one fourth feature extraction module, wherein the step of extracting low-frequency feature information from the received speech signal using the low-frequency feature extraction network in the second feature extraction network includes: extracting speech feature information of the received speech signal using the at least one third feature extraction module; obtaining low-frequency feature information according to the extracted speech feature information using the at least one fourth feature extraction module, wherein the high-frequency feature extraction network includes at least one fifth feature extraction module and at least one sixth feature extraction module, wherein the step of restoring high-frequency feature information according to the low-frequency feature information and fusing the high-frequency feature information with the low-frequency feature information using the high-frequency feature extraction network in the second feature extraction network includes: restoring high-frequency feature information according to the low-frequency feature information using the at least one fifth feature extraction module; fusing the high-frequency feature information with the corresponding low-frequency feature information extracted by the fourth feature extraction module using at least one sixth feature extraction module to obtain feature information including the high-frequency feature information and the low-frequency feature information.

[0019] At least one feature extraction module in the second feature extraction network extracts features from the input information through at least two convolution processing parameters, and outputs the extracted feature information; and / or the at least one sixth feature extraction module fuses the high-frequency feature information corresponding to the at least two convolution processing parameters respectively input, and the low-frequency feature information extracted by the corresponding fourth feature extraction module according to the corresponding convolution processing parameters, and outputs feature information containing the high-frequency feature information and the low-frequency feature information.

[0020] The convolution processing parameters include the convolution kernel size corresponding to the convolution operation.

[0021] The second feature extraction network includes at least one seventh feature extraction module and at least one eighth feature extraction module,

[0022] The step of using a second feature extraction network to extract low-frequency feature information from a received speech signal and restore high-frequency feature information includes: using at least one seventh feature extraction module to extract speech feature information of the received speech signal; using at least one eighth feature extraction module to extract low-frequency feature information containing relevant information between high-frequency features and low-frequency features based on the extracted speech feature information, and restore high-frequency feature information to obtain feature information containing high-frequency feature information and low-frequency feature information.

[0023] The at least one seventh feature extraction module extracts features from the input information through at least two convolution processing parameters, and outputs the extracted feature information; and / or the at least one eighth feature extraction module extracts low-frequency feature information from the input information through at least two deconvolution processing parameters, and restores high-frequency feature information to obtain feature information containing high-frequency feature information and low-frequency feature information.

[0024] The step of outputting a speech signal containing the low-frequency feature information and the high-frequency feature information includes: using a second feature fusion network to fuse the feature information containing the high-frequency feature information and the low-frequency feature information output by the second feature extraction network, to obtain a speech signal corresponding to the feature information containing the high-frequency feature information and the low-frequency feature information; and outputting the speech signal.

[0025] The step of using a second feature extraction network to extract low-frequency feature information from a received speech signal and restore high-frequency feature information may include: using a low-frequency feature extraction network in the second feature extraction network to extract low-frequency feature information from the received speech signal; using a low-frequency feature fusion network in the second feature fusion network to fuse the low-frequency feature information output by the low-frequency feature extraction network to obtain a low-frequency time domain signal with low-frequency repair.

[0026] The low-frequency feature extraction network may include at least one ninth feature extraction module, wherein the step of extracting low-frequency feature information from the received speech signal using the low-frequency feature extraction network in the second feature extraction network may include: extracting the low-frequency feature information of the received speech signal using the at least one ninth feature extraction module.

[0027] The step of using a second feature extraction network to extract low-frequency feature information from the received speech signal and restore high-frequency feature information may also include: upsampling the low-frequency time domain signal repaired by the low frequency to obtain a low-frequency time domain signal with an increased sampling rate; using a high-frequency feature extraction network in the second feature extraction network to extract low-frequency feature information containing relevant information between high-frequency features and low-frequency features from the low-frequency time domain signal with an increased sampling rate, and restoring high-frequency feature information from the low-frequency feature information based on the relevant information; using a high-frequency feature fusion network in the second feature fusion network to fuse the high-frequency feature information output by the high-frequency feature extraction network to obtain a high-frequency time domain signal.

[0028] The high-frequency feature extraction network may include at least one tenth feature extraction module, wherein the step of extracting low-frequency feature information containing relevant information between high-frequency features and low-frequency features from the low-frequency time domain signal with increased sampling rate using the high-frequency feature extraction network in the second feature extraction network and restoring high-frequency feature information from the low-frequency feature information according to the relevant information may include: extracting low-frequency feature information containing relevant information between high-frequency features and low-frequency features from the low-frequency time domain signal with increased sampling rate using the at least one tenth feature extraction module, and restoring high-frequency feature information from the low-frequency feature information according to the relevant information.

[0029] The at least one ninth feature extraction module can respectively perform feature extraction on the input information through at least two convolution processing parameters, and output the extracted low-frequency feature information, wherein the at least one tenth feature extraction module can respectively extract low-frequency feature information on the input information through at least two convolution processing parameters, and restore high-frequency feature information.

[0030] The step of outputting a speech signal containing the low-frequency feature information and the high-frequency feature information may include: performing time domain summing processing on the low-frequency time domain signal with the increased sampling rate and the high-frequency time domain signal to obtain a final speech signal; and outputting the speech signal.

[0031] Another aspect of the present invention is to provide a speech sending device, which may include: a low-frequency feature information extraction unit, configured to extract low-frequency feature information from an input speech signal using a first feature extraction network; and a sending unit, configured to send a speech signal corresponding to the low-frequency feature information to a receiving end.

[0032] The first feature extraction network includes at least one first feature extraction module and at least one second feature extraction module, wherein the low-frequency feature information extraction unit includes: a speech feature information extraction unit, configured to use the at least one first feature extraction module to extract speech feature information of an input speech signal; and a low-frequency feature information acquisition unit, using the at least one second feature extraction module to obtain low-frequency feature information based on the extracted speech feature information.

[0033] The low-frequency feature information extraction unit is configured to use at least one feature extraction module in the first feature extraction network to perform feature extraction on the input information respectively through at least two convolution processing parameters, and output the extracted feature information.

[0034] The at least two convolution processing parameters include at least two of the following: a convolution processing parameter corresponding to a receptive field between adjacent sampling points of a speech signal, a convolution processing parameter corresponding to a receptive field of one pitch length, and a convolution processing parameter corresponding to a receptive field of at least two pitch lengths.

[0035] The convolution processing parameters include a convolution kernel size corresponding to the convolution operation.

[0036] The sending unit is configured to send the speech signal corresponding to the low-frequency feature information to the receiving end through the following operations: using the first feature fusion network to fuse all the low-frequency feature information output by the first feature extraction network to obtain the speech signal corresponding to the low-frequency feature information; and sending the speech signal corresponding to the low-frequency feature information to the receiving end.

[0037] The low-frequency feature information extracted by the first feature extraction network includes relevant information between high-frequency features and low-frequency features.

[0038] The voice sending device may also include: a receiving unit, configured to receive the voice signal sent by the receiving end; a high-frequency feature information recovery unit, configured to extract low-frequency feature information from the received voice signal and recover high-frequency feature information using a second feature extraction network; and an output unit, configured to output a voice signal containing the low-frequency feature information and the high-frequency feature information.

[0039] Another aspect of the present invention provides a speech receiving device, which may include: a receiving unit, configured to receive a speech signal sent by a transmitting end; a high-frequency feature information recovery unit, configured to extract low-frequency feature information from the received speech signal and recover high-frequency feature information using a second feature extraction network; and an output unit, outputting a speech signal containing the low-frequency feature information and the high-frequency feature information.

[0040] The speech receiving device may also include: a preprocessing unit, which is configured to expand the data scale of the received speech signal by performing data replication on the received speech signal before the step of extracting low-frequency feature information from the received speech signal and restoring high-frequency feature information using a second feature extraction network.

[0041] The high-frequency feature information recovery unit is configured to extract low-frequency feature information from the received speech signal and recover high-frequency feature information using the second feature extraction network through the following operations: using the low-frequency feature extraction network in the second feature extraction network to extract low-frequency feature information from the received speech signal, the low-frequency feature information containing relevant information between the high-frequency feature and the low-frequency feature; using the high-frequency feature extraction network in the second feature extraction network to recover high-frequency feature information based on the low-frequency feature information and fuse the high-frequency feature information with the low-frequency feature information to obtain feature information containing high-frequency feature information and low-frequency feature information.

[0042] The low-frequency feature extraction network includes at least one third feature extraction module and at least one fourth feature extraction module, wherein the high-frequency feature information recovery unit is configured to extract low-frequency feature information from the received speech signal using the low-frequency feature extraction network in the second feature extraction network through the following operations: using the at least one third feature extraction module to extract speech feature information of the received speech signal; using the at least one fourth feature extraction module to obtain low-frequency feature information according to the extracted speech feature information, wherein the high-frequency feature extraction network includes at least one fifth feature extraction module and at least one sixth feature extraction module, and the high-frequency feature information recovery unit is configured to recover high-frequency feature information according to the low-frequency feature information and fuse the high-frequency feature information and the low-frequency feature information using the high-frequency feature extraction network in the second feature extraction network through the following operations: using the at least one fifth feature extraction module to recover high-frequency feature information according to the low-frequency feature information; using at least one sixth feature extraction module to fuse the high-frequency feature information and the corresponding low-frequency feature information extracted by the fourth feature extraction module to obtain feature information including the high-frequency feature information and the low-frequency feature information.

[0043] At least one feature extraction module in the second feature extraction network extracts features from the input information through at least two convolution processing parameters, and outputs the extracted feature information, and / or the at least one sixth feature extraction module fuses the high-frequency feature information corresponding to the at least two convolution processing parameters input respectively with the low-frequency feature information extracted by the corresponding fourth feature extraction module according to the corresponding convolution processing parameters, and outputs feature information containing the high-frequency feature information and the low-frequency feature information.

[0044] The convolution processing parameters include the convolution kernel size corresponding to the convolution operation.

[0045] The second feature extraction network includes at least one seventh feature extraction module and at least one eighth feature extraction module, and the high-frequency feature information recovery unit is configured to use the second feature extraction network to extract low-frequency feature information from the received speech signal and recover high-frequency feature information through the following operations: using at least one seventh feature extraction module to extract speech feature information of the received speech signal; using at least one eighth feature extraction module to extract low-frequency feature information containing relevant information between high-frequency features and low-frequency features based on the extracted speech feature information, and recover the high-frequency feature information to obtain feature information containing high-frequency feature information and low-frequency feature information.

[0046] The at least one seventh feature extraction module extracts features from the input information through at least two convolution processing parameters, and outputs the extracted feature information; and / or the at least one eighth feature extraction module extracts low-frequency feature information from the input information through at least two deconvolution processing parameters, and restores high-frequency feature information to obtain feature information containing high-frequency feature information and low-frequency feature information.

[0047] The output unit is configured to output a speech signal containing the low-frequency feature information and the high-frequency feature information by the following operations: using a second feature fusion network to fuse the feature information containing the high-frequency feature information and the low-frequency feature information output by the second feature extraction network, to obtain a speech signal corresponding to the feature information containing the high-frequency feature information and the low-frequency feature information; and output the speech signal.

[0048] The high-frequency feature information recovery unit can be configured to extract low-frequency feature information from the received speech signal and restore high-frequency feature information using the second feature extraction network through the following operations: extract low-frequency feature information from the received speech signal using the low-frequency feature extraction network in the second feature extraction network; and fuse the low-frequency feature information output by the low-frequency feature extraction network using the low-frequency feature fusion network in the second feature fusion network to obtain a low-frequency repaired low-frequency time domain signal.

[0049] The low-frequency feature extraction network may include at least one ninth feature extraction module, wherein the high-frequency feature information recovery unit may be configured to extract low-frequency feature information from the received speech signal using the low-frequency feature extraction network in the second feature extraction network by the following operation: using the at least one ninth feature extraction module to extract the low-frequency feature information of the received speech signal.

[0050] The high-frequency feature information recovery unit can also be configured to extract low-frequency feature information and restore high-frequency feature information from the received speech signal using the second feature extraction network through the following operations: upsampling the low-frequency repaired low-frequency time domain signal to obtain a low-frequency time domain signal with an increased sampling rate; using the high-frequency feature extraction network in the second feature extraction network to extract low-frequency feature information containing relevant information between high-frequency features and low-frequency features from the low-frequency time domain signal with an increased sampling rate, and restore high-frequency feature information from the low-frequency feature information based on the relevant information; using the high-frequency feature fusion network in the second feature fusion network to fuse the high-frequency feature information output by the high-frequency feature extraction network to obtain a high-frequency time domain signal.

[0051] The high-frequency feature extraction network may include at least one tenth feature extraction module, wherein the high-frequency feature information recovery unit may be configured to extract low-frequency feature information containing the relevant information using the high-frequency feature extraction network in the second feature extraction network and recover the high-frequency feature information based on the relevant information by the following operations: using the at least one tenth feature extraction module to extract low-frequency feature information containing relevant information between high-frequency features and low-frequency features from the low-frequency time domain signal with increased sampling rate, and recover high-frequency feature information from the low-frequency feature information based on the relevant information.

[0052] The at least one ninth feature extraction module can respectively perform feature extraction on the input information through at least two convolution processing parameters, and output the extracted low-frequency feature information, wherein the at least one tenth feature extraction module can respectively extract low-frequency feature information on the input information through at least two convolution processing parameters, and restore high-frequency feature information.

[0053] The output unit can be configured to output a speech signal containing the low-frequency feature information and the high-frequency feature information by the following operations: performing time domain summing processing on the low-frequency time domain signal with the increased sampling rate and the high-frequency time domain signal to obtain a final speech signal; and outputting the speech signal.

[0054] An aspect of the present invention is to provide a computer-readable storage medium storing a program, wherein the program, when executed by a processor, implements the above-mentioned voice sending method and / or voice receiving method.

[0055] An aspect of the present invention is to provide a computer including a readable medium storing a computer program, wherein the computer program implements the above-mentioned voice sending method and / or voice receiving method when executed by the computer.

[0056] One aspect of the present invention is to provide an electronic device, which includes a memory and a processor; the memory stores a computer program; the processor is used to execute the voice sending method and / or voice receiving method when running the computer program. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] These and / or other aspects and advantages of the present application will become clearer and easier to understand from the following detailed description of the embodiments of the present application in conjunction with the accompanying drawings, in which:

[0058] Figure 1 is a schematic diagram showing the method of downsampling a sampled SWB signal to a WB signal in the prior art;

[0059] Figure 2 is a general flow chart showing a method for sending a voice according to an exemplary embodiment of the present invention;

[0060] Figure 3 is a diagram showing an exemplary embodiment of the present invention. Figure 2 A schematic diagram of an example of an AI-based downsampling process corresponding to the voice sending method shown in ;

[0061] Figure 4 is a schematic diagram showing that different convolution kernel sizes cover features of different scales according to an exemplary embodiment of the present invention;

[0062] Figure 5 is a diagram showing an exemplary embodiment of the present invention. Figure 2 A schematic diagram of another example of an AI-based downsampling process corresponding to the voice transmission method shown in ;

[0063] Figure 6 is a general flow chart showing a method for receiving a voice according to an exemplary embodiment of the present invention;

[0064] Figure 7 is a specific flow chart showing a method for receiving a voice according to an exemplary embodiment of the present invention;

[0065] Figure 8 is a diagram showing an exemplary embodiment of the present invention. Figure 6 A schematic diagram of an example of an AI-based upsampling process corresponding to the voice receiving method shown in ;

[0066] Fig. 9 is a specific flow chart showing a method for receiving a voice according to another exemplary embodiment of the present invention;

[0067] Fig.10a is a diagram showing another exemplary embodiment according to the present invention. Figure 6The speech receiving method shown in the figure corresponds to an example of an AI-based upsampling process;

[0068] Fig.10b is a specific flow chart showing a method for receiving a voice according to another exemplary embodiment of the present invention;

[0069] Fig.10c is a diagram showing another exemplary embodiment according to the present invention. Figure 6 A schematic diagram of an example of an AI-based upsampling process corresponding to the voice receiving method shown in ;

[0070] Fig.10d It is shown that according to the present invention Fig.10b A schematic diagram showing a comparison between an AI-based upsampling method and a traditional upsampling method;

[0071] Fig.10e is a diagram showing an exemplary embodiment of the present invention. Figure 6 A schematic diagram of another example of an AI-based upsampling process corresponding to the speech receiving method shown in ;

[0072] like Fig.10f is an overall schematic diagram showing an AI-based downsampling method and an AI-based upsampling method according to an exemplary embodiment of the present invention;

[0073] like Figure 10g is a schematic diagram showing a comparison between an AI-based downsampling method according to an exemplary embodiment of the present invention and a traditional downsampling method;

[0074] like Fig.10h is a network architecture showing a joint Scale Down and Scale Up suitable for voice calls according to an exemplary embodiment of the present invention;

[0075] Fig.11 is a block diagram showing a configuration of a speech transmitting apparatus according to an exemplary embodiment of the present invention;

[0076] Fig.12 is a block diagram showing a configuration of a low-frequency feature information extraction unit according to an exemplary embodiment of the present invention;

[0077] Fig.13 is a block diagram showing the configuration of a voice receiving apparatus according to an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0078] Reference will now be made in detail to exemplary embodiments of the present disclosure, examples of which are shown in the accompanying drawings, wherein like reference numerals refer to like parts throughout. The embodiments will be described below with reference to the accompanying drawings to facilitate explanation of the present invention.

[0079] In the embodiments of the present application, the terminal to which the voice sending method or voice receiving method proposed in the embodiments of the present application is applied may include, but is not limited to, terminal devices such as mobile phones, PADs, and smart watches that can send or receive voice. The terminal can send or receive voice in a voice call scenario according to the method proposed in the embodiments of the present application. For example, the terminal, as the sending end in the voice call, can perform sampling rate conversion according to the voice sending method proposed in the embodiments of the present application, and encode the voice and send it to the receiving end. The terminal can also serve as the receiving end in the voice call, and decode the code stream corresponding to the received voice signal according to the voice sending method proposed in the embodiments of the present application to obtain a voice signal, and perform sampling rate conversion on the voice signal and play it to the user. In addition, the method proposed in the embodiments of the present application can also be applied in other scenarios of sending or receiving voice, and this application does not make further restrictions.

[0080] In the embodiment of the present application, the sampling rate of the NB voice signal may be 8 kHz; the sampling rate of the WB voice signal may be 16 kHz; and the sampling rate of the SWB voice signal may be 32 kHz.

[0081] The process of converting a speech signal from a lower sampling rate to a higher sampling rate can be called upsampling (UpSampling), which can also be called bandwidth expansion, super-resolution, or Scale Up, such as converting a speech signal from WB to SWB.

[0082] The process of converting a speech signal from a higher sampling rate to a lower sampling rate can be called downsampling (DownSampling), which can also be called Scale Down, for example, converting a speech signal from SWB to WB, or from SWB to NB, or from WB to NB.

[0083] At present, in terms of downsampling of speech signals, traditional signal processing methods are generally used to downsample the collected SWB signals to WB or NB signals. However, the filter-based signal processing downsampling method requires the design of the range of the passband and stopband as well as the characteristics of the transition band. Due to the uneven amplitude of the passband frequency characteristics and the attenuation of the low-frequency signal by the transition band frequency characteristics, the low-frequency part of the WB signal finally obtained has a certain loss relative to the low-frequency part of the original SWB signal. This kind of loss will reduce the intelligibility of the sound. Figure 1 As shown in FIG. 1 , for the unvoiced pronunciation in the sentence (such as the pronunciation of “s” in “ask” and “test”), the pronunciation is mainly concentrated in the high-frequency part, but the filter-based signal processing downsampling method will cause the high-frequency part to be lost (such as Figure 1 The part above the horizontal line 1 in the "WB" diagram on the left side of the figure) and a part of the low-frequency part near the cutoff frequency will be lost (such as Figure 1The dark area at the top of the “NB” diagram on the right side of the figure) ultimately causes the pronunciation of “s” in the down-sampling result to be greatly weakened, resulting in the loss of speech information.

[0084] In order to preserve the low-frequency information of the SWB signal as much as possible and reduce the loss of signal intelligibility, it is necessary to adopt a signal processing method different from the traditional filter-based method.

[0085] In addition, in terms of bandwidth expansion, existing bandwidth expansion algorithms are all based on blind bandwidth expansion, that is, in the design of bandwidth expansion algorithms, it is considered that the receiving end is completely unaware of how downsampling is performed at the transmitting end (that is, it is considered that the process of how the original high-bandwidth signal is converted into a low-bandwidth signal is unknown). In other words, the existing bandwidth expansion algorithms do not consider the characteristics of the voice signal downsampling end and the correlation between high-frequency and low-frequency signals, so they cannot achieve optimization. However, since mobile communications are symmetrical, downsampling is completed at the transmitting end, and upsampling or bandwidth expansion is completed at the receiving end. If the receiving end knows the algorithm characteristics of the downsampling processing at the transmitting end when upsampling, the receiving end can make full use of the algorithm characteristics of the downsampling to perform upsampling processing, which will help improve the performance of upsampling. Among them, upsampling is also called bandwidth expansion or superresolution.

[0086] In addition, when using neural networks to achieve upsampling, a single-scale convolution operation is usually used in each layer of the neural network. However, since the frequency periodic characteristics of speech signals are affected by many factors such as different speakers, age, gender, speech content, etc., if a single-scale convolution operation is used in each layer, it is impossible to cover the variation range of the speech signal. This makes it impossible to cover the diverse speech audio features when using neural networks to obtain speech signal feature information, thereby reducing the performance of downsampling and upsampling.

[0087] The existing bandwidth extension algorithms used in calls have the following problems: 1) Due to the frequency domain Fourier transform and inverse transform and spectrum amplitude, frequency domain operations cannot avoid algorithm delay. This delay limits the application of frequency domain bandwidth extension as a post-processing module in existing networks. 2) Although the time domain method avoids the algorithm delay of frame stacking and accumulation caused by the time-frequency conversion of the time-frequency domain model. However, the model is not causal and is too complex and computationally expensive, so it cannot be deployed in voice calls. 3) In network training, although the time-frequency network and time-frequency loss function can obtain better time-domain and frequency-domain indicator results, it is difficult to obtain the appropriate weighted summation calculation of time-domain and frequency-domain losses stably in diverse data training. Moreover, considering the impact of codec loss on the running waveform, the time domain similarity will fail and the time domain loss function will be difficult to converge. 4) In the voice call system, the downsampling and upsampling (bandwidth extension) processing are completed by software and algorithms on the user device, but the existing algorithm implementation is only implemented at the receiving end of the call.

[0088] Therefore, there is a need for a method and device that can overcome the above problems and enable a receiving end to provide a user with a better auditory experience.

[0089] Figure 2 is a general flow chart showing a voice sending method according to an exemplary embodiment of the present application. Figure 2 The speech transmission method shown in is a speech transmission method based on a neural network, wherein the neural network may be a convolutional neural network (CNN). That is, the speech transmission method may be as follows: Figure 3 The CNN-based speech transmission method shown in Figure 3 As shown in , the voice transmission method may include two processes: a multi-convolution kernel scale feature extraction (low frequency) process and a feature fusion process, wherein the multi-convolution kernel scale feature extraction may also be referred to as multi-filter scale feature extraction. The multi-convolution kernel scale feature extraction (low frequency) process is composed of a feature extraction layer and a downsampling layer of a CNN, and the feature fusion process is composed of a feature fusion layer of a CNN. Figure 2 Step S110 in may correspond to Figure 3 In the multi-convolution kernel scale feature extraction (low frequency) process, step S120 may correspond to Figure 3 However, the present invention is not limited thereto, and the feature extraction network in the sending method of the present invention can also be implemented using deep neural networks (DNN).

[0090] This application proposes a joint Scale Down and Scale Up network architecture (which can be called X-net) suitable for voice calls, which makes full use of the downsampling behavior on the device to improve the quality of voice calls. The model is trained using a two-step training method. The Scale Down and Scale Up networks are deployed at the transmitting and receiving ends respectively to complete the downsampling and upsampling (bandwidth expansion) of the voice signal. The two-step training uses different loss functions to improve the perceived similarity. Objective test results show that X-net can achieve significant improvements over the method that only performs traditional bandwidth extension (Bandwidth Extension, BWE), and subjective test results show that the solution of this application can improve MOS scores and voice quality.

[0091] In order to facilitate understanding of the present invention, in the following description, Figure 3 The processing process of the CNN-based speech transmission method is shown in Figure 2 The voice sending method is described in detail.

[0092] like Figure 2 As shown in , in step S110, a first feature extraction network is used to extract low-frequency feature information from an input speech signal.

[0093] The first feature extraction network may be composed of one or more feature extraction modules, and multiple feature extraction modules may be cascaded. Each feature extraction module may include one or more convolution operations. Among them, each feature extraction module may perform multi-scale convolution operations, that is, feature extraction of input information is performed respectively through at least two convolution processing parameters and the extracted feature information is output. In other words, the feature extraction module may include multiple convolution operations, and the convolution processing parameters of at least two convolution operations are different, and different convolution processing parameters correspond to different receptive fields, corresponding to different frequency domain resolution capabilities, that is, corresponding to different sampling scales. Among them, the above-mentioned convolution processing parameters may include the convolution kernel size corresponding to the convolution operation, the convolution kernel size of the convolution operation is different, the receptive field corresponding to the convolution layer operation is different, the frequency domain resolution capability is different, and different receptive fields can extract features of different scales (also referred to as granularity), rather than a single scale filter, so it can avoid the influence of many factors such as different speakers, age, gender, and speech content, and can cover a large range of changes in the voice signal. The feature information of the voice signal obtained by the solution of the embodiment of the present application can cover a variety of voice audio features, thereby improving the performance of downsampling.

[0094] In the feature extraction module, the scale of a convolution operation can correspond to one or more convolution operations. That is to say, in the feature extraction module, the convolution kernel sizes of each convolution operation can be different, or at least two convolution operations can have the same convolution kernel size, that is, corresponding to the same scale.

[0095] Since the first feature extraction network needs to extract low-frequency feature information from the input voice signal, the input of the first feature extraction network is the voice signal collected by the transmitter, and its output is low-frequency feature information. The output low-frequency feature information is sent to the voice receiving end through the communication network. Therefore, the voice sampling rate of the first feature extraction network is determined by the sampling rate of the input voice signal and the bandwidth supported by the communication network. If the voice signal collected by the transmitter is a SWB voice signal, and the communication network between the terminals supports the bandwidth of WB, the first feature extraction network needs to downsample from SWB to NB.

[0096] When the first feature extraction network contains multiple feature extraction modules, the step sizes of the multiple feature extraction modules may be different, and the step size of the feature extraction module corresponds to the sampling rate. For example, the step size of feature extraction module A and feature extraction module B may be equal, both are 1. At this time, the sampling rate of the output and the output has not changed. The step size of feature extraction module C is 2, and it performs a sampling rate conversion.

[0097] If the feature extraction module includes multiple convolution operations, the step sizes of the convolution operations may be the same.

[0098] The embodiment of the present application proposes that the speech signal to be sent to the receiving end is downsampled by a neural network (also referred to as a neural network model, a deep learning model, a deep neural network model, a deep network model, etc.), which can effectively reduce the loss of low-frequency feature information. Among them, the convolution operation on the input data can be considered as performing a filtering process, such as performing a finite impulse response (FIR) filtering.

[0099] Specifically, the first feature extraction network may include at least one first feature extraction module and at least one second feature extraction module. Figure 3 As shown in , the first feature extraction network may correspond to Figure 3 In the multi-convolution kernel scale feature extraction (low frequency) process, the at least one first feature extraction module may correspond to Figure 3 The feature extraction layer in the at least one second feature extraction module may correspond to Figure 3 The first feature extraction network may include one or more feature extraction layers, and may also include one or more downsampling layers. Figure 3 The first feature extraction network including a feature extraction layer and a downsampling layer is taken as an example for explanation.

[0100] Specifically, the step of extracting low-frequency feature information from the input speech signal using the first feature extraction network may include: extracting speech feature information of the input speech signal using the at least one first feature extraction module.

[0101] Among them, at least one first feature extraction module in the first feature extraction network can perform feature extraction on the input information and output the extracted feature information through at least two convolution processing parameters. Among them, the at least two convolution processing parameters include at least two of the following: a convolution processing parameter corresponding to the receptive field between adjacent sampling points of the speech signal, a convolution processing parameter corresponding to the receptive field of one pitch length, and a convolution processing parameter corresponding to the receptive field of at least two pitch lengths, wherein the convolution processing parameter may include a convolution kernel size corresponding to the convolution operation.

[0102] Specifically, if Figure 3 As shown in Figure 3The multi-scale convolution operation in the feature extraction layer is learned at a high-frequency sampling rate to learn the high-frequency and low-frequency features of the speech. That is, the speech feature information includes high-frequency and low-frequency features. Specifically, in order to learn the characteristics of different high and low-frequency sub-bands, the convolution operation can adopt different convolution kernel sizes. This is because different convolution kernel sizes are equivalent to different frequency domain resolution capabilities. The larger the convolution kernel size, the more complex the convolution operation that can be fitted, and the more and more refined the correlation information that can be learned. Therefore, under different convolution kernel sizes, the convolution operation can learn characteristics covering different frequency resolutions. Therefore, in Figure 3 In , the feature extraction layer performs multi-scale feature extraction on the input speech signal under different convolution kernel sizes, such as Figure 3 As shown in , there are three convolution kernel sizes in the feature extraction layer, namely 8, 16 and 32, so Figure 4 As shown in the figure, convolution kernels of different sizes can cover the receptive field between adjacent sampling points of the input speech signal (corresponding to the "neighbors" in the figure), the receptive field with a pitch length (corresponding to the "inside the pitch" in the figure), and the receptive field with at least two pitch lengths (corresponding to the "between the pitches" in the figure). That is, for features that need to cover the scale of adjacent sampling points, the speech sending method selects a small convolution kernel size of 8, for features that need to cover the scale within the same pitch, the speech sending method selects a medium convolution kernel size of 16, and for features that need to cover the scale between different pitches, the speech sending method selects a large convolution kernel size of 32. However, this is only an example, and the speech sending method can select any convolution kernel size that can cover the receptive field between adjacent sampling points, the receptive field with a pitch length, and the receptive field with at least two pitch lengths.

[0103] In addition, each scale of the convolution operation in the multi-scale convolution operation in the feature extraction layer may include at least one convolution operation. Figure 3 In the multi-scale convolution operation in the feature extraction layer, each scale of the convolution operation includes two convolution operations, and the step size of each scale of the convolution operation in the multi-scale convolution operation in the feature extraction layer can be 1, so that the signal scale does not change. For example, if the shape of the input speech signal is [640,1], where 640 represents the length of the input speech signal and 1 represents the number of speech, then the shape of the output from the feature extraction layer is 3*[640,2], where 3 represents that the output of the feature extraction layer is data of three scales, that is, the shape of the signal output by each scale of the multi-scale convolution operation in the feature extraction layer is [640,2], where 640 represents the length of the output feature information and 2 represents the number of the output feature information. Since each scale of the convolution operation includes two convolution operations, the number of feature information output by each scale of the convolution operation is 2.

[0104] Thereafter, the step of extracting low-frequency feature information from the input speech signal using the first feature extraction network includes: obtaining low-frequency feature information based on the extracted speech feature information using the at least one second feature extraction module.

[0105] Specifically, the step of obtaining low-frequency feature information may include: extracting features from the input information through at least two convolution processing parameters and outputting the extracted feature information. That is, at least one second feature extraction module in the first feature network may extract features from the input speech feature information through at least two convolution processing parameters and output the extracted feature information. Among them, the at least two convolution processing parameters include at least two of the following: convolution processing parameters corresponding to the receptive field between adjacent sampling points of the speech signal, convolution processing parameters corresponding to the receptive field of one fundamental pitch length, and convolution processing parameters corresponding to the receptive field of at least two fundamental pitch lengths, wherein the convolution processing parameters may include the convolution kernel size corresponding to the convolution operation. Since the above has been referred to Figure 3 The convolution processing parameters have been described in detail, so they will not be repeated here.

[0106] like Figure 3 As shown in , the speech features are convolved by setting the step size of the convolution operation in the downsampling layer to 2 to obtain low-frequency feature information including the relevant information between the high-frequency features and the low-frequency features. In detail, the speech sending method can use the multi-scale convolution operation in the downsampling layer to convolve and convert the sampling rate of the speech features extracted by step S110. In other words, the speech sending method can convolve and convert the sampling rate by convolving the speech features through the multi-scale convolution operation in the downsampling layer, thereby integrating the relevant information between the high-frequency features and the low-frequency features into the low-frequency feature information to obtain the low-frequency feature information including the relevant information. The receiving end of the speech signal can better restore the high-frequency feature information based on the relevant information.

[0107] In addition, Figure 3 In the example shown in , the speech sending method uses the same convolution processing parameters as the convolution processing parameters in the at least one first feature extraction module in the at least one second feature extraction module, that is, the multi-scale convolution operations with the same number and size as the multi-scale convolution operations in the feature extraction layer are used in the downsampling layer, for example, Figure 3 As shown in , the multi-scale convolution operation in the downsampling layer has three convolution kernel sizes, namely 8, 16 and 32, and each convolution operation in the multi-scale convolution operation has two convolution operations, and each convolution operation is implemented by a convolution kernel. Therefore, Figure 3As shown in , in order to perform convolution and sampling rate conversion through the multi-scale convolution operation in the downsampling layer, Figure 3 The example shown in sets the step size of each convolution operation in the multi-scale convolution operation in the downsampling layer to 2. Therefore, if the shape of the signal output from the feature extraction layer is 3*[640,2], the shape of the signal output from the downsampling layer is 3*[320,2]. Here, the reason for setting the step size of the filter to 2 is that the frequency of SWB is twice the frequency of WB.

[0108] In addition, although the speech sending method uses the same convolution processing parameters as the convolution processing parameters in the at least one first feature extraction module in the at least one second feature extraction module, the present invention is not limited to this. The at least one second feature extraction module may also use convolution processing parameters different from the convolution processing parameters used by the at least one first feature extraction module. In this case, the speech feature information extracted by the at least one first feature extraction module may be fused and output to obtain a form of speech feature information suitable for being processed by the at least one second feature extraction module. For example, assuming Figure 3 The multi-scale convolution operation in the downsampling layer has three convolution kernel sizes, namely 8, 16 and 32, while the multi-scale convolution operation in the downsampling layer has two convolution kernel sizes, namely 16 and 32. In this case, the feature extraction layer can fuse the speech feature information output by each convolution operation and output it to the downsampling layer. Each convolution operation of the downsampling layer performs multi-scale convolution operation on the fused feature information output by the feature extraction layer.

[0109] Thereafter, in step S120, the speech signal corresponding to the low-frequency feature information is sent to the receiving end.

[0110] Specifically, the step of sending the speech signal corresponding to the low-frequency feature information to the receiving end may include: using a first feature fusion network to fuse all the low-frequency feature information output by the first feature extraction network to obtain a speech signal corresponding to the low-frequency feature information, and sending the speech signal corresponding to the low-frequency feature information to the receiving end. Figure 3 In the example shown, the first feature fusion network corresponds to Figure 3The downsampling layer outputs the operation results obtained by each convolution operation (i.e., the extracted low-frequency feature information), and the feature fusion layer fuses all the low-frequency feature information output by the downsampling layer. In detail, through step S110, the speech sending method obtains low-frequency feature information at different scales through the downsampling layer. Thereafter, the speech sending method performs convolution processing on the low-frequency feature information at different scales obtained through the downsampling layer by setting the step size of a single convolution operation in the feature fusion layer to 1 to obtain a speech signal corresponding to the low-frequency feature information, such as Figure 3 As shown in, the speech sending method obtains low-frequency feature information at three scales through the downsampling layer, and each scale may also contain multiple low-frequency feature information. Thereafter, the speech sending method performs convolution processing on the low-frequency feature information at the three scales through the feature fusion layer to obtain the speech signal corresponding to the low-frequency feature information that needs to be output in the end, and sends the speech signal to the receiving end. For example, if the shape of the signal output from the downsampling layer is 3*[320,2], the shape of the signal output from the feature fusion layer is [320,1].

[0111] The signal output by the feature fusion layer is a low-frequency time domain signal, which is encoded to form a code stream, and the transmitter sends the code stream to the receiver.

[0112] also, Figure 3 The number of convolution operations in the feature extraction layer, downsampling layer, and feature fusion layer in the AI-based downsampling process shown in and the number of convolution kernels included in each convolution operation are only exemplary, and the present invention is not limited thereto. In addition, since the design of the number of convolution operations determines the richness of the types of different frequency information that the voice sending method can learn, when designing the number of convolution operations, the model corresponding to the voice sending method should be able to run on the corresponding device (such as a smartphone), and the test results of the mobile voice quality test standard (such as POLQA) of the voice sending method must also be better than the test results of the traditional bandwidth extension (Bandwidth Extension, BWE) method. Therefore, in the present invention, the total number of convolution operations in the feature extraction layer, the downsampling layer, and the feature fusion layer can be determined based on the performance of the platform to which the voice sending method is applied and the test results based on the mobile voice quality test standard.

[0113] Compared with the traditional method, the voice transmission method corresponding to the AI-based downsampling process described above can completely retain the low-frequency features of the input voice signal, and at the same time include the relevant information between the high-frequency features and the low-frequency features into the low-frequency feature information, wherein the feature extraction layer can completely retain more information in the input voice signal, making the low-frequency information more complete, and the downsampling layer can integrate the relevant information between the high-frequency features and the low-frequency features into the low-frequency feature information so that the high-frequency features can be better reconstructed in the voice receiving device. In addition, since the traditional downsampling method only uses a simple filter of a single scale, the low-frequency information of the voice signal will be lost during the sampling rate conversion process, making it difficult to restore the complete low-frequency features in the voice receiving device, and it is also difficult to better restore the high-frequency features during the upsampling process. The two technologies adopted in the embodiments of the present application will improve the clarity of the voice, thereby enhancing the user's subjective experience.

[0114] In addition, in practical applications, the voice sender is actually also the voice receiver of another voice sender, so the voice sending method may also include: receiving a voice signal sent from the other voice sender; extracting low-frequency feature information from the received voice signal and restoring high-frequency feature information using a second feature extraction network; and outputting a voice signal containing the low-frequency feature information and the high-frequency feature information. This will be described in detail below when describing the voice receiving method.

[0115] Receiving the voice signal sent from the other voice sending end includes: receiving a code stream corresponding to the voice signal sent by the sending end through a communication network, and decoding the received code stream to obtain the voice signal.

[0116] In addition, although the above description takes CNN as an example to describe the speech transmission method based on the neural network, the neural network based on the speech transmission method of the present invention can also be a deep neural network (DNN). In other words, the first feature extraction network and the first feature fusion network can be implemented based on the DNN, such as Figure 5 As shown in the figure, the input high-frequency speech signal is first transformed from the time domain to the frequency domain, and then the frequency domain information is transformed from high frequency to low frequency through the DNN to obtain the low-frequency frequency domain information, and finally the low-frequency frequency domain information is transformed from the frequency domain to the time domain to obtain the output low-frequency time domain signal, and the low-frequency time domain signal is sent to the receiving end after being encoded.

[0117] Figure 6 is an overall flow chart illustrating a voice receiving method according to an exemplary embodiment of the present invention.

[0118] In step S210, a voice signal sent by a transmitting end is received.

[0119] In step S220, a second feature extraction network is used to extract low-frequency feature information from the received speech signal and restore high-frequency feature information.

[0120] In step S230, a speech signal including the low-frequency feature information and the high-frequency feature information is output.

[0121] The following will refer to Figures 7 to 10a to Figure 10g right Figure 6 The illustrated speech receiving method is described in detail.

[0122] After receiving the code stream corresponding to the voice signal through the communication network, the receiving end decodes the code stream to obtain the voice signal, and then uses the second feature network to extract low-frequency feature information and restore high-frequency feature information.

[0123] Figure 7 is a specific flow chart showing a voice receiving method according to an exemplary embodiment of the present invention. Figure 7 The speech receiving method shown in is a speech receiving method based on a neural network, wherein the neural network may be a convolutional neural network (CNN), that is, the speech receiving method may be as follows Figure 8 The CNN-based speech reception method shown in .

[0124] The second feature extraction network can be composed of one or more feature extraction modules, and multiple feature extraction modules can be cascaded. Each feature extraction module can include one or more convolution operations. Among them, each feature extraction module can perform multi-scale convolution operations, and different convolution processing parameters (such as convolution kernel size) correspond to different receptive fields and different frequency domain resolution capabilities. Therefore, it is possible to avoid the influence of many factors such as different speakers, ages, genders, and speech content, and can cover a large range of changes in speech signals. The feature information of the speech signal obtained by the solution of the embodiment of the present application can cover a variety of speech audio features, thereby improving the performance of downsampling.

[0125] In the feature extraction module, the scale of a convolution operation can correspond to one or more convolution operations. That is to say, in the feature extraction module, the convolution kernel sizes of each convolution operation can be different, or at least two convolution operations can have the same convolution kernel size, that is, corresponding to the same scale.

[0126] When the second feature extraction network contains multiple feature extraction modules, the step sizes of the multiple feature extraction modules can be different, and the step size of the feature extraction module corresponds to the sampling rate. For example, the step size of feature extraction module A and feature extraction module B can be equal, both are 1. At this time, the sampling rate of the output and the output has not changed. The step size of feature extraction module C is 2, and it performs a sampling rate conversion.

[0127] If the feature extraction module includes multiple convolution operations, the step sizes of the convolution operations may be the same.

[0128] like Figure 8 As shown in , the speech reception method includes three processes: multi-filter scale feature extraction (low frequency) process, multi-filter scale feature extraction (high frequency) process and feature fusion process, wherein the multi-filter scale feature extraction can also be called multi-convolution kernel scale feature extraction. The multi-convolution kernel scale feature extraction (low frequency) process is composed of a feature extraction layer and a downsampling layer and corresponds to Figure 7 In steps S310 and S320, the multi-convolution kernel scale feature extraction (high frequency) process is composed of the bottleneck layer of CNN and the upsampling layer of CNN and corresponds to Figure 7 In steps S330 and S340, the feature fusion process is composed of the feature fusion layer of CNN and corresponds to Figure 7 However, the present invention is not limited thereto, and the feature extraction network in the receiving method of the present invention can also be implemented using a DNN (deep neural network).

[0129] In order to facilitate understanding of the present invention, in the following description, Figure 8 The processing process of the CNN-based speech transmission method is shown in Figure 7 The voice sending method is described in detail.

[0130] In step S310, the received speech signal is replicated to expand the data scale of the received speech signal. Specifically, this step is equivalent to a preprocessing process, that is, the received data is scaled to obtain the data to be processed. This process corresponds to Figure 8 The speech receiving method completes the scale expansion by performing interval repetitive copying on the received data. For example, if the received data is [0, 1, 2, 3, 4], the expanded data is [0, 0, 1, 1, 2, 2, 3, 3, 4, 4]. This step can complete the formal symmetrical expansion from low-frequency frequency domain information to high-frequency frequency domain information.

[0131] In step S320, a low-frequency feature extraction network in the second feature extraction network is used to extract low-frequency feature information from the received speech signal, where the low-frequency feature information includes correlation information between high-frequency features and low-frequency features.

[0132] Specifically, the low-frequency feature extraction network includes at least one third feature extraction module and at least one fourth feature extraction module, such as Figure 8 As shown in , the second feature extraction network can correspond to Figure 8The multi-convolution kernel scale feature extraction (low frequency) process and the multi-convolution kernel scale feature extraction (high frequency) process in the low-frequency feature extraction network may correspond to Figure 8 In the multi-convolution kernel scale feature extraction (low frequency) process, the at least one third feature extraction module may correspond to Figure 8 The feature extraction layer in the embodiment of the present invention, the at least one fourth feature extraction module may correspond to Figure 8 The downsampling layer in the second feature extraction network. The low-frequency feature extraction network may include one or more feature extraction layers, and may also include one or more downsampling layers. Specifically, the step of extracting low-frequency feature information from the received speech signal using the low-frequency feature extraction network in the second feature extraction network may include: using the at least one third feature extraction module to extract the speech feature information of the received speech signal. Among them, the at least one third feature extraction module in the second feature extraction network performs feature extraction on the input information respectively through at least two convolution processing parameters, and outputs the extracted feature information. Among them, the at least two convolution processing parameters include at least two of the following: convolution processing parameters corresponding to the receptive field between adjacent sampling points of the speech signal, convolution processing parameters corresponding to the receptive field of one fundamental pitch length, and convolution processing parameters corresponding to the receptive field of at least two fundamental pitch lengths, wherein the convolution processing parameters may include the convolution kernel size corresponding to the convolution operation. Since the above has been referred to Figure 3 This has been described in detail, so the convolution processing parameters will not be repeated here.

[0133] Specifically, if Figure 8 As shown in , the step of extracting speech feature information may include: performing multi-scale feature extraction on the expanded speech signal by setting the step size of the multi-scale convolution operation in the feature extraction layer to 1 to obtain the speech feature information. In this step, the speech receiving method does not change the scale of the signal, so the step size of the multi-scale convolution operation in the feature extraction layer is set to 1. In addition, the feature extraction layer reuses the design of the feature extraction layer in the speech sending method. In other words, the number and size of the convolution operations in the feature extraction layer in the speech receiving method are consistent with the number and size of the convolution operations in the feature extraction layer in the speech sending method. In other words, the feature extraction layer in the speech receiving method reuses the design of the feature extraction layer in the speech sending method, which enables the AI-based upsampling process of the speech receiving device to better restore the information obtained by the AI-based downsampling process of the speech sending device. Therefore, compared with Figure 3 The schematic diagram of the AI-based downsampling process shown in corresponds to, Figure 8The multi-scale convolution operation in the feature extraction layer in the AI-based upsampling process shown in also has three convolution kernel sizes, namely 8, 16 and 32, and each scale of the multi-scale convolution operation has two convolution operations, so the speech receiving method can extract speech features by utilizing multi-scale convolution operations of different sizes. For example, if the shape of the input expanded speech signal is [640,1], then since the step size is set to 1, the shape output from the feature extraction layer is 3*[640,2].

[0134] Among them, the convolution operation of each scale of the feature extraction layer may also include one convolution operation, or include more than two convolution operations.

[0135] In addition, the step of extracting low-frequency feature information from the received speech signal using the low-frequency feature extraction network in the second feature extraction network may also include: using the at least one fourth feature extraction module to obtain low-frequency feature information based on the extracted speech feature information. Wherein, the at least one fourth feature extraction module in the second feature extraction network performs feature extraction on the input information respectively through at least two convolution processing parameters, and outputs the extracted feature information. Wherein, the at least two convolution processing parameters include at least two of the following: convolution processing parameters corresponding to the receptive field between adjacent sampling points of the speech signal, convolution processing parameters corresponding to the receptive field of one fundamental pitch length, and convolution processing parameters corresponding to the receptive field of at least two fundamental pitch lengths, wherein the convolution processing parameters may include the convolution kernel size corresponding to the convolution operation. Since the above has been referred to Figure 3 This has been described in detail, so the convolution processing parameters will not be repeated here.

[0136] Specifically, for example, Figure 8 As shown in , when obtaining the low-frequency feature information, the low-frequency feature information can be obtained by setting the step size of the multi-scale convolution operation in the downsampling layer to 2 and convolving the speech features.

[0137] In addition, the downsampling layer reuses the design of the downsampling layer in the speech sending method. In other words, the number and size of the convolution operations in the downsampling layer in the speech receiving method are consistent with the number and size of the convolution operations in the downsampling layer in the speech sending method. That is to say, the downsampling layer in the speech receiving method reuses the design of the downsampling layer in the speech sending method, which enables the AI-based upsampling process of the speech receiving device to better restore the information obtained by the AI-based downsampling process of the speech sending device. Accordingly, the speech receiving method uses the same number and size of multi-scale convolution operations in the downsampling layer as the multi-scale convolution operations in the feature extraction layer, for example, Figure 8As shown in , the multi-scale convolution operation in the downsampling layer has three convolution kernel sizes, namely 8, 16 and 32, and each convolution operation in the multi-scale convolution operation has two convolution operations, and each convolution operation has one convolution kernel. Therefore, Figure 8 As shown in , in order to perform sampling rate conversion by convolution through the multi-scale convolution operation in the downsampling layer, Figure 8 The example shown in sets the step size of each convolution operation in the multi-scale convolution operation in the downsampling to 2. Therefore, if the shape of the signal output from the feature extraction layer is 3*[640,2], the shape of the signal output from the downsampling layer is 3*[320,2]. Here, the reason for setting the step size of the convolution operation to 2 is that the frequency of SWB is twice the frequency of WB.

[0138] Reference above Figure 8 The feature extraction layer and downsampling layer in the AI-based upsampling process of the described speech receiving method and the feature extraction layer and downsampling layer in the AI-based downsampling process of the speech sending method use the same size and number of convolution operations on the same signal scale, so that the AI-based upsampling process of the speech receiving device and the AI-based downsampling process of the speech sending device can extract low-frequency information and recover signals from the information in a consistent manner. Such operations enable the AI-based upsampling process of the speech receiving device to better recover the information processed by the AI-based downsampling process of the speech sending device.

[0139] In step S330, a high-frequency feature extraction network in the second feature extraction network is used to restore high-frequency feature information according to the low-frequency feature information and the high-frequency feature information and the low-frequency feature information are fused to obtain feature information including high-frequency feature information and low-frequency feature information.

[0140] Specifically, the high-frequency feature extraction network includes at least one fifth feature extraction module and at least one sixth feature extraction module, such as Figure 8 As shown in , the high-frequency feature extraction network may correspond to Figure 8 In the multi-convolution kernel scale feature extraction (high frequency) process, the at least one fifth feature extraction module may correspond to Figure 8 The bottleneck layer in Figure 8 The convolution layer in the upsampling layer in (not shown in the figure), the at least one sixth feature extraction module may correspond to Figure 8Specifically, the step of using the high-frequency feature extraction network in the second feature extraction network to restore the high-frequency feature information according to the low-frequency feature information and fusing the high-frequency feature information with the low-frequency feature information may include: using the at least one fifth feature extraction module to restore the high-frequency feature information according to the low-frequency feature information.

[0141] Among them, at least one fifth feature extraction module in the second feature extraction network extracts features from the input information respectively through at least two convolution processing parameters, and outputs the extracted feature information. Among them, the at least two convolution processing parameters include at least two of the following: convolution processing parameters corresponding to the receptive field between adjacent sampling points of the speech signal, convolution processing parameters corresponding to the receptive field of one pitch length, and convolution processing parameters corresponding to the receptive field of at least two pitch lengths, wherein the convolution processing parameters may include the convolution kernel size corresponding to the convolution operation. As the above has been referred to Figure 3 This has been described in detail, so the convolution processing parameters will not be repeated here.

[0142] Specifically, if Figure 8 As shown in , the step of restoring the high-frequency feature information may include: convolving the low-frequency feature information by setting the step size of the multi-scale convolution operation in the bottleneck layer to 2 to obtain the convolved feature information. The low-frequency feature information obtained by the downsampling layer is actually low-frequency feature information that incorporates the relevant information between the high-frequency features and the low-frequency features, and the bottleneck layer can further fuse the low-frequency feature information containing the relevant information. The bottleneck layer can be implemented by bottleneck convolution. For example, the multi-scale low-frequency feature information output from the downsampling layer can be processed by setting the step size of the multi-scale convolution operation in the bottleneck layer to 2. The multi-scale convolution operation with the same number and size as the multi-scale convolution operation in the downsampling layer can be used in the bottleneck layer, such as Figure 8 As shown in , each scale of the convolution operation in the multi-scale convolution operation in the bottleneck layer includes two convolution operations, and as Figure 8 As shown in , the convolution kernel size can be 8, 16, and 32. Therefore, Figure 8 As shown in , if the shape of the signal output from the downsampling layer is 3*[320,2], the shape of the signal output from the bottleneck layer is 3*[160,2].

[0143] The convolution operation of each scale of the feature extraction layer may also include one convolution operation, or include more than two convolution operations.

[0144] In addition, if Figure 8As shown in , the step of restoring the high-frequency feature information may also include: convolving the low-frequency feature information by setting the step size of the multi-scale convolution operation in the upsampling layer to 1 and arranging the convolution result to obtain the high-frequency feature information. Wherein, the speech receiving method uses the same number and size of convolution operations as the multi-scale convolution operations in the bottleneck layer in the upsampling layer, such as Figure 8 As shown in , each of the multi-scale convolution operations of the convolution layer in the upsampling layer includes four convolution operations, and each convolution operation has a convolution kernel, and as shown in Figure 8 As shown in , the convolution kernel size can be 8, 16 and 32. In this step, the speech receiving method first sets the step size of the multi-scale convolution operation of the convolution layer in the upsampling layer to 1 to perform convolution processing on the low-frequency feature information output by the bottleneck layer. For example, if the shape of the signal output from the bottleneck layer is 3*[160,2], then the result of the convolution is 3*[160,4]. This convolution processing integrates the information obtained by the bottleneck layer. Thereafter, the upsampling layer rearranges the result of the convolution to further obtain the rearranged high-frequency feature information. For example, if the result of the convolution is 3*[160,4], then the rearranged high-frequency feature information is 3*[320,2]. The rearrangement method adopted is a cross-arrangement method, for example, as Figure 8 As shown in, under each convolution operation, the result of the convolution is 4 sets of data (such as Figure 8 In the embodiment of the present invention, the data in the first group and the second group are arranged crosswise, and the data in the third group and the fourth group are arranged crosswise, but the method of rearrangement is only exemplary and the present invention is not limited thereto.

[0145] Through the above operations, high-frequency feature information can be restored.

[0146] Since in the embodiment of the present application, the low-frequency feature information of the voice signal sent by the sending end includes more relevant information between the high-frequency features and the low-frequency features, the receiving end can learn the relationship between the high-frequency features and the low-frequency features by using the relevant information, and can better restore the high-frequency features based on the low-frequency features.

[0147] In addition, the step of using the high-frequency feature extraction network in the second feature extraction network to restore the high-frequency feature information based on the low-frequency feature information and fusing the high-frequency feature information and the low-frequency feature information may also include: using at least one sixth feature extraction module to fuse the high-frequency feature information and the low-frequency feature information extracted by the corresponding fourth feature extraction module to obtain feature information containing the high-frequency feature information and the low-frequency feature information.

[0148] Among them, the at least one sixth feature extraction module fuses the high-frequency feature information corresponding to the at least two input convolution processing parameters and the low-frequency feature information extracted by the corresponding fourth feature extraction module according to the corresponding convolution processing parameters, and outputs feature information containing the high-frequency feature information and the low-frequency feature information. Among them, the at least two convolution processing parameters include at least two of the following: a convolution processing parameter corresponding to the receptive field between adjacent sampling points of the speech signal, a convolution processing parameter corresponding to the receptive field of one fundamental pitch length, and a convolution processing parameter corresponding to the receptive field of at least two fundamental pitch lengths, wherein the convolution processing parameter may include a convolution kernel size corresponding to the convolution operation. As the above has been referred to Figure 3 This has been described in detail, so the convolution processing parameters will not be repeated here.

[0149] Specifically, if Figure 8 As shown in , the high-frequency feature information and the low-frequency feature information can be spliced ​​to obtain the data containing high- and low-frequency information. Figure 8 , the high-frequency feature information output by the convolution layer in the upsampling layer and the low-frequency feature information output by the convolution processing in the downsampling layer can be spliced ​​together to obtain feature information containing high-frequency feature information and low-frequency feature information at different scales. For example, if the shape of the signal output from the convolution layer in the upsampling layer is 3*[320,2], and the shape of the signal output by the convolution processing in the downsampling layer is 3*[320,2], the result of splicing is 3*[640,2]. This step is to simply splice the high-frequency feature information in front of the low-frequency feature information to obtain feature information containing high-frequency feature information and low-frequency feature information.

[0150] In step S340, the feature information output by the second feature extraction network including high-frequency feature information and low-frequency feature information is fused using the second feature fusion network to obtain a speech signal corresponding to the feature information including high-frequency feature information and low-frequency feature information. Figure 8As shown in , the feature information including high-frequency feature information and low-frequency feature information can be fused by setting the step size of a single convolution operation in the feature fusion layer of CNN to 1 to obtain the output signal. In addition, the feature fusion layer reuses the design of the feature fusion layer in the speech transmission method. In other words, the number and size of the convolution operations of the feature fusion layer are consistent with the number and size of the convolution operations of the feature fusion layer in the speech transmission method. In other words, the feature fusion layer in the speech reception method reuses the design of the feature fusion layer in the speech transmission method. Figure 8 The speech receiving method obtains a speech signal corresponding to the feature information including high-frequency feature information and low-frequency feature information by convolution processing the feature information including high-frequency feature information and low-frequency feature information at different scales using the feature fusion layer. For example, if the shape of the signal input to the feature fusion layer is 3*[640,2], the shape of the signal output from the feature fusion layer is [640,1].

[0151] In step S350, a speech signal corresponding to the feature information including the high-frequency feature information and the low-frequency feature information is output.

[0152] Compared with the traditional downsampling method, the downsampling method of the present invention can better preserve the downsampled information and improve the language intelligibility. In addition, the method of jointly designing upsampling and downsampling of the present invention can more effectively improve the SNR value and POLQA test results.

[0153] Fig. 9 is a detailed flow chart showing a voice receiving method according to another exemplary embodiment of the present invention. Fig. 9 The speech receiving method shown in is a speech receiving method based on a neural network, wherein the neural network may be a convolutional neural network (CNN), that is, the speech receiving method may be as follows Fig.10a The CNN-based speech reception method shown in .

[0154] like Fig.10a As shown in , the speech receiving method may include two processes: a multi-filter scale feature extraction (high frequency) process and a feature fusion process, wherein the multi-filter scale feature extraction can be called multi-convolution kernel scale feature extraction. The multi-convolution kernel scale feature extraction (high frequency) process is composed of a CNN feature extraction layer and an upsampling layer and corresponds to Fig. 9 In steps S410 and S420, the feature fusion process is composed of the feature fusion layer of CNN and corresponds to Fig. 9 Steps S430 and S440 in .

[0155] However, the present invention is not limited to this, and the feature extraction network in the receiving method of the present invention can also be implemented using DNN (deep neural network).

[0156] In order to facilitate understanding of the present invention, in the following description, Fig.10a The processing process of the CNN-based speech transmission method is shown in Fig. 9 The voice sending method is described in detail.

[0157] In step S410, at least one seventh feature extraction module is used to extract speech feature information of the received speech signal. The at least one seventh feature extraction module is included in the second feature extraction network, and the second feature extraction network also includes at least one eighth feature extraction module. Fig.10a As shown in , the second feature extraction network can correspond to Fig.10a In the multi-convolution kernel scale feature extraction (high frequency) process, the at least one seventh feature extraction module may correspond to Fig.10a The feature extraction layer in the embodiment of the present invention, the at least one eighth feature extraction module may correspond to Fig.10a The upsampling layer in .

[0158] Specifically, the at least one seventh feature extraction module extracts features from the input information respectively through at least two convolution processing parameters, and outputs the extracted speech feature information. The at least two convolution processing parameters include at least two of the following: a convolution processing parameter corresponding to the receptive field between adjacent sampling points of the speech signal, a convolution processing parameter corresponding to the receptive field of one pitch length, and a convolution processing parameter corresponding to the receptive field of at least two pitch lengths, wherein the convolution processing parameter may include a convolution kernel size corresponding to the convolution operation. Figure 3 This has been described in detail, so the convolution processing parameters will not be repeated here.

[0159] Specifically, if Fig.10a As shown in , the step of extracting speech feature information may include: performing multi-scale feature extraction on the received speech signal by setting the step size of the multi-scale convolution operation in the feature extraction layer to 1 to obtain the speech feature information. In this step, the speech receiving method does not change the scale of the signal, so the step size of the multi-scale convolution operation in the feature extraction layer is set to 1. In addition, the feature extraction layer reuses the design of the feature extraction layer in the speech sending method. In other words, the number and size of the convolution operations in the feature extraction layer in the speech receiving method are consistent with the number and size of the convolution operations in the feature extraction layer in the speech sending method. In other words, the feature fusion layer in the speech receiving method reuses the design of the feature fusion layer in the speech sending method, which enables the speech receiving device to better restore the information processed by the speech sending device. Therefore, compared with Figure 3 The schematic diagram of the AI-based downsampling process shown in corresponds to, Fig.10aThe multi-scale convolution operation in the feature extraction layer in the AI-based upsampling process shown in also has three convolution kernel sizes, namely 8, 16 and 32, and each of the multi-scale convolution operations has two convolution operations, and each convolution operation has one convolution kernel, so the speech receiving method can extract speech features by utilizing multi-scale convolution operations of different sizes. For example, if the shape of the input data to be processed is [320,1], then since the step size is set to 1, the shape output from the feature extraction layer is 3*[320,2].

[0160] In step S420, at least one eighth feature extraction module is used to extract low-frequency feature information containing relevant information between high-frequency features and low-frequency features based on the extracted speech feature information, and the high-frequency feature information is restored to obtain feature information containing high-frequency feature information and low-frequency feature information. The at least one eighth feature extraction module extracts low-frequency feature information from the input information respectively through at least two deconvolution processing parameters, and restores high-frequency feature information to obtain feature information containing high-frequency feature information and low-frequency feature information. Among them, the at least two convolution processing parameters include at least two of the following: a convolution processing parameter corresponding to the receptive field between adjacent sampling points of the speech signal, a convolution processing parameter corresponding to the receptive field of one fundamental pitch length, and a convolution processing parameter corresponding to the receptive field of at least two fundamental pitch lengths, wherein the convolution processing parameter may include a convolution kernel size corresponding to the convolution operation. Since the above has been referred to Figure 3 This has been described in detail, so the convolution processing parameters will not be repeated here.

[0161] Specifically, if Fig.10a As shown in , when extracting feature information including high-frequency feature information and low-frequency feature information, the speech feature can be deconvolved by setting the step size of the multi-scale convolution operation in the upsampling layer to 2 to extract low-frequency feature information from the input information, and restore the high-frequency feature information, thereby obtaining feature information including high-frequency feature information and low-frequency feature information. In this step, the speech receiving method needs to complete the scale expansion and upsampling rate recovery of the feature, so the speech receiving method implements the scale expansion and upsampling rate recovery of the feature by deconvolving the data input to the upsampling layer to obtain feature information including high-frequency feature information and low-frequency feature information, wherein the receiving method uses the same number and size of multi-scale convolution operations as the multi-scale convolution operations in the feature extraction layer in the upsampling layer, as Fig.10a As shown in , if the shape of the signal output from the feature extraction layer is 3*[320,2], the shape of the signal output from the upsampling layer is 3*[640,2].

[0162] In step S430, the feature information output by the second feature extraction network including high-frequency feature information and low-frequency feature information is fused using the second feature fusion network to obtain a speech signal corresponding to the feature information including high-frequency feature information and low-frequency feature information. Fig.10a As shown in , the feature information including high-frequency feature information and low-frequency feature information can be fused by setting the step size of a single convolution operation in the feature fusion layer of CNN to 1 to obtain the output signal. In addition, the feature fusion layer reuses the design of the feature fusion layer in the speech transmission method. In other words, the number and size of the convolution operations of the feature fusion layer are consistent with the number and size of the convolution operations of the feature fusion layer in the speech transmission method. In other words, the feature fusion layer in the speech reception method reuses the design of the feature fusion layer in the speech transmission method. Fig.10a The speech receiving method obtains a speech signal corresponding to the feature information including high-frequency feature information and low-frequency feature information by convolution processing the feature information including high-frequency feature information and low-frequency feature information at different scales using the feature fusion layer. For example, if the shape of the signal input to the feature fusion layer is 3*[640,2], the shape of the signal output from the feature fusion layer is [640,1].

[0163] In step S440, a speech signal corresponding to the feature information including the high-frequency feature information and the low-frequency feature information is output. Fig.10b is a detailed flow chart showing a voice receiving method according to another exemplary embodiment of the present invention. Fig.10b The speech receiving method shown in is a speech receiving method based on a neural network, wherein the neural network may be a convolutional neural network (CNN), that is, the speech receiving method may be as follows Fig.10c The CNN-based speech reception method shown in .

[0164] like Fig.10c As shown in , the speech receiving method may include three processes: a low-frequency repair process, a high-frequency prediction process and a high-low frequency fusion process, wherein the low-frequency repair process includes a multi-filter scale feature extraction (low-frequency repair) process and a low-frequency feature fusion process, the high-frequency prediction process includes an upsampling process and a multi-filter scale feature extraction (high frequency) process, and the multi-filter scale feature extraction (low-frequency repair) process and the multi-filter scale feature extraction (high frequency) process are both composed of a CNN feature extraction layer, wherein the low-frequency feature fusion process and the high-frequency feature fusion process are both composed of a CNN feature fusion layer.

[0165] Reference Fig.10b and Fig.10c , the multi-filter scale feature extraction (low-frequency restoration) process corresponds to Fig.10bIn step S420, the low-frequency feature fusion process corresponds to Fig.10b In step S430, the upsampling process corresponds to Fig.10b Step S440 in the multi-filter scale feature extraction (high frequency) process corresponds to Fig.10b In step S450, the high-frequency feature fusion process corresponds to Fig.10b In step S460, the high and low frequency fusion process corresponds to Fig.10b However, the present invention is not limited thereto, and the feature extraction network in the receiving method of the present invention can also be implemented using a DNN (deep neural network).

[0166] In order to facilitate understanding of the present invention, in the following description, Fig.10c The processing process of the CNN-based speech reception method is shown in Fig.10b The voice receiving method is described in detail.

[0167] In step S410 and Figure 6 The operation of step S210 in is the same as that of step S211, that is, receiving the voice signal sent by the sending end.

[0168] In step S420, the low-frequency feature extraction network in the second feature extraction network is used to extract low-frequency feature information from the received speech signal. Specifically, Fig.10c As shown in , the second feature extraction network can correspond to Fig.10c The multi-filter scale feature extraction (low-frequency restoration) process and the multi-filter scale feature extraction (high-frequency) process in the low-frequency feature extraction network may correspond to Fig.10c In the multi-filter scale feature extraction (low-frequency restoration) process, the at least one ninth feature extraction module may correspond to Fig.10c The feature extraction layer in the multi-filter scale feature extraction (low-frequency restoration) process in . In addition, the low-frequency feature extraction network can contain one or more feature extraction layers.

[0169] Specifically, the low-frequency feature extraction network includes at least one ninth feature extraction module, and the step of extracting low-frequency feature information from the received speech signal using the low-frequency feature extraction network in the second feature extraction network includes: using the at least one ninth feature extraction module to extract the low-frequency feature information of the received speech signal. Wherein, the at least one ninth feature extraction module performs feature extraction on the input information respectively through at least two convolution processing parameters, and outputs the extracted low-frequency feature information. Wherein, the at least two convolution processing parameters are at least two of the following: a convolution processing parameter corresponding to the receptive field between adjacent sampling points of the speech signal, a convolution processing parameter corresponding to the receptive field of one fundamental pitch length, and a convolution processing parameter corresponding to the receptive field of at least two fundamental pitch lengths, wherein the convolution processing parameter may include a convolution kernel size corresponding to the convolution operation. Since the above has been referred to Figure 3 This has been described in detail, so the convolution processing parameters will not be repeated here.

[0170] Specifically, if Fig.10c As shown in , the step of extracting low-frequency feature information may include: performing multi-scale feature extraction on the received speech signal to obtain low-frequency feature information by setting the step size of the multi-scale convolution operation in the feature extraction layer in the multi-filter scale feature extraction (low-frequency repair) process to 1. In this step, the speech receiving method does not change the scale of the signal, so the step size of the multi-scale convolution operation in the feature extraction layer in the multi-filter scale feature extraction (low-frequency repair) process is set to 1.

[0171] In addition, the feature extraction layer reuses the design of the feature extraction layer in the speech transmission method. In other words, the number and size of convolution operations in the feature extraction layer in the multi-filter scale feature extraction (low-frequency repair) process of the speech receiving method are consistent with the number and size of convolution operations in the feature extraction layer in the speech transmission method. That is to say, the feature extraction layer in the multi-filter scale feature extraction (low-frequency repair) process of the speech receiving method reuses the design of the feature extraction layer in the speech transmission method, which enables the AI-based upsampling process of the speech receiving device to better restore the information obtained by the AI-based downsampling process of the speech transmitting device. Therefore, compared with Figure 3 The schematic diagram of the AI-based downsampling process shown in corresponds to Fig.10cIn the AI-based upsampling process shown in , the multi-scale convolution operation in the feature extraction layer in the multi-filter scale feature extraction (low-frequency repair) process also has three convolution kernel sizes, namely 8, 16, and 32, and each scale of the multi-scale convolution operation has two convolution operations, so the speech reception method can extract speech features by utilizing multi-scale convolution operations of different sizes. For example, if the shape of the received speech signal is [320,1], since the step size is set to 1, the shape of the feature extraction layer output from the multi-filter scale feature extraction (low-frequency repair) process is 3*[320,2].

[0172] Among them, the convolution operation of each scale of the feature extraction layer in the multi-filter scale feature extraction (low-frequency restoration) process may also include one convolution operation, or include more than two convolution operations.

[0173] In step S430, the low-frequency feature information output by the low-frequency feature extraction network is fused using the low-frequency feature fusion network in the second feature fusion network to obtain a low-frequency time domain signal of low-frequency repair.

[0174] Specifically, if Fig.10c As shown in , the second feature fusion network can correspond to Fig.10c The low-frequency feature fusion process and the high-frequency feature fusion process in the low-frequency feature fusion network correspond to Fig.10c The low-frequency feature fusion process in . Fig.10c As shown in , the low-frequency feature information output from the feature extraction layer in the multi-filter scale feature extraction (low-frequency repair) process can be fused by setting the step size of a single convolution operation in the feature fusion layer in the low-frequency feature fusion process to 1 to perform low-frequency repair, thereby obtaining a low-frequency time domain signal with the low-frequency part repaired.

[0175] In addition, the feature fusion layer in the low-frequency feature fusion process reuses the design of the feature fusion layer in the speech transmission method. In other words, the number and size of convolution operations of the feature fusion layer in the low-frequency feature fusion process are consistent with the number and size of convolution operations of the feature fusion layer in the speech transmission method. That is, in the speech receiving method, the feature fusion layer in the low-frequency feature fusion process reuses the design of the feature fusion layer in the speech transmission method. Fig.10c The speech receiving method obtains a low-frequency repaired low-frequency time domain signal by convolving low-frequency feature information at different scales using a feature fusion layer in a low-frequency feature fusion process. For example, if the shape of the signal input to the feature fusion layer in the low-frequency feature fusion process is 3*[320,2], the shape of the signal output from the feature fusion layer in the low-frequency feature fusion process is [320,1].

[0176] Steps S420 and S430 described above are used to recover a low-frequency time domain signal from a received speech signal, which repairs the low-frequency spectrum lost when the speech signal is encoded at the encoding end. In other words, steps S420 and S430 implement a low-frequency repair process.

[0177] In step S440, up-sampling is performed on the low-frequency repaired low-frequency time domain signal to obtain a low-frequency time domain signal with an increased sampling rate.

[0178] Specifically, refer to Fig.10c The upsampling process can use the upsampling method based on digital signal processing in the prior art to upsample the low-frequency time domain signal output from the low-frequency feature fusion process and the low-frequency repaired low-frequency signal, so that the sampling rate of the low-frequency time domain signal is improved, but the upsampled time domain signal does not add any high-frequency frequency information, such as Fig.10c As shown in , if the shape of the signal output from the feature fusion layer in the low-frequency feature fusion process is [320, 1], the shape of the signal output from the upsampling process is [640, 1].

[0179] In step S450, a high-frequency feature extraction network in the second feature extraction network is used to extract low-frequency feature information containing relevant information between high-frequency features and low-frequency features from the low-frequency time domain signal with the increased sampling rate, and high-frequency feature information is restored from the low-frequency feature information based on the relevant information.

[0180] Specifically, the high-frequency feature extraction network includes at least one tenth feature extraction module, such as Fig.10c As shown in , the high-frequency feature extraction network may correspond to Fig.10c In the multi-filter scale feature extraction (high frequency) process, the at least one tenth feature extraction module may correspond to Fig.10c Feature extraction layer in the multi-filter scale feature extraction (high frequency) process.

[0181] Specifically, the step of using the high-frequency feature extraction network in the second feature extraction network to extract low-frequency feature information containing relevant information between high-frequency features and low-frequency features from the low-frequency time domain signal with increased sampling rate includes: using the at least one tenth feature extraction module to extract the low-frequency feature information containing relevant information between high-frequency features and low-frequency features of the received speech signal, and restore the high-frequency feature information. Wherein, the at least one tenth feature extraction module extracts the low-frequency feature information of the input information respectively through at least two convolution processing parameters, and restores the high-frequency feature information. Wherein, the at least two convolution processing parameters are at least two of the following: a convolution processing parameter corresponding to the receptive field between adjacent sampling points of the speech signal, a convolution processing parameter corresponding to the receptive field of one fundamental pitch length, and a convolution processing parameter corresponding to the receptive field of at least two fundamental pitch lengths, wherein the convolution processing parameter may include a convolution kernel size corresponding to the convolution operation. Since the above has been referred to Figure 3 This has been described in detail, so the convolution processing parameters will not be repeated here.

[0182] Specifically, if Fig.10c As shown in , the steps of extracting low-frequency feature information and restoring high-frequency feature information may include: performing convolution processing on the low-frequency time domain signal with increased sampling rate by setting the step size of the multi-scale convolution operation in the feature extraction layer in the multi-filter scale feature extraction (high frequency) process to 1. In this process, low-frequency feature information containing relevant information between high-frequency features and low-frequency features is first obtained from the low-frequency time domain signal with increased sampling rate, and then high-frequency feature information is predicted from the low-frequency feature information based on the relevant information.

[0183] In addition, the feature extraction layer in the multi-filter scale feature extraction (high frequency) process reuses the design of the feature extraction layer in the speech transmission method. In other words, the number and size of convolution operations in the feature extraction layer in the multi-filter scale feature extraction (high frequency) process are consistent with the number and size of convolution operations in the feature extraction layer in the speech transmission method. Fig.10c As shown in , if the shape of the low-frequency time domain signal with increased sampling rate is [640, 1], then since the step size is set to 1, the shape of the feature extraction layer output from the multi-filter scale feature extraction (high frequency) process is 3*[640, 2].

[0184] Among them, the convolution operation of each scale of the feature extraction layer in the multi-filter scale feature extraction (high frequency) process may also include one convolution operation, or include more than two convolution operations.

[0185] In step S460, the high-frequency feature information output by the high-frequency feature extraction network is fused using the high-frequency feature fusion network in the second feature fusion network to obtain a high-frequency time domain signal.

[0186] Specifically, if Fig.10c As shown in , the high-frequency feature fusion network corresponds to Fig.10c The high-frequency feature fusion process in . Fig.10c As shown in , the high-frequency feature information output from the feature extraction layer in the multi-filter scale feature extraction (high frequency) process can be fused by setting the step size of a single convolution operation in the feature fusion layer in the high-frequency feature fusion process to 1 to obtain a high-frequency time domain signal.

[0187] In addition, the feature fusion layer in the high-frequency feature fusion process reuses the design of the feature fusion layer in the speech transmission method. In other words, the number and size of convolution operations of the feature fusion layer in the high-frequency feature fusion process are consistent with the number and size of convolution operations of the feature fusion layer in the speech transmission method. That is to say, in the speech receiving method, the feature fusion layer in the high-frequency feature fusion process reuses the design of the feature fusion layer in the speech transmission method. Fig.10c The speech receiving method obtains a high-frequency time domain signal by convolving high-frequency feature information at different scales using a feature fusion layer in a high-frequency feature fusion process. For example, if the shape of the signal input to the feature fusion layer in the high-frequency feature fusion process is 3*[640,2], the shape of the signal output from the feature fusion layer in the high-frequency feature fusion process is [640,1].

[0188] Since in step S460, the speech receiving method centrally predicts high-frequency information in the high-frequency prediction process instead of predicting high and low frequency information of the entire frequency band, the high-frequency spectrum of the speech signal can be well restored, making the high-frequency spectrum cleaner and more accurate.

[0189] The above-described steps S440 to S460 are used to restore the high-frequency time domain signal from the low-frequency time domain signal restored by steps S420 and S430. In other words, steps S440 to S460 implement a high-frequency prediction process.

[0190] In step S470, the low-frequency time domain signal with increased sampling rate is processed with the high-frequency time domain signal in the time domain to obtain a final speech signal, and the speech signal is output. In other words, step S470 implements the high-low frequency fusion process.

[0191] like Fig.10c As shown in , the speech signal to be output can be finally obtained by performing time domain summation processing on the low-frequency time domain signal with increased sampling rate output by the upsampling process and the high-frequency time domain signal output by the high-frequency feature fusion process, for example, as Fig.10cAs shown in , if the shape of the low-frequency time domain signal with increased sampling rate output from the upsampling process is [640, 1], and the shape of the high-frequency time domain signal output from the high-frequency feature fusion process is [640, 1], then the shape of the speech signal to be output is still [640, 1].

[0192] like Fig.10d As shown, Fig.10c The figure shows a comparison between the AI-based upsampling method (i.e., speech reception method) and the traditional upsampling method. The left figure shows the traditional upsampling method, and the right figure shows the AI-based upsampling method. The schematic diagram is a spectrogram, with the horizontal axis being time and the vertical axis being frequency. The stripes are the spectral distribution of speech. Fig.10d As can be seen from the area in the lower box, Fig.10c The AI-based upsampling method shown can repair the low-frequency spectrum lost by the encoder when encoding the speech signal. Fig.10d From the area in the upper box, we can see that when predicting high-frequency information, Fig.10c The AI-based upsampling method shown can focus on predicting the high-frequency area during the high-frequency prediction process, rather than predicting the high- and low-frequency information of the entire frequency band. Therefore, it can well restore the high-frequency spectrum of the speech signal, making the high-frequency spectrum cleaner and more accurate, thereby improving language intelligibility.

[0193] Although the above description takes CNN as an example to describe the speech receiving method based on the neural network, the neural network based on the speech receiving method of the present invention can also be a deep neural network (DNN). In other words, the second feature extraction network and the second feature fusion network can be implemented based on the DNN, such as Fig.10e As shown, the input low-frequency speech signal is first transformed from the time domain to the frequency domain, and then the high-frequency information is recovered from the low-frequency information through a deep neural network to obtain the frequency domain information containing high and low frequency information, and finally the frequency domain information is transformed from the frequency domain to the time domain to obtain the output high-frequency time domain signal.

[0194] In addition, the embodiment of the present application proposes that the above-mentioned voice signal sending method can be called an AI-based downsampling method, and the above-mentioned voice signal receiving method can be called an AI-based upsampling method. When designing the AI-based downsampling method and the AI-based upsampling method, a joint design can be performed, such as Fig.10fAs shown, when training the above-mentioned AI-based downsampling method and AI-based upsampling method, a joint training method can be adopted. For example, the first feature extraction network, the first feature fusion network, the second feature extraction network, and the second feature fusion network are jointly trained. Then, the AI-based upsampling process at the receiving end and the AI-based downsampling process at the sending end can use the same filter to learn the corresponding information of high-frequency features and low-frequency features, which is beneficial to improving performance, can better retain low-frequency feature information, better restore high-frequency feature information, the high-frequency feature information is of higher quality, and the user's auditory effect is better.

[0195] like Figure 10g As shown in the figure, it is a comparison between the AI-based downsampling method proposed in the embodiment of the present application and the traditional downsampling method. The left figure is the traditional method, and the right figure is the AI-based downsampling method. The schematic diagram is a spectrogram, with the horizontal axis being time, the vertical axis being frequency, and the stripes being the spectral distribution of the speech. From the area in the box in the figure, it can be seen that the AI-based downsampling method proposed in the embodiment of the present application can better preserve the downsampling information and improve the intelligibility of the language.

[0196] Taking the signal-to-noise ratio (SNR) in the signal domain as the evaluation index, the joint design method in the embodiment of the present application has at least 0.9dB performance improvement over the existing method. The time domain similarity between the recovered SWB signal and the input SWB signal is tested based on the SNR, and the higher the SNR value, the better. After MOS test comparison, the joint design method in the embodiment of the application is compared with the traditional BWE method, and the results show that the joint design method effectively improves the subjective test score.

[0197] In an example of this application, Fig.10hAs shown in the figure, the network architecture of joint Scale Down and Scale Up suitable for voice calls can be called X-net. The X-net model structure is similar to the autoencoder and U-net without jump connection. The voice encoding and decoding process can be regarded as the bottleneck module in U-net. The Scale Down module (also called Scale Down network) is responsible for converting the signal from high bandwidth to low bandwidth, similar to signal downsampling. The Scale Up module (also called Scale Up network) is responsible for converting the signal from low bandwidth to high bandwidth, similar to bandwidth expansion and super resolution. These two modules are used jointly at the transmitter and receiver in voice calls. In voice calls, algorithm latency is a key indicator. Scale Down and Scale Up are designed to use end-to-end time domain processing of causal CNN to avoid frame stacking and accumulation operations caused by frequency domain processing and future sampling point information required by CNN. The Scale Down network is composed of cascaded linear one-dimensional CNNs, without activation functions and offset weights. The Scale Up network uses cascaded one-dimensional CNNs and adopts Swish activation function. The sampling rate is increased by sampling point replication before the Scale Up network. Typical network parameters are as follows, where f, k, and s represent the number of convolution kernels, the size of the convolution kernel, and the step size, respectively. The corresponding parameter settings in Scale Down and Scale Up are the same. The following Tables 1 and 2 provide two typical parameter settings:

[0198] Table 1 - Typical parameters 1:

[0199] Scale Down Scale Up Layer 1 f=16,k=16,s=1 f=16,k=16,s=1 Layer 2 f=2,k=16,s=2 f=2,k=16,s=1 Layer 3 f=1,k=16,s=1 f=1,k=16,s=1

[0200] Table 2--Typical parameters 2:

[0201]

[0202]

[0203] Layer1 / Layer2 / Layer3 in the table represent network layers in the Scale Down and Scale Up networks, respectively.

[0204] This application proposes a joint Scale Down and Scale Up network architecture suitable for voice calls. This network combines the advantages of Unet and autoencoder, while greatly saving network parameters and computing overhead. In the experiment, under the same network settings, the performance of X-net exceeded the blind bandwidth expansion, and in voice calls, X-net achieved a significant improvement in MOS scores.

[0205] The network architecture proposed in this application adopts two-step training when training the model, using different loss functions to train X-net to optimize the auditory experience and adapt to call speech coding.

[0206] In the first step of training, the minimum mean square error of the time domain waveform is used as the loss function, and in the second step of training, the minimum mean square error of the spectrum amplitude is used as the loss function. The spectrum amplitude is obtained by short-time Fourier transform of the time domain signal. The correct initial phase obtained in the first step of training is used for the second step of training to avoid the mismatch between the phase of the input and the result caused by not using the spectrum phase information when fading out the frequency domain loss function. The loss function of the network is the weighted sum of the Scale Down and Scale Up network loss functions. The typical weighted sum is that the weight coefficients of both are 1.

[0207] Loss X-net =α·Loss scale down +β·Loss scale up

[0208] Among them, Loss X-net is the loss function of X-net, Loss scale down is the loss function of the Scale Down network, α is its weight, Loss scale up is the loss function of the Scale Up network, and β is its weight.

[0209] Fig.11 is a block diagram illustrating a voice transmitting apparatus 100 according to an exemplary embodiment of the present invention.

[0210] Fig.11 The speech transmission device shown in is a speech transmission device based on a neural network, wherein the neural network may be a convolutional neural network (CNN), that is, the speech transmission device may be based on Figure 3 The speech transmission device of the speech transmission method based on CNN shown in . Fig.11 As shown in , the speech sending device 100 may include a low-frequency feature information extraction unit 110 and a sending unit 120. Among them, the low-frequency feature information extraction unit 110 corresponds to Figure 3 In the multi-filter scale feature extraction (low frequency) process, the sending unit 120 corresponds to Figure 3 However, the present invention is not limited thereto, and the feature extraction network in the receiving method of the present invention can also be implemented using a DNN (deep neural network).

[0211] In order to facilitate understanding of the present invention, in the following description, Figure 3 The processing process of the CNN-based speech transmission method is shown in Fig.11 The voice sending device is described in detail.

[0212] The low-frequency feature information extraction unit 110 may be configured to extract low-frequency feature information from the input speech signal using a first feature extraction network.

[0213] Specifically, the first feature extraction network may include at least one first feature extraction module and at least one second feature extraction module. Figure 3 As shown in , the first feature extraction network may correspond to Figure 3 In the multi-convolution kernel scale feature extraction (low frequency) process, the at least one first feature extraction module may correspond to Figure 3 The feature extraction layer in the at least one second feature extraction module may correspond to Figure 3 Specifically, Fig.12 As described in the text, the low-frequency feature information extraction unit 110 may include a speech feature information extraction unit 111 and a low-frequency feature information acquisition unit 112.

[0214] The speech feature information extraction unit 111 may be configured to extract speech feature information of an input speech signal using the at least one first feature extraction module. At least one first feature extraction module in the first feature extraction network may perform feature extraction on the input information and output the extracted feature information respectively through at least two convolution processing parameters, and the at least two convolution processing parameters include at least two of the following: a convolution processing parameter corresponding to a receptive field between adjacent sampling points of a speech signal, a convolution processing parameter corresponding to a receptive field of a pitch length, and a convolution processing parameter corresponding to a receptive field of at least two pitch lengths, wherein the convolution processing parameter may include a convolution kernel size corresponding to the convolution operation.

[0215] Specifically, if Figure 3 As shown in FIG. 1 , the speech feature information extraction unit 111 can Figure 3 The multi-scale convolution operation in the feature extraction layer is learned at a high-frequency sampling rate to learn the high-frequency and low-frequency features of the speech. That is, the speech feature information includes high-frequency and low-frequency features. Specifically, in order to learn the characteristics of different high and low-frequency sub-bands, the convolution operation can adopt different convolution kernel sizes. This is because different convolution kernel sizes are equivalent to different frequency domain resolution capabilities. The larger the convolution kernel size, the more complex the convolution operation that can be fitted, and the more and more refined the correlation information that can be learned. Therefore, under different convolution kernel sizes, the convolution operation can learn characteristics covering different frequency resolutions. Therefore, in Figure 3 In , the feature extraction layer performs multi-scale feature extraction on the input speech signal under different convolution kernel sizes, such as Figure 3 As shown in , there are three convolution kernel sizes in the feature extraction layer, namely 8, 16 and 32, so Figure 4 As shown in , convolution kernels of different sizes can cover the receptive field between adjacent sampling points of the input speech signal, the receptive field with one pitch length, and the receptive field with at least two pitch lengths. That is to say, for features that need to cover the scale of adjacent sampling points, a small convolution kernel size of 8 can be selected, for features that need to cover the scale within the same pitch, a medium convolution kernel size of 16 can be selected, and for features that need to cover the scale between different pitches, a large convolution kernel size of 32 can be selected. However, this is only an example, and any convolution kernel size that can cover the receptive field between adjacent sampling points, the receptive field with one pitch length, and the receptive field with at least two pitch lengths can be selected.

[0216] In addition, Figure 3 In the multi-scale convolution operation in the feature extraction layer, each scale of the convolution operation includes two convolution operations, and each convolution operation includes a convolution kernel, and the step size of each scale of the convolution operation in the multi-scale convolution operation in the feature extraction layer is 1, so that the signal scale does not change. For example, if the shape of the input speech signal is [640,1], where 640 represents the length of the input speech signal and 1 represents the number of speech, then the shape of the output from the feature extraction layer is 3*[640,2], where 3 represents that the output of the feature extraction layer is data of three scales, that is, the shape of the signal output by each scale of the multi-scale convolution operation in the feature extraction layer is [640,2].

[0217] The low-frequency feature information obtaining unit 112 may obtain the low-frequency feature information according to the extracted speech feature information by using the at least one second feature extraction module.

[0218] Specifically, the low-frequency feature information acquisition unit 112 can extract features from the input information through at least two convolution processing parameters and output the extracted feature information. That is to say, at least one second feature extraction module in the first feature network can extract features from the input speech feature information through at least two convolution processing parameters and output the extracted feature information. Among them, the at least two convolution processing parameters include at least two of the following: convolution processing parameters corresponding to the receptive field between adjacent sampling points of the speech signal, convolution processing parameters corresponding to the receptive field of one fundamental pitch length, and convolution processing parameters corresponding to the receptive field of at least two fundamental pitch lengths, wherein the convolution processing parameters may include the convolution kernel size corresponding to the convolution operation. Since the above has been referred to Figure 3 The convolution processing parameters have been described in detail, so they will not be repeated here.

[0219] like Figure 3As shown in , the low-frequency feature information extraction unit 110 convolves the speech features by setting the step size of the convolution operation in the downsampling layer to 2 to obtain low-frequency feature information including the relevant information between the high-frequency features and the low-frequency features. In detail, the low-frequency feature information extraction unit 110 can perform convolution and sampling rate conversion on the extracted speech features using the multi-scale convolution operation in the downsampling layer. In other words, the low-frequency feature information extraction unit 110 can perform convolution and sampling rate conversion on the speech features through the multi-scale convolution operation in the downsampling layer, thereby integrating the relevant information between the high-frequency features and the low-frequency features into the low-frequency feature information to obtain the low-frequency feature information including the relevant information.

[0220] In addition, Figure 3 In the example shown in , the low-frequency feature information extraction unit 110 may use the same convolution processing parameters as the convolution processing parameters in the at least one first feature extraction module in the at least one second feature extraction module, that is, the multi-scale convolution operations with the same number and size as the multi-scale convolution operations in the feature extraction layer are used in the downsampling layer, for example, Figure 3 As shown in , the multi-scale convolution operation in the downsampling layer has three convolution kernel sizes, namely 8, 16 and 32, and each convolution operation in the multi-scale convolution operation has two convolution operations, and each convolution operation is implemented by a convolution kernel. Therefore, Figure 3 As shown in , in order to perform convolution and sampling rate conversion through the multi-scale convolution operation in the downsampling layer, Figure 3 The example shown in sets the step size of each convolution operation in the multi-scale convolution operation in the downsampling layer to 2. Therefore, if the shape of the signal output from the feature extraction layer is 3*[640,2], the shape of the signal output from the downsampling layer is 3*[320,2]. Here, the reason for setting the step size of the convolution operation to 2 is that the frequency of SWB is twice the frequency of WB.

[0221] In addition, although the low-frequency feature information extraction unit 110 uses the same convolution processing parameters as the convolution processing parameters in the at least one first feature extraction module in the at least one second feature extraction module, the present invention is not limited thereto, and the at least one second feature extraction module may also use convolution processing parameters different from the convolution processing parameters used by the at least one first feature extraction module. In this case, the form of speech feature information suitable for being processed by the at least one second feature extraction module can be obtained by fusing the speech feature information extracted by the at least one first feature extraction module. For example, assuming Figure 3The multi-scale convolution operation in the downsampling layer has three convolution kernel sizes, namely 8, 16 and 32, while the multi-scale convolution operation in the upsampling layer has two convolution kernel sizes, namely 16 and 32. In this case, the feature extraction layer can fuse the speech feature information output from the convolution operation with a convolution kernel size of 8 to obtain the same form of speech feature information as the speech feature information output by the convolution operation with a convolution kernel size of 16 or 32.

[0222] The sending unit 120 may be configured to send a speech signal corresponding to the low-frequency feature information to a receiving end.

[0223] Specifically, the sending unit 120 may be configured to send the speech signal corresponding to the low-frequency feature information to the receiving end by: using the first feature fusion network, fusing all the low-frequency feature information output by the first feature extraction network to obtain the speech signal corresponding to the low-frequency feature information, and sending the speech signal corresponding to the low-frequency feature information to the receiving end. Figure 3 In the example shown, the first feature fusion network corresponds to Figure 3 In detail, after the low-frequency feature information extraction unit 110 obtains the low-frequency feature information at different scales through the downsampling layer, the sending unit 120 may perform convolution processing on the low-frequency feature information at different scales obtained through the downsampling layer by setting the step size of a single convolution operation in the feature fusion layer to 1 to obtain a speech signal corresponding to the low-frequency feature information, such as Figure 3 As shown in , the low-frequency feature information extraction unit 110 can obtain low-frequency feature information at three scales through the downsampling layer. Thereafter, the sending unit 120 can perform convolution processing on the low-frequency feature information at the three scales through the feature fusion layer to obtain the speech signal corresponding to the low-frequency feature information that needs to be output in the end, and send the speech signal to the receiving end. For example, if the shape of the signal output from the downsampling layer is 3*[320,2], the shape of the signal output from the feature fusion layer is [320,1].

[0224] In addition, in practical applications, the speech sending device 100 is actually another speech receiving device, so the speech sending device 100 may also include: a receiving unit configured to receive a speech signal sent by the receiving end; a high-frequency feature information recovery unit configured to extract low-frequency feature information from the received speech signal using a second feature extraction network and recover high-frequency feature information; an output unit configured to output a speech signal containing the low-frequency feature information and the high-frequency feature information. This will be described in detail below when describing the speech receiving method.

[0225] In addition, although the above description takes CNN as an example to describe the speech transmission device 100 based on the neural network, the neural network based on the speech transmission device 100 of the present invention can also be a deep neural network (DNN). In other words, the first feature extraction network and the first feature fusion network can be implemented based on the DNN, such as Figure 5 As shown in the figure, the input high-frequency speech signal is first transformed from the time domain to the frequency domain, and then the frequency domain information is transformed from high frequency to low frequency through the DNN to obtain the low-frequency frequency domain information, and finally the low-frequency frequency domain information is transformed from the frequency domain to the time domain to obtain the output low-frequency time domain signal.

[0226] Compared with the traditional voice sending device, the voice sending device 100 described above can completely retain the low-frequency features of the input voice signal, and at the same time integrate the high-frequency features into the low-frequency features, wherein the feature extraction layer can completely retain more information in the input voice signal, making the low-frequency information more complete, and the downsampling layer can integrate the high-frequency features of the voice signal into the low-frequency features so that the high-frequency features can be better reconstructed in the voice receiving device. In addition, since the traditional downsampling method only uses a simple convolution operation of a single scale, the low-frequency information of the voice signal will be lost during the sampling rate conversion process, making it difficult to restore the complete low-frequency features in the voice receiving device, and it is also difficult to better restore the high-frequency features during the upsampling process. The two technologies adopted by the present invention will improve the clarity of the voice, thereby enhancing the user's subjective experience.

[0227] Fig.13 is a block diagram showing the configuration of the voice receiving apparatus 200 according to an exemplary embodiment of the present invention.

[0228] like Fig.13 As shown in , the speech receiving device 200 may include a receiving unit 210 , a high frequency feature information restoration unit 220 and an output unit 230 .

[0229] The receiving unit 210 may be configured to receive a speech signal sent by a transmitting end. The high-frequency feature information recovery unit 220 may be configured to extract low-frequency feature information from the received speech signal and recover high-frequency feature information using a second feature extraction network. The output unit 230 may output a speech signal including the low-frequency feature information and the high-frequency feature information.

[0230] Fig.13 The speech receiving device shown in is a speech receiving device 200 based on a neural network, wherein the neural network may be a convolutional neural network (CNN), that is, the speech receiving device 200 may correspond to Figure 8 The CNN-based speech reception method shown in . Figure 8As shown in , the speech receiving process includes three processes: a multi-filter scale feature extraction (low frequency) process, a multi-filter scale feature extraction (high frequency) process and a feature fusion process, wherein the multi-filter scale feature extraction can also be referred to as a multi-convolution kernel scale feature extraction. The multi-convolution kernel scale feature extraction (low frequency) process and the multi-convolution kernel scale feature extraction (high frequency) process can correspond to the high frequency feature information recovery unit 220, and the feature fusion process can correspond to the output unit 230. However, the present invention is not limited thereto, and the feature extraction network in the speech receiving device of the present invention can also be implemented using a DNN (deep neural network).

[0231] In one embodiment, the speech receiving device 200 may further include a preprocessing unit (not shown), which may be configured to expand the data scale of the received speech signal by performing data replication on the received speech signal before extracting low-frequency feature information from the received speech signal and restoring high-frequency feature information using the second feature extraction network. The preprocessing unit corresponds to Figure 8 The preprocessing unit can complete the scale expansion by performing interval repeated copying on the received data. For example, if the received data is [0, 1, 2, 3, 4], the expanded data is [0, 0, 1, 1, 2, 2, 3, 3, 4, 4]. This step can complete the formal symmetric expansion from low-frequency frequency domain information to high-frequency frequency domain information.

[0232] The high-frequency feature information recovery unit 220 can be configured to extract low-frequency feature information from the received speech signal and recover the high-frequency feature information using the second feature extraction network through the following operations: using the low-frequency feature extraction network in the second feature extraction network to extract low-frequency feature information from the received speech signal, wherein the low-frequency feature information contains relevant information between the high-frequency features and the low-frequency features.

[0233] Specifically, the low-frequency feature extraction network includes at least one third feature extraction module and at least one fourth feature extraction module, such as Figure 8 As shown in , the second feature extraction network can correspond to Figure 8 The multi-convolution kernel scale feature extraction (low frequency) process and the multi-convolution kernel scale feature extraction (high frequency) process in the low-frequency feature extraction network may correspond to Figure 8 In the multi-convolution kernel scale feature extraction (low frequency) process, the at least one third feature extraction module may correspond to Figure 8 The feature extraction layer in the embodiment of the present invention, the at least one fourth feature extraction module may correspond to Figure 8The downsampling layer in the second feature extraction network. The low-frequency feature extraction network may include one or more feature extraction layers, and may also include one or more downsampling layers. Specifically, the high-frequency feature information recovery unit 220 may be configured to extract low-frequency feature information from the received speech signal using the low-frequency feature extraction network in the second feature extraction network by the following operation: using the at least one third feature extraction module to extract the speech feature information of the received speech signal. Among them, at least one third feature extraction module in the second feature extraction network performs feature extraction on the input information respectively through at least two convolution processing parameters, and outputs the extracted feature information. Among them, the at least two convolution processing parameters include at least two of the following: a convolution processing parameter corresponding to the receptive field between adjacent sampling points of the speech signal, a convolution processing parameter corresponding to the receptive field of one fundamental pitch length, and a convolution processing parameter corresponding to the receptive field of at least two fundamental pitch lengths, wherein the convolution processing parameter may include a convolution kernel size corresponding to the convolution operation. Since the above has been referred to Figure 3 This has been described in detail, so the convolution processing parameters will not be repeated here.

[0234] Specifically, if Figure 8 As shown in , the high-frequency feature information recovery unit 220 can perform multi-scale feature extraction on the expanded speech signal by setting the step size of the multi-scale convolution operation in the feature extraction layer to 1 to obtain the speech feature information. In this process, the high-frequency feature information recovery unit 220 does not change the scale of the signal, so the step size of the multi-scale convolution operation in the feature extraction layer is set to 1. In addition, the feature extraction layer reuses the design of the feature extraction layer in the speech transmission method. In other words, the number and size of the convolution operations in the feature extraction layer in the speech receiving device 200 are consistent with the number and size of the convolution operations in the feature extraction layer in the speech transmitting device 100. In other words, the feature extraction layer in the speech receiving device 200 reuses the design of the feature extraction layer in the speech transmitting device 100, which enables the AI-based upsampling process of the speech receiving device 200 to better restore the information obtained by the AI-based downsampling process of the speech transmitting device 100. Therefore, compared with Figure 3 The schematic diagram of the AI-based downsampling process shown in corresponds to, Figure 8The multi-scale convolution operation in the feature extraction layer in the AI-based upsampling process shown in also has three convolution kernel sizes, namely 8, 16 and 32, and each of the multi-scale convolution operations has two convolution operations, so the speech receiving device 200 is able to extract speech features by utilizing multi-scale convolution operations of different sizes. For example, if the shape of the input expanded speech signal is [640,1], since the step size is set to 1, the shape output from the feature extraction layer is 3*[640,2]. Among them, the convolution operation of each scale of the feature extraction layer may also include one convolution operation, or include more than two convolution operations.

[0235] In addition, the high-frequency feature information recovery unit 220 may be configured to extract low-frequency feature information from the received speech signal using the low-frequency feature extraction network in the second feature extraction network by further performing the following operation: using the at least one fourth feature extraction module to obtain low-frequency feature information based on the extracted speech feature information. Wherein, the at least one fourth feature extraction module in the second feature extraction network performs feature extraction on the input information respectively through at least two convolution processing parameters, and outputs the extracted feature information. Wherein, the at least two convolution processing parameters include at least two of the following: a convolution processing parameter corresponding to the receptive field between adjacent sampling points of the speech signal, a convolution processing parameter corresponding to the receptive field of one fundamental pitch length, and a convolution processing parameter corresponding to the receptive field of at least two fundamental pitch lengths, wherein the convolution processing parameter may include a convolution kernel size corresponding to the convolution operation. Since the above has been referred to Figure 3 This has been described in detail, so the convolution processing parameters will not be repeated here.

[0236] Specifically, for example, Figure 8 As shown in , when obtaining the low-frequency feature information, the low-frequency feature information can be obtained by setting the step size of the multi-scale convolution operation in the downsampling layer to 2 and convolving the speech features.

[0237] In addition, the downsampling layer reuses the design of the downsampling layer in the speech transmitting device 100. In other words, the number and size of the convolution operations in the downsampling layer in the speech receiving device 200 are consistent with the number and size of the convolution operations in the downsampling layer in the speech transmitting device 100. That is, the downsampling layer in the speech receiving device 200 reuses the design of the downsampling layer in the speech transmitting device 100, which enables the AI-based upsampling process of the speech receiving device 200 to better restore the information obtained by the AI-based downsampling process of the speech transmitting device 100. Accordingly, the speech receiving device 200 uses the same number and size of multi-scale convolution operations in the downsampling layer as the multi-scale convolution operations in the feature extraction layer, for example, Figure 8As shown in , the multi-scale convolution operation in the downsampling layer has three convolution kernel sizes, namely 8, 16 and 32, and each convolution operation in the multi-scale convolution operation has two convolution operations, and each convolution operation has one convolution kernel. Therefore, Figure 8 As shown in , in order to perform sampling rate conversion by convolution through the multi-scale convolution operation in the downsampling layer, Figure 8 The example shown in sets the step size of each convolution operation in the multi-scale convolution operation in the downsampling to 2. Therefore, if the shape of the signal output from the feature extraction layer is 3*[640,2], the shape of the signal output from the downsampling layer is 3*[320,2]. Here, the reason for setting the step size of the convolution operation to 2 is that the frequency of SWB is twice the frequency of WB.

[0238] Reference above Figure 8 The feature extraction layer and downsampling layer in the AI-based upsampling process of the described speech receiving device 200 and the feature extraction layer and downsampling layer in the AI-based downsampling process of the speech sending device 100 use the same size and number of convolution operations on the same signal scale, so that the AI-based upsampling process of the speech receiving device 200 and the AI-based downsampling process of the speech sending device 100 can extract low-frequency information and recover signals from the information in a consistent manner. Such operations enable the AI-based upsampling process of the speech receiving device 200 to better recover the information processed by the AI-based downsampling process of the speech sending device 100.

[0239] The high-frequency feature information recovery unit 220 is configured to also use the second feature extraction network to extract low-frequency feature information from the received speech signal and recover the high-frequency feature information through the following operations: using the high-frequency feature extraction network in the second feature extraction network to recover the high-frequency feature information based on the low-frequency feature information and to fuse the high-frequency feature information and the low-frequency feature information to obtain feature information containing high-frequency feature information and low-frequency feature information.

[0240] Specifically, the high-frequency feature extraction network includes at least one fifth feature extraction module and at least one sixth feature extraction module, such as Figure 8 As shown in , the high-frequency feature extraction network may correspond to Figure 8 In the multi-convolution kernel scale feature extraction (high frequency) process, the at least one fifth feature extraction module may correspond to Figure 8 The bottleneck layer in Figure 8 The convolution layer in the upsampling layer in (not shown in the figure), the at least one sixth feature extraction module may correspond to Figure 8Specifically, the high-frequency feature information recovery unit 220 may be configured to recover the high-frequency feature information according to the low-frequency feature information using the high-frequency feature extraction network in the second feature extraction network and fuse the high-frequency feature information and the low-frequency feature information by performing the following operations: recover the high-frequency feature information according to the low-frequency feature information using the at least one fifth feature extraction module.

[0241] Among them, at least one fifth feature extraction module in the second feature extraction network extracts features from the input information respectively through at least two convolution processing parameters, and outputs the extracted feature information. Among them, the at least two convolution processing parameters include at least two of the following: convolution processing parameters corresponding to the receptive field between adjacent sampling points of the speech signal, convolution processing parameters corresponding to the receptive field of one pitch length, and convolution processing parameters corresponding to the receptive field of at least two pitch lengths, wherein the convolution processing parameters may include the convolution kernel size corresponding to the convolution operation. As the above has been referred to Figure 3 This has been described in detail, so the convolution processing parameters will not be repeated here.

[0242] Specifically, if Figure 8 As shown in , the high-frequency feature information recovery unit 220 may convolve the low-frequency feature information by setting the step size of the multi-scale convolution operation in the bottleneck layer to 2 to obtain the convolved feature information. The low-frequency feature information obtained by the downsampling layer is actually low-frequency feature information that incorporates the relevant information between the high-frequency features and the low-frequency features, and the bottleneck layer may further fuse the low-frequency feature information containing the relevant information. The bottleneck layer may be implemented by bottleneck convolution. For example, the multi-scale low-frequency feature information output from the downsampling layer may be processed by setting the step size of the multi-scale convolution operation in the bottleneck layer to 2. Multi-scale convolution operations with the same number and size as those of the multi-scale convolution operations in the downsampling layer may be used in the bottleneck layer, such as Figure 8 As shown in , each scale of the convolution operation in the multi-scale convolution operation in the bottleneck layer includes two convolution operations, and as Figure 8 As shown in , the convolution kernel size can be 8, 16, and 32. Therefore, Figure 8 As shown in , if the shape of the signal output from the downsampling layer is 3*[320,2], the shape of the signal output from the bottleneck layer is 3*[160,2].

[0243] The convolution operation of each scale of the feature extraction layer may also include one convolution operation, or include more than two convolution operations.

[0244] In addition, if Figure 8As shown in , the high-frequency feature information recovery unit 220 can obtain the high-frequency feature information by setting the step size of the multi-scale convolution operation in the upsampling layer to 1 to convolve the low-frequency feature information and arrange the convolution results. The high-frequency feature information recovery unit 220 can use the same number and size of convolution operations as the multi-scale convolution operation in the bottleneck layer in the upsampling layer, such as Figure 8 As shown in , each of the multi-scale convolution operations of the convolution layer in the upsampling layer includes four convolution operations, and each convolution operation has a convolution kernel, and as shown in Figure 8 As shown in , the convolution kernel size can be 8, 16 and 32. In this process, the high-frequency feature information recovery unit 220 first sets the step size of the multi-scale convolution operation of the convolution layer in the upsampling layer to 1 to perform convolution processing on the low-frequency feature information output by the bottleneck layer. For example, if the shape of the signal output from the bottleneck layer is 3*[160,2], then the result of the convolution is 3*[160,4]. This convolution processing integrates the information obtained by the bottleneck layer. Thereafter, the upsampling layer rearranges the result of the convolution to further obtain the rearranged high-frequency feature information. For example, if the result of the convolution is 3*[160,4], then the rearranged high-frequency feature information is 3*[320,2], wherein the rearrangement method adopted is a cross-arrangement method, for example, as Figure 8 As shown in, under each convolution operation, the result of the convolution is 4 sets of data (such as Figure 8 In the embodiment of the present invention, the data in the first group and the second group are arranged crosswise, and the data in the third group and the fourth group are arranged crosswise, but the method of rearrangement is only exemplary and the present invention is not limited thereto.

[0245] Through the above operations, the high-frequency feature information restoration unit 220 can restore the high-frequency feature information.

[0246] Since in the embodiment of the present application, the low-frequency feature information of the voice signal sent by the sending end includes more relevant information between the high-frequency features and the low-frequency features, the receiving end can learn the relationship between the high-frequency features and the low-frequency features by using the relevant information, and can better restore the high-frequency features based on the low-frequency features.

[0247] In addition, the high-frequency feature information recovery unit 220 can be configured to also use the high-frequency feature extraction network in the second feature extraction network to recover the high-frequency feature information based on the low-frequency feature information and fuse the high-frequency feature information and the low-frequency feature information through the following operations: using at least one sixth feature extraction module to fuse the high-frequency feature information and the low-frequency feature information extracted by the corresponding fourth feature extraction module to obtain feature information containing the high-frequency feature information and the low-frequency feature information.

[0248] Among them, the at least one sixth feature extraction module fuses the high-frequency feature information corresponding to the at least two input convolution processing parameters and the low-frequency feature information extracted by the corresponding fourth feature extraction module according to the corresponding convolution processing parameters, and outputs feature information containing the high-frequency feature information and the low-frequency feature information. Among them, the at least two convolution processing parameters include at least two of the following: a convolution processing parameter corresponding to the receptive field between adjacent sampling points of the speech signal, a convolution processing parameter corresponding to the receptive field of one fundamental pitch length, and a convolution processing parameter corresponding to the receptive field of at least two fundamental pitch lengths, wherein the convolution processing parameter may include a convolution kernel size corresponding to the convolution operation. As the above has been referred to Figure 3 This has been described in detail, so the convolution processing parameters will not be repeated here.

[0249] Specifically, if Figure 8 As shown in , the high-frequency feature information and the low-frequency feature information can be spliced ​​to obtain the data containing high- and low-frequency information. Figure 8 , the high-frequency feature information output by the convolution layer in the upsampling layer and the low-frequency feature information output by the convolution processing in the downsampling layer can be spliced ​​together to obtain feature information containing high-frequency feature information and low-frequency feature information at different scales. For example, if the shape of the signal output from the convolution layer in the upsampling layer is 3*[320,2], and the shape of the signal output by the convolution processing in the downsampling layer is 3*[320,2], the result of splicing is 3*[640,2]. This step is to simply splice the high-frequency feature information in front of the low-frequency feature information to obtain feature information containing high-frequency feature information and low-frequency feature information.

[0250] The output unit 230 may use the second feature fusion network to fuse the feature information including the high-frequency feature information and the low-frequency feature information output by the second feature extraction network, obtain a speech signal corresponding to the feature information including the high-frequency feature information and the low-frequency feature information, and output the speech signal. Figure 8As shown in , the feature information including high-frequency feature information and low-frequency feature information can be fused by setting the step size of a single convolution operation in the feature fusion layer of CNN to 1 to obtain the output signal. In addition, the feature fusion layer reuses the design of the feature fusion layer in the speech transmission method. In other words, the number and size of the convolution operations of the feature fusion layer are consistent with the number and size of the convolution operations of the feature fusion layer in the speech transmission device 100. In other words, the feature fusion layer in the speech receiving device 200 reuses the design of the feature fusion layer in the speech transmission method. Figure 8 The speech receiving device 200 obtains a speech signal corresponding to the feature information including high-frequency feature information and low-frequency feature information by convolving the feature information including high-frequency feature information and low-frequency feature information at different scales using the feature fusion layer. For example, if the shape of the signal input to the feature fusion layer is 3*[640,2], the shape of the signal output from the feature fusion layer is [640,1].

[0251] Compared with the traditional downsampling method, the downsampling method of the present invention can better preserve the downsampled information and improve the language intelligibility. In addition, the method of jointly designing upsampling and downsampling of the present invention can more effectively improve the SNR value and POLQA test results.

[0252] Reference above Figure 8 and Fig.13 An exemplary embodiment of the speech receiving device 200 is described below. Fig.10a and Fig.13 Another exemplary embodiment of the speech receiving apparatus 200 will be described.

[0253] In this embodiment, the speech receiving device 200 may not include the preprocessing unit described above, but include a receiving unit 210, a high-frequency feature information recovery unit 220 and an output unit 230. In this case, the high-frequency feature information recovery unit 220 may be configured to extract low-frequency feature information from the received speech signal and recover high-frequency feature information using the second feature extraction network by the following operations: using at least one seventh feature extraction module to extract speech feature information of the received speech signal. The at least one seventh feature module is included in the second feature extraction network, and the second feature extraction network also includes at least one eighth feature extraction module. Fig.10a As shown in , the second feature extraction network can correspond to Fig.10a In the multi-convolution kernel scale feature extraction (high frequency) process, the at least one seventh feature extraction module may correspond to Fig.10a The feature extraction layer in the embodiment of the present invention, the at least one eighth feature extraction module may correspond to Fig.10a The upsampling layer in .

[0254] Specifically, the at least one seventh feature extraction module extracts features from the input information respectively through at least two convolution processing parameters, and outputs the extracted speech feature information. The at least two convolution processing parameters include at least two of the following: a convolution processing parameter corresponding to the receptive field between adjacent sampling points of the speech signal, a convolution processing parameter corresponding to the receptive field of one pitch length, and a convolution processing parameter corresponding to the receptive field of at least two pitch lengths, wherein the convolution processing parameter may include a convolution kernel size corresponding to the convolution operation. Figure 3 This has been described in detail, so the convolution processing parameters will not be repeated here.

[0255] Specifically, if Fig.10a As shown in , the high-frequency feature information recovery unit 220 can perform multi-scale feature extraction on the received speech signal by setting the step size of the multi-scale convolution operation in the feature extraction layer to 1 to obtain the speech feature information. In this process, the speech receiving device 200 does not change the scale of the signal, so the step size of the multi-scale convolution operation in the feature extraction layer is set to 1. In addition, the feature extraction layer reuses the design of the feature extraction layer in the speech sending device 100. In other words, the number and size of the convolution operations in the feature extraction layer in the speech receiving device 200 are consistent with the number and size of the convolution operations in the feature extraction layer in the speech sending device 100. In other words, the feature fusion layer in the speech receiving device 200 reuses the design of the feature fusion layer in the speech sending device 100, which enables the speech receiving device 200 to better restore the information processed by the speech sending device 100. Therefore, compared with Figure 3 The schematic diagram of the AI-based downsampling process shown in corresponds to, Fig.10a The multi-scale convolution operation in the feature extraction layer in the AI-based upsampling process shown in also has three convolution kernel sizes, namely 8, 16 and 32, and each convolution operation in the multi-scale convolution operation has two convolution operations, and each convolution operation has a convolution kernel, so the speech receiving device 200 can extract speech features by using multi-scale convolution operations of different sizes. For example, if the shape of the input data to be processed is [320,1], then since the step size is set to 1, the shape output from the feature extraction layer is 3*[320,2].

[0256] The high-frequency feature information recovery unit 220 may be configured to further extract low-frequency feature information and recover high-frequency feature information from the received speech signal using the second feature extraction network through the following operations: using at least one eighth feature extraction module, based on the extracted speech feature information, extract low-frequency feature information containing relevant information between high-frequency features and low-frequency features, and recover high-frequency feature information to obtain feature information containing high-frequency feature information and low-frequency feature information. The at least one eighth feature extraction module extracts low-frequency feature information from the input information respectively through at least two deconvolution processing parameters, and recovers high-frequency feature information to obtain feature information containing high-frequency feature information and low-frequency feature information. Among them, the at least two convolution processing parameters include at least two of the following: a convolution processing parameter corresponding to the receptive field between adjacent sampling points of the speech signal, a convolution processing parameter corresponding to the receptive field of one fundamental pitch length, and a convolution processing parameter corresponding to the receptive field of at least two fundamental pitch lengths, wherein the convolution processing parameter may include a convolution kernel size corresponding to the convolution operation. Since the above has been referred to Figure 3 This has been described in detail, so the convolution processing parameters will not be repeated here.

[0257] Specifically, if Figure 8 As shown in , when the high-frequency feature information recovery unit 220 obtains feature information including high-frequency feature information and low-frequency feature information, the high-frequency feature information recovery unit 220 can deconvolve the speech feature by setting the step size of the multi-scale convolution operation in the upsampling layer to 2 to extract low-frequency feature information from the input information, and restore the high-frequency feature information, thereby obtaining feature information including high-frequency feature information and low-frequency feature information. In this process, the speech receiving device 200 needs to complete the scale expansion and upsampling rate recovery of the feature, so the speech receiving device 200 implements the scale expansion and upsampling rate recovery of the feature by deconvolving the data input to the upsampling layer to obtain feature information including high-frequency feature information and low-frequency feature information, wherein the speech receiving device 200 uses the same number and size of convolution operations as the multi-scale convolution operations in the feature extraction layer in the upsampling layer, as shown in FIG. Fig.10a As shown in , if the shape of the signal output from the feature extraction layer is 3*[320,2], the shape of the signal output from the upsampling layer is 3*[640,2].

[0258] The output unit 230 may be configured to output a speech signal containing the low-frequency feature information and the high-frequency feature information by the following operation: using the second feature fusion network, the feature information containing the high-frequency feature information and the low-frequency feature information output by the second feature extraction network is fused to obtain a speech signal corresponding to the feature information containing the high-frequency feature information and the low-frequency feature information, and output the speech signal. Fig.10a As shown in , the output unit 230 can fuse the feature information including high-frequency feature information and low-frequency feature information by setting the step size of a single convolution operation in the feature fusion layer of CNN to 1 to obtain the speech signal. In addition, the feature fusion layer reuses the design of the feature fusion layer in the speech transmission device 100. In other words, the number and size of the convolution operations of the feature fusion layer are consistent with the number and size of the convolution operations of the feature fusion layer in the speech transmission method. In other words, the feature fusion layer in the speech receiving device 200 reuses the design of the feature fusion layer in the speech transmission method. Fig.10a The speech receiving device 200 obtains a speech signal corresponding to the feature information including high-frequency feature information and low-frequency feature information by convolving the feature information including high-frequency feature information and low-frequency feature information at different scales using the feature fusion layer. For example, if the shape of the signal input to the feature fusion layer is 3*[640,2], the shape of the signal output from the feature fusion layer is [640,1].

[0259] Reference above Fig.10a and Fig.13 An exemplary embodiment of the speech receiving device 200 is described below. Fig.10c and Fig.13 Another exemplary embodiment of the speech receiving device 200 is described below.

[0260] In this embodiment, the high-frequency feature information recovery unit 220 can not only repair the low-frequency time domain signal, but also recover the high-frequency time domain signal. Therefore, the high-frequency feature information recovery unit 220 can also be called a high- and low-frequency time domain signal generation unit or a high- and low-frequency time domain signal recovery unit.

[0261] First, the high-frequency feature information recovery unit 220 needs to recover a low-frequency time domain signal from the received speech signal, where the low-frequency time domain signal repairs the low-frequency spectrum lost when the speech signal is encoded at the encoding end.

[0262] Specifically, the high-frequency feature information recovery unit 220 can be configured to extract low-frequency feature information from the received speech signal and recover high-frequency feature information using the second feature extraction network through the following operations: extract low-frequency feature information from the received speech signal using the low-frequency feature extraction network in the second feature extraction network.

[0263] like Fig.10c As shown in , the second feature extraction network can correspond to Fig.10c The multi-filter scale feature extraction (low-frequency restoration) process and the multi-filter scale feature extraction (high-frequency) process in the low-frequency feature extraction network may correspond to Fig.10cIn the multi-filter scale feature extraction (low-frequency restoration) process, the at least one ninth feature extraction module may correspond to Fig.10c The feature extraction layer in the multi-filter scale feature extraction (low-frequency restoration) process in . In addition, the low-frequency feature extraction network can contain one or more feature extraction layers.

[0264] Specifically, the low-frequency feature extraction network includes at least one ninth feature extraction module, wherein the high-frequency feature information recovery unit 220 can be configured to extract low-frequency feature information from the received speech signal using the low-frequency feature extraction network in the second feature extraction network through the following operation: using the at least one ninth feature extraction module to extract the low-frequency feature information of the received speech signal. The at least one ninth feature extraction module performs feature extraction on the input information respectively through at least two convolution processing parameters, and outputs the extracted low-frequency feature information. Among them, the at least two convolution processing parameters are at least two of the following: a convolution processing parameter corresponding to the receptive field between adjacent sampling points of the speech signal, a convolution processing parameter corresponding to the receptive field of one fundamental pitch length, and a convolution processing parameter corresponding to the receptive field of at least two fundamental pitch lengths, wherein the convolution processing parameter may include a convolution kernel size corresponding to the convolution operation. Since the above has been referred to Figure 3 This has been described in detail, so the convolution processing parameters will not be repeated here.

[0265] Specifically, if Fig.10c As shown in , the high-frequency feature information recovery unit 220 can perform multi-scale feature extraction on the received speech signal to obtain low-frequency feature information by setting the step size of the multi-scale convolution operation in the feature extraction layer in the multi-filter scale feature extraction (low-frequency repair) process to 1. In this process, the high-frequency feature information recovery unit 220 does not change the scale of the signal, so the step size of the multi-scale convolution operation in the feature extraction layer in the multi-filter scale feature extraction (low-frequency repair) process is set to 1.

[0266] In addition, the feature extraction layer reuses the design of the feature extraction layer in the voice transmitting device 100. In other words, in the voice receiving device 200, the number and size of convolution operations in the feature extraction layer in the multi-filter scale feature extraction (low-frequency restoration) process are consistent with the number and size of convolution operations in the feature extraction layer in the voice transmitting device 100. That is to say, the feature extraction layer in the multi-filter scale feature extraction (low-frequency restoration) process in the voice receiving device 200 reuses the design of the feature extraction layer in the voice transmitting device 100, which enables the AI-based upsampling process of the voice receiving device 200 to better restore the information obtained by the AI-based downsampling process of the voice transmitting device 100. Therefore, compared with Figure 3 The schematic diagram of the AI-based downsampling process shown in corresponds to Fig.10c In the AI-based upsampling process shown in , the multi-scale convolution operation in the feature extraction layer in the multi-filter scale feature extraction (low-frequency repair) process also has three convolution kernel sizes, namely 8, 16 and 32, and each scale of the multi-scale convolution operation has two convolution operations, so the speech receiving device 200 can extract speech features by utilizing multi-scale convolution operations of different sizes. For example, if the shape of the received speech signal is [320,1], since the step size is set to 1, the shape of the feature extraction layer output from the multi-filter scale feature extraction (low-frequency repair) process is 3*[320,2].

[0267] Among them, the convolution operation of each scale of the feature extraction layer in the multi-filter scale feature extraction (low-frequency restoration) process may also include one convolution operation, or include more than two convolution operations.

[0268] In addition, the high-frequency feature information recovery unit 220 can be configured to use the second feature extraction network to extract low-frequency feature information from the received speech signal and restore high-frequency feature information through the following operations: using the low-frequency feature fusion network in the second feature fusion network, the low-frequency feature information output by the low-frequency feature extraction network is fused to obtain a low-frequency repaired low-frequency time domain signal.

[0269] Specifically, if Fig.10c As shown in , the second feature fusion network can correspond to Fig.10c The low-frequency feature fusion process and the high-frequency feature fusion process in the low-frequency feature fusion network correspond to Fig.10c The low-frequency feature fusion process in . Fig.10c As shown in , the high-frequency feature information recovery unit 220 can fuse the low-frequency feature information output from the feature extraction layer in the multi-filter scale feature extraction (low-frequency repair) process by setting the step size of a single convolution operation in the feature fusion layer in the low-frequency feature fusion process to 1 to perform low-frequency repair, thereby obtaining a low-frequency time domain signal with the low-frequency part repaired.

[0270] In addition, in the speech receiving device 200, the feature fusion layer in the low-frequency feature fusion process reuses the design of the feature fusion layer in the speech transmitting device 100. In other words, the number and size of the convolution operations of the feature fusion layer in the low-frequency feature fusion process are consistent with the number and size of the convolution operations of the feature fusion layer in the speech transmitting device 100. That is to say, in the speech receiving device 200, the feature fusion layer in the low-frequency feature fusion process reuses the design of the feature fusion layer in the speech transmitting device 100. Fig.10cThe speech receiving device 200 performs convolution processing on the low-frequency feature information at different scales by using the feature fusion layer in the low-frequency feature fusion process to obtain a low-frequency repaired low-frequency time domain signal. For example, if the shape of the signal input to the feature fusion layer in the low-frequency feature fusion process is 3*[320,2], the shape of the signal output from the feature fusion layer in the low-frequency feature fusion process is [320,1].

[0271] Through the above operations, the high-frequency feature information recovery unit 200 recovers the low-frequency time domain signal from the received speech signal, and the low-frequency time domain signal repairs the low-frequency spectrum lost when the speech signal is encoded at the encoding end.

[0272] Thereafter, the high-frequency feature information restoration unit 220 needs to restore the high-frequency time domain signal from the restored low-frequency time domain signal, which is described in detail below.

[0273] Specifically, the high-frequency feature information recovery unit 200 is also configured to extract low-frequency feature information from the received speech signal and restore high-frequency feature information using a second feature extraction network through the following operations: upsampling the low-frequency repaired low-frequency time domain signal to obtain a low-frequency time domain signal with an increased sampling rate.

[0274] Specifically, refer to Fig.10c The upsampling process can use the upsampling method based on digital signal processing in the prior art to upsample the low-frequency time domain signal output from the low-frequency feature fusion process and the low-frequency repaired low-frequency signal, so that the sampling rate of the low-frequency time domain signal is improved, but the upsampled time domain signal does not add any high-frequency frequency information, such as Fig.10c As shown in , if the shape of the signal output from the feature fusion layer in the low-frequency feature fusion process is [320, 1], the shape of the signal output from the upsampling process is [640, 1].

[0275] In addition, the high-frequency feature information recovery unit 200 is also configured to extract low-frequency feature information from the received speech signal and restore high-frequency feature information using the second feature extraction network through the following operations: using the high-frequency feature extraction network in the second feature extraction network, extract low-frequency feature information containing relevant information between high-frequency features and low-frequency features from the low-frequency time domain signal with the sampling rate increased, and restore high-frequency feature information from the low-frequency feature information based on the relevant information.

[0276] Specifically, the high-frequency feature extraction network includes at least one tenth feature extraction module, such as Fig.10c As shown in , the high-frequency feature extraction network may correspond to Fig.10c In the multi-filter scale feature extraction (high frequency) process, the at least one tenth feature extraction module may correspond to Fig.10cFeature extraction layer in the multi-filter scale feature extraction (high frequency) process.

[0277] Specifically, the high-frequency feature information recovery unit 220 can be configured to use the high-frequency feature extraction network in the second feature extraction network to extract low-frequency feature information containing the relevant information and restore high-frequency feature information based on the relevant information through the following operations: using the at least one tenth feature extraction module to extract the low-frequency feature information of the received speech signal containing the relevant information between the high-frequency features and the low-frequency features, and restore the high-frequency feature information. Wherein, the at least one tenth feature extraction module extracts the low-frequency feature information of the input information respectively through at least two convolution processing parameters, and restores the high-frequency feature information. Wherein, the at least two convolution processing parameters are at least two of the following: a convolution processing parameter corresponding to the receptive field between adjacent sampling points of the speech signal, a convolution processing parameter corresponding to the receptive field of one fundamental pitch length, and a convolution processing parameter corresponding to the receptive field of at least two fundamental pitch lengths, wherein the convolution processing parameter may include a convolution kernel size corresponding to the convolution operation. Since the above has been referred to Figure 3 This has been described in detail, so the convolution processing parameters will not be repeated here.

[0278] Specifically, if Fig.10c As shown in , the high-frequency feature information recovery unit 220 can extract low-frequency feature information and restore high-frequency feature information by the following operations: by setting the step size of the multi-scale convolution operation in the feature extraction layer in the multi-filter scale feature extraction (high frequency) process to 1 to perform convolution processing on the low-frequency time domain signal with increased sampling rate. In this process, the high-frequency feature information recovery unit 220 can first obtain low-frequency feature information containing relevant information between high-frequency features and low-frequency features from the low-frequency time domain signal with increased sampling rate, and then predict high-frequency feature information from the low-frequency feature information based on the relevant information.

[0279] In addition, the feature extraction layer in the multi-filter scale feature extraction (high frequency) process reuses the design of the feature extraction layer in the voice transmission device 100. In other words, the number and size of convolution operations in the feature extraction layer in the multi-filter scale feature extraction (high frequency) process are consistent with the number and size of convolution operations in the feature extraction layer in the voice transmission device 100. Fig.10c As shown in , if the shape of the low-frequency time domain signal with increased sampling rate is [640, 1], then since the step size is set to 1, the shape of the feature extraction layer output from the multi-filter scale feature extraction (high frequency) process is 3*[640, 2].

[0280] Among them, the convolution operation of each scale of the feature extraction layer in the multi-filter scale feature extraction (high frequency) process may also include one convolution operation, or include more than two convolution operations.

[0281] In addition, the high-frequency feature information recovery unit 220 can also be configured to use the second feature extraction network to extract low-frequency feature information from the received speech signal and restore high-frequency feature information through the following operations: using the high-frequency feature fusion network in the second feature fusion network to fuse the high-frequency feature information output by the high-frequency feature extraction network to obtain a high-frequency time domain signal.

[0282] Specifically, if Fig.10c As shown in , the high-frequency feature fusion network corresponds to Fig.10c The high-frequency feature fusion process in . Fig.10c As shown in , the high-frequency feature information recovery unit 220 can obtain a high-frequency time domain signal by fusing the high-frequency feature information output from the feature extraction layer in the multi-filter scale feature extraction (high frequency) process by setting the step size of a single convolution operation in the feature fusion layer in the high-frequency feature fusion process to 1.

[0283] In addition, the feature fusion layer in the high-frequency feature fusion process reuses the design of the feature fusion layer in the voice transmitting device 100. In other words, the number and size of convolution operations of the feature fusion layer in the high-frequency feature fusion process are consistent with the number and size of convolution operations of the feature fusion layer in the voice transmitting device 100. That is, in the voice receiving device 200, the feature fusion layer in the high-frequency feature fusion process reuses the design of the feature fusion layer in the voice transmitting device 100. Fig.10c , the speech receiving device 200 obtains a high-frequency time domain signal by convolving high-frequency feature information at different scales using the feature fusion layer in the high-frequency feature fusion process. For example, if the shape of the signal input to the feature fusion layer in the high-frequency feature fusion process is 3*[640,2], the shape of the signal output from the feature fusion layer in the high-frequency feature fusion process is [640,1].

[0284] Through the above operations, the high-frequency feature information restoration unit 220 restores the high-frequency time domain signal from the restored low-frequency time domain signal.

[0285] In the above process, the speech receiving device 200 centrally predicts high-frequency information in the high-frequency prediction process, rather than predicting high and low frequency information of the entire frequency band, so it can well restore the high-frequency spectrum of the speech signal, making the high-frequency spectrum cleaner and more accurate.

[0286] The output unit 230 can be configured to output a speech signal containing the low-frequency feature information and the high-frequency feature information by performing time domain summing processing on the low-frequency time domain signal with the increased sampling rate and the high-frequency time domain signal to obtain a final speech signal, and output the speech signal.

[0287] like Fig.10c As shown in , the output unit 230 may perform time domain summation processing on the low-frequency time domain signal with increased sampling rate output by the upsampling process and the high-frequency time domain signal output by the high-frequency feature fusion process to finally obtain the speech signal to be output, for example, Fig.10c As shown in , if the shape of the low-frequency time domain signal with increased sampling rate output from the upsampling process is [640, 1], and the shape of the high-frequency time domain signal output from the high-frequency feature fusion process is [640, 1], then the shape of the speech signal to be output is still [640, 1].

[0288] The voice receiving device 200 can repair the low-frequency spectrum lost by the voice sending device 100 when encoding the voice signal through the AI-based upsampling process described above. In addition, since the voice receiving device 200 can concentrate on predicting the high-frequency area in the high-frequency prediction process when predicting high-frequency information, rather than predicting the high and low frequency information of the entire frequency band, it is able to well restore the high-frequency spectrum of the voice signal, making the high-frequency spectrum cleaner and more accurate, thereby improving language intelligibility.

[0289] The voice sending method and device and the voice receiving method and device described above replace the traditional downsampling method with an AI-based downsampling method, and replace the traditional deep learning-based BWE method with an AI-based upsampling method, so that the receiving end can restore better high-frequency information, thereby providing users with a better audio-visual experience. In addition, since the model of the AI-based downsampling method adopted by the sending end and the model of the AI-based upsampling method sampled by the receiving end can be jointly trained, the feature extraction part of the AI-based upsampling process of the receiving end can reuse the feature extraction part of the AI-based downsampling process of the sending end, and then the AI-based upsampling process of the receiving end and the AI-based downsampling process of the sending end can use the same convolution operation to learn the corresponding information of high-frequency features and low-frequency features, which is beneficial to improve performance.

[0290] In addition, the present invention also provides a computer-readable storage medium storing a program, and when the program is executed by a processor, the program implements the above-mentioned voice sending method and / or voice receiving method.

[0291] In addition, an aspect of the present invention is to provide a computer including a readable medium storing a computer program, wherein the computer program implements the above-mentioned voice sending method and / or voice receiving method when executed by the computer.

[0292] One aspect of the present invention is to provide an electronic device, which includes a memory and a processor; the memory stores a computer program; the processor is used to execute the voice sending method and / or voice receiving method when running the computer program.

[0293] The above describes various exemplary embodiments of the present invention, it should be understood that the above description is only exemplary, not exhaustive, and the present invention is not limited to the disclosed exemplary embodiments. Without departing from the scope and spirit of the present invention, many modifications and changes are obvious to those of ordinary skill in the art. Therefore, the scope of protection of the present invention should be based on the scope of the claims.

Claims

1. A method for sending a voice signal, the method comprising: Extracting low-frequency feature information from an input speech signal by using a first feature extraction network, wherein the first feature extraction network includes at least one first feature extraction module and at least one second feature extraction module arranged in a cascade manner, wherein each of the at least one first feature extraction module and the at least one second feature extraction module performs a multi-scale convolution operation based on at least two convolution processing parameters, and wherein each of the at least one first feature extraction module and the at least one second feature extraction module performs the multi-scale convolution operation using one or more neural network models obtained by machine learning using high-frequency features and low-frequency features of speech, and the one or more neural network models are learned at a high-frequency sampling rate by using convolution operations with different convolution kernel sizes covering different frequency resolutions; and The speech signal corresponding to the low-frequency feature information is sent to a receiving end.

2. The method of claim 1, wherein: The step of extracting low-frequency feature information from the input speech signal by using a first feature extraction network includes: extracting speech feature information of an input speech signal by using the at least one first feature extraction module; and The low-frequency feature information is obtained according to the speech feature information by using the at least one second feature extraction module.

3. The method of claim 2, wherein: The step of extracting speech feature information of the input speech signal includes: performing feature extraction on the input information based on the at least two convolution processing parameters respectively, and outputting the speech feature information.

4. The method of claim 3, wherein: The at least two convolution processing parameters include at least two of the following: A first convolution processing parameter corresponding to a first receptive field between adjacent sampling points of a speech signal, a second convolution processing parameter corresponding to a second receptive field of a pitch length, and a third convolution processing parameter corresponding to a third receptive field of at least two pitch lengths.

5. The method of claim 3, wherein: The at least two convolution processing parameters include a convolution kernel size corresponding to the convolution operation.

6. The method of claim 2, wherein: The step of obtaining the low-frequency feature information according to the speech feature information by using at least one second feature extraction module includes: downsampling the speech feature information at one or more scales.

7. The method of claim 1, wherein: The low-frequency feature information extracted by the first feature extraction network includes relevant information between high-frequency features and low-frequency features.

8. The method according to claim 1 or 7, further comprising: One or more low-frequency feature information output by the first feature extraction network is fused by using a first feature fusion network to obtain a speech signal corresponding to the low-frequency feature information, wherein the multi-scale convolution operation is performed by the first feature extraction network to extract each of the one or more low-frequency feature information using different parameters of the at least two convolution processing parameters.

9. A method for receiving a speech signal, the method comprising: Receiving a first voice signal sent by a transmitting end; Extracting low-frequency feature information from a first speech signal by using a second feature extraction network and restoring high-frequency feature information based on the low-frequency feature information, wherein the second feature extraction network includes at least one seventh feature extraction module and at least one eighth feature extraction module arranged in a cascade manner, wherein each of the at least one seventh feature extraction module and the at least one eighth feature extraction module performs a multi-scale convolution operation based on at least two convolution processing parameters, and wherein each of the at least one seventh feature extraction module and the at least one eighth feature extraction module performs the multi-scale convolution operation using one or more neural network models obtained by machine learning using high-frequency features and low-frequency features of speech, and the one or more neural network models are learned at a high-frequency sampling rate by using convolution operations with different convolution kernel sizes covering different frequency resolutions; and Output a second speech signal including the low-frequency feature information and the high-frequency feature information.

10. A device for sending a voice signal, comprising: Transceiver; at least one memory storing one or more instructions; At least one processor executes the one or more instructions and is configured to perform the method of any one of claims 1 to 8.

11. A device for receiving a voice signal, comprising: Transceiver; at least one memory storing one or more instructions; At least one processor executes the one or more instructions and is configured to: perform the method of claim 9. 12 . A non-transitory computer-readable recording medium having recorded thereon a computer program for executing the method according to claim 1 .

Images