A voice processing method, device, equipment and storage medium
Through deep learning technology, the prediction and reconstruction of voice frames in VoIP system is solved, which solves the shortcomings of PLC technology in burst packet loss scenarios. Combined with FEC technology, efficient voice call quality assurance is achieved, suitable for real-time communication.
Patent Information
- Application Number
- CN202010416636.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-05-15
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2040-05-15
AI Technical Summary
In existing VoIP systems, classic PLC technology cannot effectively handle continuous multi-frame voice frame loss in burst packet loss scenarios, resulting in a decline in voice call quality, and FEC technology occupies additional bandwidth, is costly, and lacks flexibility.
Combined with deep learning technology, the target speech frame is reconstructed by predicting the frequency domain characteristics and time domain parameters of historical speech frames, supporting continuous packet loss compensation, and can be used in combination with FEC technology to flexibly respond to different scenarios.
It improves voice processing capabilities, simplifies network structure, reduces processing complexity, ensures voice call quality, is suitable for real-time communication scenarios, and avoids the adverse effects of sound quality damage.
Smart Images

Figure CN111554309B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of Internet technology, specifically to the field of VoIP (Voice over Internet Protocol, voice transmission based on IP) call technology, especially to a voice processing method, a voice processing device, a voice processing equipment and a computer-readable storage medium. Background Art
[0002] The voice signal may be damaged during transmission through the VoIP system. In the prior art, a mainstream solution to the problem of voice quality damage is the classic PLC technology. The main principle is that if the receiving end does not receive the nth (n is a positive integer) voice frame, it will perform signal analysis and processing on the n-1th voice frame to compensate for the nth voice frame. However, practice has found that due to the limited signal analysis and processing capabilities, the voice processing capabilities of the classic PLC technology are limited and cannot be applied to the scenario of sudden packet loss in the existing network. Summary of the invention
[0003] The embodiments of the present application provide a speech processing method, apparatus, device and storage medium, which can make up for the deficiencies of traditional signal analysis and processing technology and improve speech processing capabilities.
[0004] On the one hand, an embodiment of the present application provides a speech processing method, including:
[0005] Determine a historical speech frame corresponding to a target speech frame to be processed;
[0006] Obtaining frequency domain features of historical speech frames and time domain parameters of historical speech frames;
[0007] Predicting a parameter set of a target speech frame according to the correlation between the frequency domain features of the historical speech frame and the time domain parameters of the historical speech frame, wherein the parameter set includes at least two parameters;
[0008] Reconstruct the target speech frame according to the parameter set.
[0009] On the one hand, an embodiment of the present application provides another speech processing method, including:
[0010] Receive voice signals transmitted via the VoIP system;
[0011] When the target speech frame in the speech signal is lost, the target speech frame is reconstructed using the above method;
[0012] A speech signal is output based on the reconstructed target speech frame.
[0013] On the one hand, an embodiment of the present application provides a speech processing device, including:
[0014] A determination unit, used to determine a historical speech frame corresponding to a target speech frame to be processed;
[0015] An acquisition unit, used to acquire frequency domain features of historical speech frames and time domain parameters of historical speech frames;
[0016] The processing unit is used to predict a parameter set of a target speech frame according to the correlation between the frequency domain features of the historical speech frame and the time domain parameters of the historical speech frame, wherein the parameter set includes at least two parameters; and to reconstruct the target speech frame according to the parameter set.
[0017] On the one hand, an embodiment of the present application provides another speech processing device, including:
[0018] A receiving unit, used for receiving a voice signal transmitted via the VoIP system;
[0019] A processing unit, used for reconstructing the target speech frame by using the above method when the target speech frame in the speech signal is lost;
[0020] The output unit is used to output a speech signal based on the reconstructed target speech frame.
[0021] On the one hand, an embodiment of the present application provides a speech processing device, the device comprising:
[0022] a processor adapted to implement one or more instructions; and,
[0023] A computer-readable storage medium stores one or more instructions, wherein the one or more instructions are suitable for being loaded by a processor and executing the speech processing method as described above.
[0024] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores one or more instructions, and the one or more instructions are suitable for being loaded by a processor and executing the speech processing method as described above.
[0025] In an embodiment of the present application, when it is necessary to reconstruct a target speech frame in a speech signal, a parameter set of the target speech frame can be obtained based on the correlation between the frequency domain features of the historical speech frame corresponding to the target speech frame and the time domain parameters of the historical speech frame, and then the reconstruction of the target speech frame can be achieved by performing inter-parameter filtering on the parameter set. This speech reconstruction and recovery process performs a comprehensive analysis and processing of the time-frequency features of the historical speech frames, and uses the correlation between the time-frequency features of the historical speech frames to reconstruct the target speech frame. The reconstruction process is relatively simple and efficient, and is more suitable for communication scenarios with high real-time requirements. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0027] Figure 1 A schematic diagram of the structure of a VoIP system provided by an exemplary embodiment of the present application is shown;
[0028] Figure 2 A schematic diagram of the structure of a speech processing system provided by an exemplary embodiment of the present application is shown;
[0029] Figure 3 A flow chart of a speech processing method provided by an exemplary embodiment of the present application is shown;
[0030] Figure 4 A flowchart of a speech processing method provided by another exemplary embodiment of the present application is shown;
[0031] Figure 5 A flowchart of a speech processing method provided by another exemplary embodiment of the present application is shown;
[0032] Figure 6 A schematic diagram of a STFT provided by an exemplary embodiment of the present application is shown;
[0033] Figure 7 A schematic diagram of the structure of a network model provided by an exemplary embodiment of the present application is shown;
[0034] Figure 8 A schematic diagram of the structure of a speech generation model based on an excitation signal provided by an exemplary embodiment of the present application is shown;
[0035] Fig. 9 A schematic diagram of the structure of a speech processing device provided by an exemplary embodiment of the present application is shown;
[0036] Fig.10 A schematic diagram of the structure of a speech processing device provided by another exemplary embodiment of the present application is shown;
[0037] Fig.11 A structural schematic diagram of a speech processing device provided by an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION
[0038] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0039] The present application embodiment relates to VoIP. VoIP is a voice call technology that uses IP to achieve voice calls and multimedia conferences, that is, to communicate via the Internet. VoIP can also be called IP phone, Internet phone, network phone, broadband phone, and broadband phone service. Figure 1 The structure diagram of a VoIP system provided by an exemplary embodiment of the present application is shown; the system includes a sending end and a receiving end, the sending end refers to a terminal that initiates a voice signal transmitted via the VoIP system; correspondingly, the receiving end refers to a terminal that receives a voice signal transmitted via VoIP; the terminal here may include but is not limited to: a mobile phone, a PC (Personal Computer), a PDA, etc. The processing flow of the voice signal in the VoIP system is roughly as follows:
[0040] On the sender side:
[0041] (1) collecting an input voice signal, which may be collected by a microphone, for example, and the voice signal is an analog signal; performing analog-to-digital conversion on the voice signal to obtain a digital signal;
[0042] (2) Encoding the digital signal to obtain multiple voice frames; here, encoding may refer to OPUS encoding. OPUS is a lossy sound coding format suitable for real-time sound transmission on the network. Its main features include: ① Supporting a sampling rate range from 8000Hz (narrowband signal) to 48000Hz (full-band signal); ② Supporting constant bit rate and variable bit rate; ③ Supporting audio bandwidth from narrowband to full-band; ④ Supporting voice and music; ⑤ Dynamically adjusting the bit rate, audio bandwidth and frame size; ⑤ Having good robustness loss rate and PLC (Packet Loss Concealment) capabilities. Based on OPUS's strong PLC capability and good VoIP sound quality, OPUS encoding is usually used in VoIP systems. The sampling rate Fs in the encoding process can be set according to actual needs. Fs can be 8000Hz (Hertz), 16000Hz, 32000Hz, 48000Hz, etc. Generally, the frame length of a speech frame is determined by the structure of an encoder used in the encoding process. The frame length of a speech frame may be, for example, 10 ms (milliseconds), 20 ms, and the like.
[0043] (3) Encapsulate multiple voice frames into one or more IP packets.
[0044] (4) Send the IP data packet to the receiving end through the network.
[0045] On the receiving side:
[0046] (5) Receive IP data packets transmitted by the network, and decapsulate the received IP data packets to obtain multiple voice frames.
[0047] (6) Decode the speech frame and restore it to digital signal.
[0048] (7) Performing digital-to-analog conversion on the digital signal, restoring it to an analog voice signal and outputting it. The output here may be, for example, played through a speaker.
[0049] The voice signal may be damaged during transmission through the VoIP system. The so-called sound quality damage refers to the phenomenon that after the normal voice signal from the sender is transmitted to the receiver, abnormal conditions such as playback jamming and unsmoothness occur on the receiving end. An important factor causing sound quality damage is network reasons. During the transmission of data packets, the receiving end cannot receive the data packets normally due to network instability or abnormalities, resulting in the loss of voice frames in the data packets, and then the receiving end cannot recover the voice signal, resulting in abnormal conditions such as jamming when outputting the voice signal. In the prior art, there are several mainstream solutions to the phenomenon of sound quality damage:
[0050] One solution involves FEC (Feedforward Error Correction) technology. FEC technology is generally deployed at the transmitting end; its main principle is: after the transmitting end packages and sends the nth (n is a positive integer) voice frame, a certain bandwidth is still allocated in the next data packet to package and send the nth voice frame again. The data packet formed by the repackaging is called a "redundant packet", and the information of the nth voice frame encapsulated in the redundant packet is called the redundant information of the nth voice frame. In order to save transmission bandwidth, the accuracy of the nth voice frame can be reduced, and the information of the low-precision version of the nth voice frame can be packaged into the redundant packet. During the voice transmission process, if the nth voice frame is lost, the receiving end can wait for the redundant packet of the nth voice frame to arrive, and reconstruct the nth voice frame according to the redundant information of the nth voice frame in the redundant packet, and restore the corresponding voice signal. FEC technology can be divided into in-band FEC and out-of-band FEC. The so-called in-band FEC refers to using the idle bytes in a voice frame to store redundant information. Out-of-band FEC refers to the storage of redundant information outside the structure of a voice frame through digital packet encapsulation technology. However, in practice, it is found that the FEC technology has the following shortcomings in solving the problem of sound quality impairment: it requires additional bandwidth to encode redundant information, and the receiving end will increase additional delay while waiting for redundant information; and different encoding mechanisms require specific FEC adaptation, which is costly and inflexible.
[0051] Another solution is the classic PLC technology, which is usually deployed at the receiving end. The main principle of the classic PLC technology is: if the receiving end does not receive the nth voice frame, it will read the n-1th voice frame, and perform signal analysis and processing on the n-1th voice frame to compensate for the nth voice frame. Compared with FEC technology, PLC technology does not require additional bandwidth. However, practice has found that PLC technology still has its shortcomings in solving the problem of sound quality damage: the signal analysis and processing capabilities are limited, and it is only applicable to the case where one voice frame is lost. However, in many cases in the existing network, there is a burst packet loss (that is, multiple consecutive frames are lost). In this case, the above-mentioned PLC-based technology is invalid.
[0052] The embodiment of the present application proposes a voice processing solution, which makes the following improvements on the above-mentioned classic PLC technology: ① Combining traditional signal analysis and processing technology with deep learning technology to improve voice processing capabilities; ② Modeling based on voice signal data, predicting the parameter set of the target voice frame by deep learning the historical voice frame, and then reconstructing the target voice frame according to the parameter set of the target voice frame. The reconstruction process is relatively simple and efficient, and is more suitable for communication scenarios with high real-time requirements; ③ The parameter set used to reconstruct the target voice frame contains two or more parameters, so that the learning target of the network model is decomposed into several parameters, each parameter corresponds to a different neural network for learning, and different neural networks can be flexibly configured and combined according to different parameter sets to form the structure of the network model. In this way, the network structure can be greatly simplified and the processing complexity can be effectively reduced; ④ Support continuous packet loss compensation, that is, in the case of loss of multiple consecutive voice frames, the reconstruction of multiple consecutive voice frames can be achieved to ensure the quality of voice calls; ⑤ Support the combined use with FEC technology, and avoid the adverse effects of sound quality damage in a relatively flexible combination.
[0053] The speech processing solution proposed in the embodiment of the present application will be described in detail below with reference to the accompanying drawings.
[0054] Figure 2 FIG. 1 shows a schematic diagram of a speech processing system provided by an exemplary embodiment of the present application; Figure 2 As shown, the improved PLC technology proposed in the embodiment of the present application is deployed on the downlink receiving end side. The reasons for this deployment are: 1) the receiving end is the last link of the system in end-to-end communication. After the reconstructed target voice frame is restored to a voice signal output (such as played through a speaker, a loudspeaker, etc.), the user can intuitively perceive the voice quality; 2) in the field of mobile communications, the communication link from the downlink air interface to the receiving end is the node most prone to quality problems. Setting a PLC mechanism at this node can achieve a more direct improvement in sound quality.
[0055] Figure 3 A flowchart of a voice processing method provided by an exemplary embodiment of the present application is shown; since the improved PLC technology is deployed at the downlink receiving end, Figure 3 The process shown is Figure 2 The receiving end shown is the execution subject; the method includes the following steps S301-S303.
[0056] S301, receiving a voice signal transmitted via a VoIP system.
[0057] The voice signal is sent from the sending end to the receiving end via the network. As can be seen from the processing flow in the aforementioned VoIP system, the voice signal received by the receiving end is a voice signal in the form of an IP data packet. The receiving end decapsulates the IP data packet to obtain a voice frame.
[0058] S302, when the target speech frame in the speech signal is lost, the target speech frame is reconstructed using the improved PLC technology proposed in the embodiment of the present application. The embodiment of the present application uses the nth speech frame to represent the target speech frame, and the speech processing method involved in the improved PLC technology will be described in detail in subsequent embodiments.
[0059] S303: output a speech signal based on the reconstructed target speech frame.
[0060] After reconstructing the target voice frame, the receiving end will decode and perform digital-to-analog conversion on the target voice frame, and finally play the voice signal through a speaker, etc., thereby realizing the restored output of the voice signal.
[0061] In one embodiment, the improved PLC technology can be used alone. In this case, when the receiving end confirms that the nth voice frame is lost, the packet loss compensation function is activated, and the nth voice frame is reconstructed through the processing flow involved in the improved PLC technology (i.e., the above step S303). In another embodiment, the improved PLC technology can also be used in combination with the FEC technology. In this case, Figure 3 The process shown may also include the following steps S304-S305:
[0062] S304, obtaining redundant information of the target speech frame.
[0063] S305, when the target speech frame in the speech signal is lost, reconstruct the target speech frame according to the redundant information of the target speech frame. If the target speech frame is not reconstructed according to the redundant information of the target speech frame, step S302 is triggered again to reconstruct the target speech frame using the improved PLC technology proposed in the embodiment of the present application.
[0064] In the scenario where the improved PLC technology is used in combination with the FEC technology, an FEC operation is performed at the sending end, that is, not only the n-th voice frame is packaged and sent, but also the redundant information of the n-th voice frame is packaged and sent; when the n-th voice frame is lost, the receiving end first relies on the redundant information of the n-th voice frame to try to reconstruct and restore the n-th voice frame. If the n-th voice frame cannot be successfully restored, the improved PLC function is activated to reconstruct the n-th voice frame through the processing flow involved in the improved PLC technology.
[0065] In an embodiment of the present application, when a target voice frame in a VoIP voice signal is lost, an improved PLC technology can be used to reconstruct the target voice frame. The reconstruction process of the improved PLC technology is relatively simple and efficient, and is more suitable for communication scenarios with high real-time requirements. In addition, it supports continuous packet loss compensation, that is, when multiple consecutive voice frames are lost, it is possible to reconstruct multiple consecutive voice frames to ensure the quality of voice calls. Moreover, the improved PLC technology can also be used in combination with the FEC technology to avoid the adverse effects of sound quality damage in a relatively flexible combination manner.
[0066] The speech processing method involved in the improved PLC technology proposed in the embodiment of the present application will be described in detail below in conjunction with the accompanying drawings.
[0067] Figure 4 A flowchart of a speech processing method provided by another exemplary embodiment of the present application is shown; the method comprises Figure 2 The method is performed by the receiving end shown in the figure; the method includes the following steps S401-S404.
[0068] S401, determining a historical speech frame corresponding to a target speech frame to be processed.
[0069] When there is a loss of voice frame in the voice signal transmitted through the VoIP system, the lost voice frame is determined as the target voice frame, and the historical voice frame refers to the voice frame that was transmitted before the target voice frame and can be successfully restored to the voice signal. In the subsequent embodiments of this application, the target voice frame is the nth (n is a positive integer) frame voice frame in the voice signal transmitted through the VoIP system; and the historical voice frame includes t (t is a positive integer) frames of voice frames from the nt frame to the n-1 frame in the voice signal transmitted through the VoIP system. The value of t can be set according to actual needs, and the embodiment of this application does not limit the value of t; for example: if you want to reduce the difficulty of calculation, the value of t can be set to be relatively small, such as t=2, that is, the two adjacent frames before the nth frame are selected as historical voice frames; if you want to obtain more accurate calculation results, the value of t can be set to be relatively large, such as t=n-1, that is, all frames before the nth frame are selected as historical voice frames.
[0070] S402, obtaining frequency domain features of historical speech frames and time domain parameters of historical speech frames.
[0071] The time domain parameter is a parameter used to represent the time domain characteristics of the time domain signal; the time domain parameter includes but is not limited to at least one of the following: a short-time correlation parameter, a long-time correlation parameter and an energy parameter. The definitions of various types of time domain parameters are detailed in the description of the subsequent embodiments. The historical speech frame is a time domain signal, and the time domain parameter of the historical speech frame is a parameter used to reflect the time domain characteristics of the historical speech frame; the time domain parameter of the historical speech frame may include but is not limited to: a short-time correlation parameter, a long-time correlation parameter and an energy parameter of the historical speech frame. In addition, to obtain the frequency domain characteristics of the historical speech frame, it is necessary to perform time-frequency conversion processing on the historical speech frame, and the time-frequency conversion processing is used to convert the historical speech frame from the time domain space to the frequency domain space, and then the frequency domain characteristics of the historical speech frame can be obtained in the frequency domain space. Here, the time-frequency conversion processing can be implemented by Fourier transform, STFT (Short-Term Fourier Transform, short-time Fourier transform) and other operations. Taking the use of STFT operation to perform time-frequency conversion processing on the historical speech frame as an example, the frequency domain characteristics of the historical speech frame may include the STFT coefficients of the historical speech frame. In one implementation, the frequency domain features of the historical speech frames further include the amplitude spectrum of the STFT coefficients of the historical speech frames to simplify the complexity of the speech processing process.
[0072] S403: predicting a parameter set of a target speech frame according to the correlation between the frequency domain features of the historical speech frames and the time domain parameters of the historical speech frames, wherein the parameter set includes at least two parameters.
[0073] Among them, the correlation between the frequency domain features of the historical speech frame and the time domain parameters of the historical speech frame means that when the parameter set of the target speech frame is predicted, the frequency domain features of the historical speech frame and the time domain parameters of the historical speech frame interact with each other and jointly affect the prediction result of the parameter set of the target speech frame. The parameter set of the target speech frame contains at least two parameters, and the parameters in the parameter set refer to the time domain parameters of the target speech frame required for reconstructing and restoring the target speech frame; referring to the relevant definition of the aforementioned time domain parameters, it can be known that the parameters in the parameter set of the target speech frame may include but are not limited to at least one of the following: short-term correlation parameters, long-term correlation parameters and energy parameters of the target speech frame. The type of the target speech frame may include but is not limited to: voiced frames and unvoiced frames; voiced frames belong to quasi-periodic signals, while unvoiced frames belong to non-periodic signals. The types of target speech frames are different, and the parameters required for their reconstruction are also different, so the parameters contained in the parameter set of the target speech frame are also different. After determining the parameters in the parameter set according to the type of the target speech frame, the parameter set of the target speech frame can be predicted by analyzing the correlation between the frequency domain features of the historical speech frame and the time domain parameters of the historical speech frame. In one embodiment, the network model can be called to predict the frequency domain features of the historical speech frame and the time domain parameters of the historical speech frame to obtain the parameter set of the target speech frame. The network model can include multiple neural networks, and the number of neural networks is determined according to the number of parameters in the parameter set. According to the parameter set of the target speech frame, the network structure of the network model can be configured accordingly. After the network structure of the network model is configured, the network model can be trained using a deep learning method to obtain an optimized network model. Reuse the optimized network model By predicting the frequency domain features of the historical speech frames and the time domain parameters of the historical speech frames, the parameter set Pa(n) of the target speech frame can be obtained.
[0074] S404: Reconstruct the target speech frame according to the parameter set.
[0075] The parameter set Pa(n) contains the predicted time domain parameters of the target speech frame, and the time domain parameters are parameters that reflect the time domain characteristics of the time domain signal. Then, the target speech frame can be reconstructed and restored using the time domain characteristics of the target speech frame represented by the predicted time domain parameters of the target speech frame. In a specific implementation, the parameters in the parameter set Pa(n) can be subjected to inter-parameter filtering to reconstruct the target speech frame.
[0076] In an embodiment of the present application, when it is necessary to reconstruct a target speech frame in a speech signal, a parameter set of the target speech frame can be obtained based on the correlation between the frequency domain features of the historical speech frame corresponding to the target speech frame and the time domain parameters of the historical speech frame, and then the reconstruction of the target speech frame can be achieved by performing inter-parameter filtering on the parameter set. This speech reconstruction and recovery process performs a comprehensive analysis and processing of the time-frequency features of the historical speech frames, and uses the correlation between the time-frequency features of the historical speech frames to reconstruct the target speech frame. The reconstruction process is relatively simple and efficient, and is more suitable for communication scenarios with high real-time requirements.
[0077] For the convenience of description, the following example scenario is used as an example for detailed description in the subsequent embodiments of this application. The example scenario includes the following information: (1) The speech signal is a broadband signal with a sampling rate of Fs = 16000 Hz; based on experience, the order of the LPC filter corresponding to the broadband signal with a sampling rate of Fs = 16000 Hz is 16; (2) The frame length of the speech frame is 20 ms, and each speech frame contains 320 samples. (3) The 320 sample points of each speech frame are decomposed into two sub-frames. The first sub-frame corresponds to the first 10 ms of the speech frame with a total of 160 sample points, and the second sub-frame corresponds to the last 10 ms of the speech frame with a total of 160 sample points. (4) Each speech frame is divided into 5 ms frames to obtain 4 5 ms sub-frames. Based on experience, the order of the LTP filter corresponding to the 5 ms sub-frame is 5. It should be noted that the above-mentioned example scenario is cited only to more clearly describe the process of the speech processing method of the embodiment of the present application, but does not constitute a limitation on the relevant technology of the embodiment of the present application. The speech processing method of the embodiment of the present application is also applicable in other scenarios. For example, in other scenarios, Fs may change accordingly, such as Fs=8000Hz, 32000Hz or 48000Hz; the speech frame may also change accordingly, for example, the frame length may be 10ms, 15ms; the decomposition method of the frame and subframe may change accordingly; for example: when decomposing the speech frame to form a frame, and when dividing the speech frame into a frame to form a subframe, both can be processed according to 5ms, that is, the frame length of the frame and the subframe are both 5ms; and so on. The speech processing process in these other scenarios can refer to the speech processing process in the example scenario of the embodiment of the present application for similar analysis.
[0078] Figure 5 A flowchart of a speech processing method provided by another exemplary embodiment of the present application is shown; the method comprises Figure 2 The method is performed by the receiving end shown in the figure; the method includes the following steps S501-S508.
[0079] S501, determining a historical speech frame corresponding to a target speech frame to be processed.
[0080] The target speech frame refers to the nth speech frame in the speech signal; the historical speech frame includes a total of t speech frames from the nt frame to the n-1th frame in the speech signal, where n and t are both positive integers. The value of t can be set according to actual needs. In this embodiment, t=5. It should be noted that the historical speech frame refers to a speech frame that is transmitted before the target speech frame and can be successfully restored to a speech signal. In one embodiment, the historical speech frame is a speech frame that is completely received by the receiving end and can be normally restored to a speech signal through decoding; in another embodiment, the historical speech frame is a speech frame that has been lost but has been successfully reconstructed through FEC technology, classic PLC technology, the improved PLC technology proposed in the embodiment of the present application, or a combination of the above various technologies. The successfully reconstructed speech frame can be normally decoded to restore the speech signal. Similarly, after the nth speech frame is successfully reconstructed by the speech processing method of the embodiment of the present application, if the n+1th speech frame is lost and needs to be reconstructed, the nth speech frame can be used as a historical speech frame for the n+1th speech frame to help the n+1th speech frame to be reconstructed. Figure 5 As shown, the historical speech frame can be represented as s_prev(n), which represents a sequence of sample points included in the nt frame to the n-1 frame speech frame. In the example shown in this embodiment, it is assumed that t=5, and s_prev(n) has a total of 1600 sample points.
[0081] S502, performing short-time Fourier transform processing on the historical speech frame to obtain frequency domain coefficients corresponding to the historical speech frame.
[0082] S503, extracting the amplitude spectrum from the frequency domain coefficients corresponding to the historical speech frame as the frequency domain feature of the historical speech frame.
[0083] In steps S502-S503, STFT can convert the historical speech frames in the time domain into frequency domain representation. Figure 6 A schematic diagram of a STFT provided by an exemplary embodiment of the present application is shown; Figure 6 In the example shown, t=5, STFT uses a 50% windowing and overlapping operation to eliminate the unevenness between frames. After STFT transformation, the frequency domain coefficients of the historical speech frame are obtained, and the frequency domain coefficients include multiple groups of STFT coefficients; Figure 6 As shown, the window function used by STFT can be a Hanning window, and the number of overlapping samples (hop-size) of the window function is 160 points; therefore, this embodiment can obtain 9 groups of STFT coefficients, each group of STFT coefficients includes 320 sample points. In one implementation, the amplitude spectrum can be directly extracted for each group of STFT coefficients, and the extracted amplitude spectrum is composed of an amplitude coefficient sequence and used as the frequency domain feature S_prev(n) of the historical speech frame.
[0084] In another implementation, considering that the STFT coefficients have a symmetrical characteristic, that is, a group of STFT coefficients can be evenly divided into two parts, a part of the STFT coefficients (such as the previous part) can be selected for each group of STFT coefficients to extract the amplitude spectrum, and the extracted amplitude spectrum is composed of an amplitude coefficient sequence and used as the frequency domain feature S_prev(n) of the historical speech frame; in the example shown in this embodiment, the first 161 sample points are selected for each group of STFT coefficients in the 9 groups of STFT coefficients, and the amplitude spectrum corresponding to each selected sample point is calculated, and finally 1449 amplitude coefficients are obtained, and the 1449 amplitude coefficients are composed of an amplitude coefficient sequence and used as the frequency domain feature S_prev(n) of the historical speech frame. In order to simplify the computational complexity, the embodiment of the present application takes the corresponding implementation method when the STFT coefficients have a symmetrical characteristic as an example for explanation.
[0085] In the embodiment of the present application, STFT uses a causal system, that is, it only performs frequency domain feature analysis based on the historical voice frames that have been obtained, and does not use future voice frames (that is, voice frames transmitted after the target voice frames) to perform frequency domain feature analysis. This can ensure real-time communication requirements, making the voice processing solution of the present application suitable for voice call scenarios with high real-time requirements.
[0086] S504, obtaining time domain parameters of historical speech frames.
[0087] As mentioned above, the time domain parameters are parameters used to represent the time domain characteristics of the time domain signal; the time domain parameters include but are not limited to at least one of the following: short-time correlation parameters, long-time correlation parameters and energy parameters. Various time domain parameters are closely related to the filtering processing of the time domain signal in the speech processing process. The embodiment of the present application involves a reconstruction filter. The definition of various time domain parameters is introduced below in conjunction with the definition of the reconstruction filter. In this embodiment, it is assumed that the i-th (i is a positive integer) frame of speech frame is any frame of speech frame in the speech signal. Taking the i-th frame of speech frame as an example, the definitions of various time domain parameters are as follows:
[0088] (1) Short-term correlation parameters.
[0089] The short-time correlation parameter is a parameter that reflects the short-time correlation characteristics of the time domain speech signal. The core of the reconstruction filter includes the LPC filter and the LTP filter. Among them, the LPC filter is responsible for processing the parameters related to the short-time correlation of the linear prediction, so the short-time correlation parameter can be considered as a parameter related to the LPC filter. First, define a p-order filter as shown in the following formula 1.1:
[0090] A p (z) = 1 + a 1 z -1 +a 2 z -2 +…+ap z -p Formula 1.1
[0091] In the above formula 1.1, p is the order of the filter. For LPC filter, a j (1≤j≤p) represents the LPC coefficient; for the LTP filter, a j (1≤j≤p) represents the LTP coefficient. z represents the speech signal. The LPC filter is implemented based on LP (Linear Prediction) analysis. The so-called LP analysis means that when LPC is used to filter the i-th speech frame, the filtering result of the i-th speech frame is obtained by convolving the first p historical speech frames of the i-th speech frame with the p-order filter shown in the above formula 1.1; this is in line with the short-term correlation characteristics of speech. According to experience, in the scenario with a sampling rate of Fs=8000Hz, the order of the LPC filter is p=10; in the scenario with a sampling rate of Fs=16000Hz, the order of the LPC filter is p=16.
[0092] In the example shown in this embodiment, the sampling rate Fs=16000 Hz, so the order of the LPC filter can be p=16; the p-order filter shown in the above formula 1.1 can be further decomposed into the following formula 1.2:
[0093]
[0094] Where P(z) = A p (z)-z -(p+1) A p (z -1 ) Formula 1.3
[0095] Q(z)=A p (z)+z -(p+1) A p (z -1 ) Formula 1.4
[0096] From a physical point of view, P(z) shown in formula 1.3 represents the periodic change law of glottis opening, Q(z) shown in formula 1.4 represents the periodic change law of glottis closing, and P(z) and Q(z) represent the periodic change law of glottis opening and closing.
[0097] The roots formed by the decomposition of the two polynomials P(z) and Q(z) appear alternately in the complex plane, so they are named LSF (Line Spectral Frequency). The LSF is represented by a series of angular frequencies w where the roots of P(z) and Q(z) are distributed on the unit circle in the complex plane. k Assume that the root of P(z) and Q(z) in the complex plane is defined as θ k, then the corresponding angular frequency is defined as follows:
[0098]
[0099] In the above formula 1.5, Re{θ k} represents θ k Real number, Im{θ k} represents θ k imaginary number.
[0100] The line spectrum frequency LSF(i) of the i-th speech frame can be calculated by the above formula 1.5. As mentioned above, the line spectrum frequency is a parameter that is strongly correlated with the short-time correlation of speech. Therefore, the short-time correlation parameter of the i-th speech frame includes the line spectrum frequency LSF(i) of the i-th speech frame. In practical applications, speech frames are usually decomposed, that is, a speech frame is decomposed into k sub-frames, and the LSF(i) of the i-th speech frame is decomposed into the line spectrum frequencies LSFk(i) of the k sub-frames; as shown in the example of this embodiment, the speech frame is divided into two sub-frames, namely the first 10ms and the second 10ms; the LSF(i) of the i-th speech frame is decomposed into the line spectrum frequency LSF1(i) of its first sub-frame and the line spectrum frequency LSF2(i) of its second sub-frame. Then, in order to further simplify the computational complexity, in one implementation, the line spectrum frequency LSF2(i) of the second sub-frame of the i-th speech frame can be obtained by the above formula 1.5; then, based on the line spectrum frequency LSF2(i-1) of the second sub-frame of the i-1-th frame and the line spectrum frequency LSF2(i) of the second sub-frame of the i-th speech frame, the line spectrum frequency LSF1(i) of the first sub-frame of the i-th speech frame can be obtained by interpolation, and the interpolation factor is expressed as α lsf (i). So further, the short-time correlation parameters of the i-th speech frame may include the line spectrum frequency LSF2(i) of the second subframe of the i-th speech frame and the interpolation factor α lsf (i).
[0101] (2) Long-term correlation parameters.
[0102] The long-term correlation parameter is a parameter that reflects the long-term correlation characteristics of the time domain speech signal. The LTP filter is responsible for processing parameters related to the long-term correlation of the pitch delay, so the long-term correlation parameter can be considered as a parameter related to the LTP filter. LTP filtering reflects the long-term correlation of speech frames (especially voiced frames), and the long-term correlation is strongly correlated with the pitch delay (Pitch Lag) of the speech frame. The pitch delay reflects the quasi-periodicity of the speech frame, that is, the pitch delay of the sample point in the i-th frame of the speech frame, which can be obtained by fixing the pitch delay of the sample point in the historical speech frame corresponding to the i-th frame of the speech frame, and then performing LTP filtering on the fixed pitch delay based on the quasi-periodicity. Therefore, the long-term correlation parameter of the i-th frame of the speech frame includes the pitch delay and LTP coefficient of each subframe of the i-th frame of the speech frame.
[0103] (3)Energy parameters.
[0104] The energy of different speech frames is also different, and the energy can be reflected by the gain value of each subframe of the speech frame. Therefore, the energy parameter of the i-th speech frame refers to the gain value of each subframe in the i-th speech frame.
[0105] Based on the above definition, in the example shown in this embodiment, t=5, so the time domain parameters of the historical speech frame are expressed as P_prev(n), and the content contained in P_prev(n) may include at least one of the following: ① The short-term correlation parameters of the historical speech frame, including the LSF coefficients of the t-frame historical speech frame; in this embodiment, the order of the LPC filter is 16, and one subframe of each frame of the historical speech frame corresponds to 16 LSF coefficients, and each frame of the historical speech frame includes two subframes, so each frame of the historical speech frame corresponds to 32 LSF coefficients, and P_prev(n) includes a total of 160 LSF coefficients corresponding to the t-frame historical speech frame. ② The long-term correlation parameters of the historical speech frame include the pitch delay and the 5th-order LTP filter coefficients of each 5ms subframe of the t-frame historical speech frame, and each subframe of each frame of the historical speech frame corresponds to a pitch delay and 5 LTP coefficients, so the long-term correlation parameters of the historical speech frame total the pitch delays of 20 subframes and 100 LTP coefficients. ③ The energy parameter of the historical speech frame includes the gain value of each 5ms subframe of the t frames of historical speech frame, a total of 20 subframe gain values.
[0106] S505, calling the network model to predict the frequency domain features of the historical speech frame and the time domain parameters of the historical speech frame to obtain a parameter set of the target speech frame. The parameter set includes at least two parameters, and the network model includes multiple neural networks, the number of which is determined according to the number of types of parameters in the parameter set.
[0107] The parameter set Pa(n) of the target speech frame includes the predicted time domain parameters of the target speech frame. Based on the relevant definitions of the aforementioned time domain parameters, in the embodiment of the present application, the parameters that may be included in the parameter set Pa(n) of the target speech frame are as follows: (1) Short-time correlation parameters of the target speech frame. If the target speech frame includes k sub-frames, the short-time correlation parameters of the target speech frame include the line spectrum frequency and interpolation factor of the kth sub-frame of the target speech frame, where k is an integer greater than 1. In the example shown in this embodiment, k=2, that is, the target speech frame includes the first sub-frame corresponding to the first 10ms and the second sub-frame corresponding to the last 10ms; so the parameter set Pa(n) may include parameter one and parameter two, parameter one refers to the line spectrum frequency LSF2(n) of the second sub-frame of the target speech frame, which includes a total of 16 LSF coefficients. Parameter two refers to the interpolation factor α of the target speech frame. lsf (n), the interpolation factor α lsf (n) may include 5 candidate values, including 0, 0.25, 0.5, 0.75, and 1.0. (2) Long-term correlation parameters of the target speech frame. The target speech frame includes m subframes, and the long-term correlation parameters of the target speech frame include the pitch delay and LTP coefficient of each subframe of the target speech frame, where m is a positive integer. In the example shown in this embodiment, m=4, so the parameter set Pa(n) may include parameter three and parameter four. Parameter three refers to the pitch delay of the 4 subframes of the target speech frame, expressed as pitch(n,0), pitch(n,1), pitch(n,2), and pitch(n,3). Parameter four refers to the LTP coefficients corresponding to the 4 subframes of the target speech frame. Assuming that the LTP filter is a 5th-order filter, each subframe corresponds to 5 LTP coefficients, so parameter four includes a total of 20 LTP coefficients. (3) Energy parameter gain(n) of the target speech frame. The energy of different speech frames is also different, and the energy can be reflected by the gain value of each subframe of the speech frame, which defines parameter five in the parameter set Pa(n), and parameter five refers to the energy parameter gain(n) of the target speech frame. In the example shown in this embodiment, the target speech frame includes 4 5ms subframes, and the energy parameter gain(n) of the target speech frame includes the gain values of the 4 5ms subframes, specifically including gain(n,0), gain(n,1), gain(n,2), gain(n,3). Gain(n) is used to amplify the signal of the target speech frame obtained by filtering and reconstruction of the reconstruction filter, so that the reconstructed target speech frame can be amplified to the energy level of the original speech signal, thereby restoring a more accurate and clearer target speech frame.
[0108] Referring to the above step S505, the embodiment of the present application predicts the parameter set Pa(n) of the nth voice frame by calling the network model. Considering the diversity of parameters, different network structures are used for different parameters, that is, the network structure of the network model is determined by the number of parameters included in the parameter set Pa(n). Specifically, the network model includes multiple neural networks, and the number of neural networks is determined according to the number of parameters included in the parameter set Pa(n). Based on the various parameters that may be included in the above parameter set Pa(n); Figure 7 FIG. 1 shows a schematic diagram of a network model provided by an exemplary embodiment of the present application; Figure 7 As shown, the network model includes a first neural network 701 and multiple second neural networks 702, and the second neural network 702 belongs to a subnetwork of the first neural network, that is, the output of the first neural network is used as the input of each second neural network 702. Each second neural network 702 is connected to the first neural network 701; the number of second neural networks 702 corresponds to a parameter in the parameter set, that is, a second neural network 702 can be used to predict a parameter in the parameter set Pa(n). It can be seen that the number of the second neural networks is determined according to the number of parameters in the parameter set. In one embodiment, the first neural network 701 includes a layer of LSTM (Long Short-Term Memory) and three layers of FC (Fully connected layer). The first neural network 701 is used to predict the virtual frequency domain feature S(n) of the target speech frame (i.e., the nth frame of the speech frame). The input of the first neural network 701 is the frequency domain feature S_prev(n) of the historical speech frame obtained in step S503, and the output is the virtual frequency domain feature S(n) of the target speech frame. In the example shown in this embodiment, S(n) is a predicted amplitude coefficient sequence of a virtual 322-dimensional STFT coefficient of the n-th speech frame. In the example shown in this embodiment, the LSTM in the first neural network 701 includes 1 hidden layer and 256 processing units. The first layer FC includes 512 processing units and an activation function. The second layer FC includes 512 processing units and an activation function. The third layer FC includes 322 processing units, which are used to output an amplitude coefficient sequence of a virtual 322-dimensional STFT coefficient of the target speech frame.
[0109] The second neural network 702 is used to predict the parameters of the target speech frame. The output of the second neural network 702 is used to reconstruct the various parameters of the target speech frame, and the input includes the virtual frequency domain features S(n) of the target speech frame output by the first neural network 701 and the time domain parameters P_prev(n) of the historical speech frame obtained in step S504. Depending on the type of parameter to be predicted, the content contained in the time domain parameters P_prev(n) of the historical speech frame input to the second neural network 702 may also be different; specifically: (1) The second neural network 702 used to predict parameter one in the parameter set of the target speech frame (i.e., the LSF coefficient of the target speech frame), whose input includes the virtual frequency domain features S(n) of the target speech frame and the time domain parameters P_prev(n) of the historical speech frame (and at this time, P_prev(n) includes the LSF coefficient of the t-frame historical speech frame). (2) The second neural network 702 used to predict parameter two in the parameter set of the target speech frame (i.e., the interpolation factor α of the target speech frame) lsf (n)) of the second neural network 702, whose input includes the virtual frequency domain feature S(n) of the target speech frame and the time domain parameter P_prev(n) of the historical speech frame (and at this time P_prev(n) contains the LSF coefficients of the historical speech frame of frame t). (3) A second neural network 702 for predicting parameter three in the parameter set of the target speech frame (i.e., the pitch delay of 4 subframes of the target speech frame), whose input includes the virtual frequency domain feature S(n) of the target speech frame and the time domain parameter P_prev(n) of the historical speech frame (and at this time P_prev(n) contains the pitch delay of a total of 20 subframes of the historical speech frame of frame t). (4) A second neural network 702 for predicting parameter four in the parameter set of the target speech frame (i.e., the 20 LTP coefficients of the target speech frame), whose input includes the virtual frequency domain feature S(n) of the target speech frame and the time domain parameter P_prev(n) of the historical speech frame (and at this time P_prev(n) contains a total of 100 LTP coefficients of the historical speech frame of frame t).
[0110] In the example shown in this embodiment, each second neural network 702 includes two layers of FC, and the last layer of FC does not include an activation function. The parameters to be predicted by each second neural unit 702 are different, and the structure of FC is also different. Among them, ① in the two-layer FC of the second neural network 702 used to predict parameter one, the first layer FC includes 512 processing units and an activation function, and the input of the first layer FC includes a virtual 322-dimensional STFT coefficient of the target speech frame, and a total of 160 LSF coefficients of t-frame historical speech frames; the second layer FC includes 16 processing units, and these 16 processing units are used to output the 16 LSF coefficients of parameter one of the target speech frame. ② in the two-layer FC of the second neural network 702 used to predict parameter two, the first layer FC includes 256 processing units and an activation function, and the input of the first layer FC includes a virtual 322-dimensional STFT coefficient of the target speech frame, and a total of 160 LSF coefficients of t-frame historical speech frames; the second layer FC includes 5 processing units, and the 5 processing units are used to output 5 candidate values of parameter two. ③ In the two-layer FC of the second neural network 702 used to predict parameter three, the first layer FC includes 256 processing units and activation functions, and the input of the first layer FC includes the virtual 322-dimensional STFT coefficients of the target speech frame, and the pitch delay of 20 subframes of t-frame historical speech frames; the second layer FC includes 4 processing units, and the 4 processing units are used to output the pitch delay of 4 subframes of parameter three. ④ In the two-layer FC of the second neural network 702 used to predict parameter four, the first layer FC includes 512 processing units and activation functions, and the input of the first layer FC includes the virtual 322-dimensional STFT coefficients of the target speech frame, and 100 LTP coefficients of t-frame historical speech frames; the second layer FC includes 20 processing units, and the 20 processing units are used to output the 20 LTP coefficients contained in parameter four.
[0111] based on Figure 7 In the network model shown, in one implementation, step S505 can be refined into the following steps s11-s12:
[0112] s11, calling the first neural network 701 to perform prediction processing on the frequency domain features S_prev(n) of the historical speech frame to obtain the virtual frequency domain features S(n) of the target speech frame.
[0113] s12, the virtual frequency domain features S(n) of the target speech frame and the time domain parameters P_prev(n) of the historical speech frames are respectively input into at least two second neural networks 702 for prediction processing to obtain the parameter set Pa(n) of the target speech frame.
[0114] See also Figure 7The network model also includes a third neural network 703, which is a parallel network with the first neural network (or the second neural network); the third neural network 703 includes a layer of LSTM and a layer of FC. Figure 7 In another embodiment of the network model shown, the time domain parameters of the historical speech frame also include energy parameters of the historical speech frame; the method further includes the following step s13:
[0115] s13, calling the third neural network to predict the energy parameters of the historical speech frames to obtain the energy parameters of the target speech frame, the energy parameters of the target speech frame belong to a parameter set Pa(n) of the target speech frame; the target speech frame includes m subframes, and the energy parameters of the target speech frame include gain values of each subframe of the target speech frame.
[0116] The energy parameters of some or all of the speech frames in the historical speech frames can be used to predict the energy parameters of the target speech frame. In this embodiment, the energy parameters of the historical speech frames are the energy parameters of the n-1th and n-2th speech frames as an example for explanation, and the energy parameter of the n-1th speech frame is expressed as gain(n-1), and the energy parameter of the n-2th speech frame is expressed as gain(n-2). In the example shown in this embodiment, m=4, that is, each speech frame contains 4 5ms subframes; then, the energy parameter gain(n-1) of the n-1th speech frame includes the gain values of the 4 5ms subframes of the n-1th speech frame, specifically including gain(n-1,0), gain(n-1,1), gain(n-1,2), gain(n-1,3); similarly, the energy parameter gain(n-2) of the n-2th speech frame includes the gain values of the 4 5ms subframes of the n-2th speech frame, specifically including gain(n-2,0), gain(n-2,1), gain(n-2,2), gain(n-2,3). Similarly, the energy parameter gain(n) of the nth speech frame includes the gain values of the 4 5mg subframes of the nth speech frame, including gain(n,0), gain(n,1), gain(n,2), gain(n,3). In the example shown in this embodiment, the LSTM in the third neural network includes 128 units; the FC layer includes 4 processing units and an activation function, wherein the 4 processing units are respectively used to output gain values of 4 subframes of the nth speech frame.
[0117] refer to Figure 7The network structure of the network model shown in the figure can be configured accordingly after the parameters in the parameter set Pa(n) are determined according to actual needs. For example, if the parameter set Pa(n) is determined to contain only parameter one, parameter two and parameter five according to the target speech frame being an unvoiced frame, then the network structure of the network model is composed of a first neural network 701, a second neural network 702 for predicting parameter one, a second neural network 703 for predicting parameter two and a third neural network 703 for predicting parameter five. For another example, if the parameter set Pa(n) is determined to contain parameters one to five at the same time according to the target speech frame being a voiced frame, then the network structure of the network model is as follows: Figure 7 After configuring the network structure of the network model, the deep learning method can be used to train the network model to obtain an optimized network model. Reuse the optimized network model The frequency domain features S_prev(n) of the historical speech frames and the time domain parameters P_prev(n) of the historical speech frames are predicted, and the energy parameters of the historical speech frames (such as gain(n-1) and gain(n-2)) can be further predicted to obtain the parameter set Pa(n) of the target speech frame.
[0118] S506: Establish a reconstruction filter according to the parameter set.
[0119] After obtaining the parameter set Pa(n) of the target speech frame, at least two parameters in the parameter set Pa(n) can be used to establish a reconstruction filter, and the subsequent process of reconstructing the target speech frame can be continued. As mentioned above, the reconstruction filter includes an LTP filter and an LPC filter. The LTP filter can be established using the long-term correlation parameters of the target speech frame (including parameter three and parameter four), and the LPC filter can be established using the short-term correlation parameters of the target speech frame. Referring to formula 1.1 above, the establishment of the filter mainly lies in determining the corresponding coefficients of the filter, and the establishment of the LTP filter lies in determining the LTP coefficients. Parameter four already includes the LTP coefficients, so the LTP filter can be established relatively simply based on parameter four.
[0120] The establishment of the LPC filter is to determine the LPC coefficient; the establishment process of the LPC coefficient is as follows:
[0121] First, parameter 1 refers to the line spectrum frequency LSF2(n) of the second subframe of the target speech frame, which contains a total of 16 LSF coefficients; parameter 2 refers to the interpolation factor α of the target speech frame lsf (n), which may include 5 candidate values, namely 0, 0.25, 0.5, 0.75, and 1.0. Then, the line spectrum frequency LSF1(n) of the first subframe of the target speech frame can be completed by interpolation, and the specific calculation formula is shown in the following formula 1.6:
[0122] LSF1(n)=(1-α LSF (n))·LSF2(n-1)+α LSF (n) LSF2(n) Formula 1.6
[0123] The above formula 1.6 indicates that the line spectral frequency LSF1(n) of the first sub-frame of the target speech frame is obtained by weighted summing the line spectral frequency LSF2(n-1) of the second sub-frame of the n-1th frame speech frame and the line spectral frequency LSF2(n) of the second sub-frame of the target speech frame, and the weight is the candidate value of the intrinsic factor.
[0124] Secondly, according to the relevant derivation of the above-mentioned formulas 1.1 to 1.5, it can be known that the LPC coefficient and the LSF coefficient are related. Combining formulas 1.1 to 1.5 can respectively obtain the 16th-order LPC coefficient of the first sub-frame 10ms before the target speech frame, that is, LPC1(n); and obtain the 16th-order LPC coefficient of the second sub-frame 10ms after the target speech frame, that is, LPC2(n).
[0125] The LPC coefficients can be determined through the above process, and the LPC filter can be established accordingly.
[0126] S507: Acquire an excitation signal of the target speech frame.
[0127] S508: Use a reconstruction filter to filter the excitation signal of the target speech frame to obtain the target speech frame.
[0128] Figure 8 The schematic diagram of the structure of a speech generation model based on an excitation signal provided by an exemplary embodiment of the present application is shown; the physical basis of the speech generation model based on the excitation signal is the human sound generation process, which can be roughly divided into two sub-processes: (1) When a person is speaking, a noise-like impact signal with a certain energy is generated in the human trachea; this impact signal corresponds to the excitation signal, and the excitation signal is a set of random signed noise-like sequences with strong fault tolerance. (2) The impact signal impacts the human vocal cords, producing a quasi-periodic opening and closing; after being amplified by the mouth, a sound is emitted; this process corresponds to a reconstruction filter, and the working principle of the reconstruction filter is to simulate this process to construct a sound. Sounds are divided into unvoiced and voiced sounds. The so-called voiced sound refers to the sound of the vocal cords vibrating when pronouncing; and the unvoiced sound refers to the sound of the vocal cords not vibrating. Taking into account the above characteristics of sound, the above human sound generation process will be further refined: (3) For quasi-periodic signals such as voiced sounds, LTP filters and LPC filters need to be used in the reconstruction process, and the excitation signal will impact the LTP filter and LPC filter respectively; (4) For non-periodic signals such as unvoiced sounds, only LPC filters need to be used in the reconstruction process, and the excitation signal will only impact the LPC filter.
[0129] Based on the above description, it can be known that the excitation signal is a set of random signed noise-like sequences, which is used as a driving source to impact (or excite) the reconstruction filter to generate a target speech frame. In step S507 of the embodiment of the present application, the excitation signal of the historical speech frame can be obtained, and the excitation signal of the target speech frame can be estimated based on the excitation signal of the historical speech frame.
[0130] In one implementation, step S507 may estimate the excitation signal of the target speech frame in a multiplexing manner, and the multiplexing manner may be as shown in the following equation 1.7:
[0131] ex(n)=ex(n-1) Formula 1.7
[0132] In the above formula 1.7, ex(n-1) represents the excitation signal of the n-1th speech frame; ex(n) represents the excitation signal of the target speech frame (ie, the nth speech frame).
[0133] In another implementation, step S507 may estimate the excitation signal of the target speech frame by an average value method, and the average value method may be expressed as shown in the following formula 1.8:
[0134]
[0135] The above formula 1.8 represents the calculation of the average value of the excitation signal of the historical speech frames from the nt frame to the n-1 frame, and obtains the excitation signal ex(n) of the target speech frame (i.e., the n-th speech frame). In formula 1.8, ex(nq)(1≤q≤t) represents the excitation signal of each speech frame from the nt frame to the n-1 frame.
[0136] In another implementation, step S507 may estimate the excitation signal of the target speech frame by weighted summation, and the weighted summation may be as shown in the following equation 1.9:
[0137]
[0138] The above formula 1.9 represents the weighted summation of the excitation signals of the historical speech frames from the nt frame to the n-1 frame, to obtain the excitation signal ex(n) of the target speech frame (i.e., the nth speech frame). In formula 1.9, ∝ q The weight corresponding to the excitation signal of each speech frame is represented. Taking t=5 as an example, a weight combination can be shown in the following Table 1:
[0139] Table 1: Weight table
[0140] project Weight <![CDATA[∝ 1 ]]> 0.40 <![CDATA[∝ 2 ]]> 0.30 <![CDATA[∝ 3 ]]> 0.15 <![CDATA[∝ 4 ]]> 0.10 <![CDATA[∝ 5 ]]> 0.05
[0141] Combination Figure 8In one implementation, if the target speech frame is a non-periodic signal such as an unvoiced frame, the reconstruction filter may only include an LPC filter, that is, only the LPC filter is needed to filter the excitation signal of the target speech frame; at this time, the parameter set Pa(n) may include the above-mentioned parameter 1, parameter 2 and parameter 5. Then, the process of generating the target speech frame in step S508 refers to the processing process of the LPC filtering stage, including:
[0142] First, parameter 1 refers to the line spectrum frequency LSF2(n) of the second subframe of the target speech frame, which contains a total of 16 LSF coefficients; parameter 2 refers to the interpolation factor α of the target speech frame lsf (n) may include five candidate values, namely 0, 0.25, 0.5, 0.75, and 1.0. Then the line spectrum frequency LSF1(n) of the first sub-frame of the target speech frame is obtained by calculation according to the above formula 1.6.
[0143] Secondly, according to the relevant derivation of the above-mentioned formulas 1.1 to 1.5, it can be known that the LPC coefficient and the LSF coefficient are related. Combining formulas 1.1 to 1.5 can respectively obtain the 16th-order LPC coefficient of the first sub-frame 10ms before the target speech frame, that is, LPC1(n); and obtain the 16th-order LPC coefficient of the second sub-frame 10ms after the target speech frame, that is, LPC2(n).
[0144] Again, under the impact of the excitation signal of the target speech frame, LPC1(n) is subjected to LPC filtering to reconstruct the first 10ms of the target speech frame, which is a total of 160 sample points, and gain(n,0) and gain(n,1) are called to amplify the first 160 sample points to obtain the first 160 sample points of the reconstructed target speech frame. Similarly, LPC2(n) is subjected to LPC filtering to reconstruct the last 10ms of the target speech frame, which is a total of 160 sample points, and gain(n,2) and gain(n,3) are called to amplify the last 160 sample points to obtain the last 160 sample points of the reconstructed target speech frame. The first 10ms and the last 10ms of the target speech frame are synthesized to obtain a complete target speech frame.
[0145] In the above LPC filtering process, the LPC filtering of the nth speech frame uses the LSF coefficients of the n-1th speech frame. That is to say, the LPC filtering of the nth speech frame needs to be implemented using the historical speech frames adjacent to the nth speech frame, which confirms the short-time correlation characteristics of the LPC filtering.
[0146] In another embodiment, if the target speech frame is a periodic signal such as a voiced frame, the reconstruction filter includes an LPC filter and an LTP filter, that is, the LTP filter and the LPC filter need to be used together to filter the excitation signal of the target speech frame. At this time, the parameter set Pa(n) may include the above-mentioned parameter 1, parameter 2, parameter 3, parameter 4 and parameter 5. Then, the process of generating the target speech frame in step S508 includes:
[0147] (I) LTP filtering stage:
[0148] First, parameter three includes the pitch delays of four subframes, namely pitch(n,0), pitch(n,1), pitch(n,2) and pitch(n,3). The pitch delay of each subframe is processed as follows: ① Compare the pitch delay of the subframe with the preset threshold. If the pitch delay of the subframe is lower than the preset threshold, the pitch delay of the subframe is set to 0, and the LTP filtering step is omitted. ② If the pitch delay of the subframe is not lower than the preset threshold, then take the historical sample point corresponding to the subframe, set the order of the LTP filter to 5, and call the 5th-order LTP filter to perform LTP filtering on the historical sample point corresponding to the subframe to obtain the LTP filtering result of the subframe. Since the LTP filter reflects the long-term correlation of the speech frame, and the long-term correlation is strongly correlated with the pitch delay, the historical sample points corresponding to the subframe in the LTP filter involved in the above step ② are selected with reference to the pitch delay of the subframe. Specifically, the subframe is used as the starting point, and the number of sample points corresponding to the value of the pitch delay is traced back as the historical sample points corresponding to the subframe. For example, if the value of the pitch delay of the subframe is 100, then the historical sample points corresponding to the subframe refer to the 100 sample points traced back from the subframe. It can be seen that setting the historical sample points corresponding to the subframe with reference to the pitch delay of the subframe actually uses the sample points contained in the historical subframe before the subframe (such as the previous 5ms subframe) to perform LTP filtering, which confirms the long-term correlation characteristics of the LTP filter.
[0149] Secondly, the LTP filtering results of each subframe are synthesized, including synthesizing the LTP filtering results of the first subframe and the LTP filtering results of the second subframe to obtain the LTP synthetic signal of the first subframe of the first 10ms of the target speech frame; synthesizing the LTP filtering results of the third subframe and the LTP filtering results of the fourth subframe to obtain the LTP synthetic signal of the second subframe of the last 10ms of the target speech frame; thus completing the processing of the LTP filtering stage.
[0150] (II) LPC filtering stage:
[0151] Referring to the processing process of the LPC filtering stage in the above implementation mode, firstly, the 16th-order LPC coefficients of the first sub-frame 10 ms before the target speech frame are obtained based on parameters one and two, i.e., LPC1(n); and the 16th-order LPC coefficients of the second sub-frame 10 ms after the target speech frame are obtained, i.e., LPC2(n).
[0152] Secondly, the LTP synthetic signal of the first subframe of the first 10ms of the target speech frame obtained in the LTP filtering stage is used together with LPC1(n) to perform LPC filtering, and the first 10ms of the target speech frame is reconstructed with a total of 160 sample points, and gain(n,0) and gain(n,1) are called to amplify the first 160 sample points to obtain the first 160 sample points of the reconstructed target speech frame. Similarly, the LTP synthetic signal of the second subframe of the last 10ms of the target speech frame obtained in the LTP filtering stage is used together with LPC2(n) to perform LPC filtering, and the last 10ms of the target speech frame is reconstructed with a total of 160 sample points, and gain(n,2) and gain(n,3) are called to amplify the last 160 sample points to obtain the last 160 sample points of the reconstructed target speech frame. The first 10ms and the last 10ms of the target speech frame are synthesized to obtain a complete target speech frame.
[0153] Through the above description of this embodiment, when the nth voice frame in the voice signal needs to be PLC, the voice processing method based on this embodiment can reconstruct the nth voice frame. If continuous packet loss occurs, for example, the n+1th voice frame, the n+2th voice frame, etc. are lost, the n+1th voice frame, the n+2th voice frame, etc. can be reconstructed and restored according to the above process to achieve continuous packet loss compensation and ensure voice call quality.
[0154] In an embodiment of the present application, when it is necessary to reconstruct a target speech frame in a speech signal, the network model can be called to predict the frequency domain features of the historical speech frame corresponding to the target speech frame and the time domain parameters of the historical speech frame to obtain a parameter set of the target speech frame, and then the reconstruction of the target speech frame can be achieved by performing inter-parameter filtering on the parameter set. This speech reconstruction and recovery process combines traditional signal analysis and processing technology with deep learning technology, which makes up for the shortcomings of traditional signal analysis and processing technology and improves speech processing capabilities. It also predicts the parameter set of the target speech frame based on deep learning of historical speech frames, and then reconstructs the target speech frame according to the parameter set of the target speech frame. The reconstruction process is relatively simple and efficient, and is more suitable for communication scenarios with high real-time requirements. In addition, the parameter set used to reconstruct the target speech frame contains two or more parameters, which decomposes the learning target of the network model into several parameters. Each parameter corresponds to a different neural network for learning. Different neural networks can be flexibly configured and combined according to different parameter sets to form the structure of the network model. In this way, the network structure can be greatly simplified, the processing complexity can be effectively reduced, and continuous packet loss compensation is supported. That is, in the case of loss of multiple consecutive speech frames, multiple consecutive speech frames can be reconstructed to ensure the quality of voice calls.
[0155] Fig. 9 The structure diagram of a speech processing device provided by an exemplary embodiment of the present application is shown; the speech processing device can be used as a computer program (including program code) running in a terminal, for example, the speech processing device can be an application program in the terminal (such as an App providing a VoIP call function); the terminal running the speech processing device can be used as Figure 1 or Figure 2 The receiving end shown; the speech processing device can be used to perform Figure 4 and Figure 5 Some or all of the steps in the method embodiment shown. Fig. 9 , the speech processing device comprises the following units:
[0156] A determination unit 901, configured to determine a historical speech frame corresponding to a target speech frame to be processed;
[0157] An acquisition unit 902 is used to acquire the frequency domain features of the historical speech frame and the time domain parameters of the historical speech frame;
[0158] The processing unit 903 is used to predict a parameter set of a target speech frame according to the correlation between the frequency domain features of the historical speech frames and the time domain parameters of the historical speech frames, wherein the parameter set includes at least two parameters; and to reconstruct the target speech frame according to the parameter set.
[0159] In one implementation, the processing unit 903 is specifically configured to:
[0160] Establishing a reconstruction filter according to the parameter set;
[0161] Obtaining an excitation signal of a target speech frame;
[0162] The reconstruction filter is used to filter the excitation signal of the target speech frame to obtain the target speech frame.
[0163] In one implementation, the processing unit 903 is specifically configured to:
[0164] Obtaining an excitation signal of a historical speech frame;
[0165] The excitation signal of the target speech frame is estimated according to the excitation signal of the historical speech frame.
[0166] In one implementation, the target voice frame refers to the nth voice frame in the voice signal transmitted via the VoIP system; the historical voice frame includes t voice frames from the nt frame to the n-1th frame in the voice signal transmitted via the VoIP system, where n and t are both positive integers.
[0167] In one implementation, the excitation signal of the historical speech frame includes the excitation signal of the n-1th speech frame; the processing unit 903 is specifically configured to determine the excitation signal of the n-1th speech frame as the excitation signal of the target speech frame.
[0168] In one implementation, the excitation signal of the historical speech frame includes the excitation signal of each speech frame from the nt frame to the n-1 frame; the processing unit 903 is specifically used to: calculate the average value of the excitation signals of the t frames of speech frames from the nt frame to the n-1 frame, to obtain the excitation signal of the target speech frame.
[0169] In one implementation, the excitation signal of the historical speech frame includes the excitation signal of each speech frame from the nt frame to the n-1 frame; the processing unit 903 is specifically used to: perform weighted summation on the excitation signals of the t frames of speech frames from the nt frame to the n-1 frame, to obtain the excitation signal of the target speech frame.
[0170] In one implementation, the processing unit 903 is specifically configured to:
[0171] The network model is called to predict the frequency domain features of the historical speech frames and the time domain parameters of the historical speech frames to obtain a parameter set of the target speech frame; the parameter set includes at least two parameters, the network model includes multiple neural networks, and the number of neural networks is determined according to the number of parameters in the parameter set.
[0172] In one implementation, the network model includes a first neural network and at least two second neural networks, the second neural network is a subnetwork of the first neural network; one second neural network corresponds to one parameter in the parameter set; the processing unit 903 is specifically used to:
[0173] Calling the first neural network to perform prediction processing on the frequency domain features of the historical speech frame to obtain the virtual frequency domain features of the target speech frame;
[0174] The virtual frequency domain features of the target speech frame and the time domain parameters of the historical speech frame are respectively input into at least two second neural networks as input information for prediction processing to obtain at least two parameters in the parameter set of the target speech frame.
[0175] In one implementation, the acquisition unit 902 is specifically configured to:
[0176] Performing short-time Fourier transform processing on the historical speech frame to obtain the frequency domain coefficients corresponding to the historical speech frame;
[0177] The amplitude spectrum is extracted from the frequency domain coefficients corresponding to the historical speech frame as the frequency domain feature of the historical speech frame.
[0178] In one implementation, if the target speech frame is an unvoiced frame, the time domain parameters of the historical speech frame include short-time correlation parameters of the historical speech frame, the parameter set includes short-time correlation parameters of the target speech frame; the reconstruction filter includes a linear predictive coding filter;
[0179] The target speech frame includes k sub-frames, and the short-time correlation parameters of the target speech frame include the line spectrum frequency and the interpolation factor of the kth sub-frame of the target speech frame, where k is an integer greater than 1.
[0180] In one implementation, if the target speech frame is a voiced frame, the time domain parameters of the historical speech frame include a short-time correlation parameter and a long-time correlation parameter of the historical speech frame, the parameter set includes the short-time correlation parameter of the target speech frame and the long-time correlation parameter of the target speech frame; the reconstruction filter includes a long-time prediction filter and a linear prediction coding filter;
[0181] The target speech frame includes k sub-frames, and the short-time correlation parameters of the target speech frame include the line spectrum frequency and the interpolation factor of the kth sub-frame of the target speech frame, where k is an integer greater than 1;
[0182] The target speech frame includes m subframes, and the long-term correlation parameters of the target speech frame include the pitch delay and the long-term prediction coefficient of each subframe of the target speech frame, and m is a positive integer.
[0183] In one implementation, the network model further includes a third neural network, and the third neural network and the first neural network are parallel networks; the time domain parameters of the historical speech frames include energy parameters of the historical speech frames; and the processing unit 903 is further configured to:
[0184] Calling a third neural network to predict the energy parameters of the historical speech frames to obtain the energy parameters of the target speech frames, where the energy parameters of the target speech frames belong to a parameter set of the target speech frames;
[0185] The target speech frame includes m subframes, and the energy parameter of the target speech frame includes the gain value of each subframe of the target speech frame.
[0186] In an embodiment of the present application, when it is necessary to reconstruct a target speech frame in a speech signal, the network model can be called to predict the frequency domain features of the historical speech frame corresponding to the target speech frame and the time domain parameters of the historical speech frame to obtain a parameter set of the target speech frame, and then the reconstruction of the target speech frame can be achieved by performing inter-parameter filtering on the parameter set. This speech reconstruction and recovery process combines traditional signal analysis and processing technology with deep learning technology, which makes up for the shortcomings of traditional signal analysis and processing technology and improves speech processing capabilities. It also predicts the parameter set of the target speech frame based on deep learning of historical speech frames, and then reconstructs the target speech frame according to the parameter set of the target speech frame. The reconstruction process is relatively simple and efficient, and is more suitable for communication scenarios with high real-time requirements. In addition, the parameter set used to reconstruct the target speech frame contains two or more parameters, which decomposes the learning target of the network model into several parameters. Each parameter corresponds to a different neural network for learning. Different neural networks can be flexibly configured and combined according to different parameter sets to form the structure of the network model. In this way, the network structure can be greatly simplified, the processing complexity can be effectively reduced, and continuous packet loss compensation is supported. That is, in the case of loss of multiple consecutive speech frames, multiple consecutive speech frames can be reconstructed to ensure the quality of voice calls.
[0187] Fig.10 A schematic diagram of the structure of a voice processing device provided by another exemplary embodiment of the present application is shown; the voice processing device can be used as a computer program (including program code) running in a terminal, for example, the voice processing device can be an application program in the terminal (such as an App that provides a VoIP call function); the terminal running the voice processing device can be used as Figure 1 or Figure 2 The receiving end shown; the speech processing device can be used to perform Figure 3 Some or all of the steps in the method embodiment shown. Fig.10 , the speech processing device comprises the following units:
[0188] The receiving unit 1001 is used to receive a voice signal transmitted via the VoIP system;
[0189] Processing unit 1002, used for when the target speech frame in the speech signal is lost, Figure 4 or Figure 5 The method shown reconstructs the target speech frame;
[0190] The output unit 1003 is configured to output a speech signal based on the reconstructed target speech frame.
[0191] In one implementation, the processing unit 1002 is further configured to:
[0192] Obtaining redundant information of a target speech frame;
[0193] When a target speech frame in a speech signal is lost, the target speech frame is reconstructed according to redundant information of the target speech frame;
[0194] If the target speech frame cannot be reconstructed based on the redundant information of the target speech frame, Figure 4 or Figure 5 The method shown reconstructs the target speech frame.
[0195] In an embodiment of the present application, when a target voice frame in a VoIP voice signal is lost, an improved PLC technology can be used to reconstruct the target voice frame. The reconstruction process of the improved PLC technology is relatively simple and efficient, and is more suitable for communication scenarios with high real-time requirements. In addition, it supports continuous packet loss compensation, that is, when multiple consecutive voice frames are lost, it is possible to reconstruct multiple consecutive voice frames to ensure the quality of voice calls. Moreover, the improved PLC technology can also be used in combination with the FEC technology to avoid the adverse effects of sound quality damage in a relatively flexible combination manner.
[0196] Fig.11 FIG. 1 shows a schematic diagram of the structure of a speech processing device provided by an exemplary embodiment of the present application. Fig.11 , the speech processing device may be Figure 1 or Figure 2At the receiving end shown, the speech processing device includes a processor 1101, an input device 1102, an output device 1103 and a computer-readable storage medium 1104. The processor 1101, the input device 1102, the output device 1103 and the computer-readable storage medium 1104 can be connected via a bus or other means. The computer-readable storage medium 1104 can be stored in the memory of the speech processing device, the computer-readable storage medium 1104 is used to store a computer program, the computer program includes program instructions, and the processor 111 is used to execute the program instructions stored in the computer-readable storage medium 1104. The processor 1101 (or CPU (Central Processing Unit)) is the computing core and control core of the speech processing device, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function.
[0197] The embodiment of the present application also provides a computer-readable storage medium (Memory), wherein the computer-readable storage medium is a memory device in a speech processing device for storing programs and data. It is understandable that the computer-readable storage medium here can include both the built-in storage medium in the speech processing device and the extended storage medium supported by the speech processing device. The computer-readable storage medium provides a storage space, which stores the operating system of the speech processing device. In addition, one or more instructions suitable for being loaded and executed by the processor 1101 are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk storage; optionally, it can also be at least one computer-readable storage medium located away from the aforementioned processor.
[0198] In one embodiment, the computer-readable storage medium stores one or more instructions; the processor 1101 loads and executes the one or more instructions stored in the computer-readable storage medium to implement Figure 4 or Figure 5 Corresponding steps of the speech processing method in the illustrated embodiment; in a specific implementation, one or more instructions in the computer-readable storage medium are loaded by the processor 1101 and execute the following steps:
[0199] Determine a historical speech frame corresponding to a target speech frame to be processed;
[0200] Obtaining frequency domain features of historical speech frames and time domain parameters of historical speech frames;
[0201] Predicting a parameter set of a target speech frame according to the correlation between the frequency domain features of the historical speech frame and the time domain parameters of the historical speech frame, wherein the parameter set includes at least two parameters;
[0202] Reconstruct the target speech frame according to the parameter set.
[0203] In one implementation, when one or more instructions in the computer-readable storage medium are loaded by the processor 1101 and the step of reconstructing the target speech frame according to the parameter set is executed, the following steps are specifically performed:
[0204] Establishing a reconstruction filter according to the parameter set;
[0205] Obtaining an excitation signal of a target speech frame;
[0206] The reconstruction filter is used to filter the excitation signal of the target speech frame to obtain the target speech frame.
[0207] In one implementation, when one or more instructions in the computer-readable storage medium are loaded by the processor 1101 and the step of obtaining the excitation signal of the target speech frame is executed, the following steps are specifically performed:
[0208] Obtaining an excitation signal of a historical speech frame;
[0209] The excitation signal of the target speech frame is estimated according to the excitation signal of the historical speech frame.
[0210] In one implementation, the target voice frame refers to the nth voice frame in the voice signal transmitted via the VoIP system; the historical voice frame includes t voice frames from the nt frame to the n-1th frame in the voice signal transmitted via the VoIP system, where n and t are both positive integers.
[0211] In one embodiment, the excitation signal of the historical speech frame includes the excitation signal of the n-1th frame speech frame; when one or more instructions in the computer-readable storage medium are loaded by the processor 1101 and execute the step of estimating the excitation signal of the target speech frame based on the excitation signal of the historical speech frame, the following steps are specifically performed: the excitation signal of the n-1th frame speech frame is determined as the excitation signal of the target speech frame.
[0212] In one embodiment, the excitation signal of the historical speech frame includes the excitation signal of each speech frame from the nt frame to the n-1 frame; when one or more instructions in the computer-readable storage medium are loaded by the processor 1101 and execute the step of estimating the excitation signal of the target speech frame based on the excitation signal of the historical speech frame, the following steps are specifically performed: the excitation signal of the t-th frame to the n-1-th frame, a total of t frames of speech frames, is averaged to obtain the excitation signal of the target speech frame.
[0213] In one implementation, the excitation signal of the historical speech frame includes the excitation signal of each speech frame from the nt frame to the n-1 frame; when one or more instructions in the computer-readable storage medium are loaded by the processor 1101 and execute the step of estimating the excitation signal of the target speech frame based on the excitation signal of the historical speech frame, the following steps are specifically performed: weighted summation of the excitation signals of the nt frame to the n-1 frame, a total of t frames of speech frames, is performed to obtain the excitation signal of the target speech frame.
[0214] In one implementation, when one or more instructions in the computer-readable storage medium are loaded by the processor 1101 and execute the step of predicting the parameter set of the target speech frame according to the correlation between the frequency domain features of the historical speech frames and the time domain parameters of the historical speech frames, the following steps are specifically performed:
[0215] The network model is called to predict the frequency domain features of the historical speech frames and the time domain parameters of the historical speech frames to obtain a parameter set of the target speech frame; the parameter set includes at least two parameters, the network model includes multiple neural networks, and the number of neural networks is determined according to the number of parameters in the parameter set.
[0216] In one implementation, the network model includes a first neural network and at least two second neural networks, the second neural network being a subnetwork of the first neural network; one second neural network corresponds to a parameter in a parameter set; one or more instructions in a computer-readable storage medium are loaded and executed by the processor 1101 to call the network model to predict and process the frequency domain features of the historical speech frames and the time domain parameters of the historical speech frames to obtain the parameter set of the target speech frame, and the following steps are specifically performed:
[0217] Calling the first neural network to perform prediction processing on the frequency domain features of the historical speech frame to obtain the virtual frequency domain features of the target speech frame;
[0218] The virtual frequency domain features of the target speech frame and the time domain parameters of the historical speech frame are respectively input into at least two second neural networks as input information for prediction processing to obtain at least two parameters in the parameter set of the target speech frame.
[0219] In one implementation, when one or more instructions in the computer-readable storage medium are loaded by the processor 1101 and the step of obtaining the frequency domain features of the historical speech frame is executed, the following steps are specifically performed:
[0220] Performing short-time Fourier transform processing on the historical speech frame to obtain the frequency domain coefficients corresponding to the historical speech frame;
[0221] The amplitude spectrum is extracted from the frequency domain coefficients corresponding to the historical speech frame as the frequency domain feature of the historical speech frame.
[0222] In one implementation, if the target speech frame is an unvoiced frame, the time domain parameters of the historical speech frame include short-time correlation parameters of the historical speech frame, the parameter set includes short-time correlation parameters of the target speech frame; the reconstruction filter includes a linear predictive coding filter;
[0223] The target speech frame includes k sub-frames, and the short-time correlation parameters of the target speech frame include the line spectrum frequency and the interpolation factor of the kth sub-frame of the target speech frame, where k is an integer greater than 1.
[0224] In one implementation, if the target speech frame is a voiced frame, the time domain parameters of the historical speech frame include a short-time correlation parameter and a long-time correlation parameter of the historical speech frame, the parameter set includes the short-time correlation parameter of the target speech frame and the long-time correlation parameter of the target speech frame; the reconstruction filter includes a long-time prediction filter and a linear prediction coding filter;
[0225] The target speech frame includes k sub-frames, and the short-time correlation parameters of the target speech frame include the line spectrum frequency and the interpolation factor of the kth sub-frame of the target speech frame, where k is an integer greater than 1;
[0226] The target speech frame includes m subframes, and the long-term correlation parameters of the target speech frame include the pitch delay and the long-term prediction coefficient of each subframe of the target speech frame, and m is a positive integer.
[0227] In one implementation, the network model further includes a third neural network, and the third neural network and the first neural network are parallel networks; the time domain parameters of the historical speech frames include energy parameters of the historical speech frames; then one or more instructions in the computer-readable storage medium are loaded by the processor 1101 and the following steps are also performed:
[0228] Calling a third neural network to predict the energy parameters of the historical speech frames to obtain the energy parameters of the target speech frames, where the energy parameters of the target speech frames belong to a parameter set of the target speech frames;
[0229] The target speech frame includes m subframes, and the energy parameter of the target speech frame includes the gain value of each subframe of the target speech frame.
[0230] In an embodiment of the present application, when it is necessary to reconstruct a target speech frame in a speech signal, the network model can be called to predict the frequency domain features of the historical speech frame corresponding to the target speech frame and the time domain parameters of the historical speech frame to obtain a parameter set of the target speech frame, and then the reconstruction of the target speech frame can be achieved by performing inter-parameter filtering on the parameter set. This speech reconstruction and recovery process combines traditional signal analysis and processing technology with deep learning technology, which makes up for the shortcomings of traditional signal analysis and processing technology and improves speech processing capabilities. It also predicts the parameter set of the target speech frame based on deep learning of historical speech frames, and then reconstructs the target speech frame according to the parameter set of the target speech frame. The reconstruction process is relatively simple and efficient, and is more suitable for communication scenarios with high real-time requirements. In addition, the parameter set used to reconstruct the target speech frame contains two or more parameters, which decomposes the learning target of the network model into several parameters. Each parameter corresponds to a different neural network for learning. Different neural networks can be flexibly configured and combined according to different parameter sets to form the structure of the network model. In this way, the network structure can be greatly simplified, the processing complexity can be effectively reduced, and continuous packet loss compensation is supported. That is, in the case of loss of multiple consecutive speech frames, multiple consecutive speech frames can be reconstructed to ensure the quality of voice calls.
[0231] In another embodiment, the processor 1101 loads and executes one or more instructions stored in a computer-readable storage medium to implement Figure 3 Corresponding steps of the speech processing method in the illustrated embodiment; in a specific implementation, one or more instructions in the computer-readable storage medium are loaded by the processor 1101 and execute the following steps:
[0232] Receive voice signals transmitted via the VoIP system;
[0233] When the target speech frame in the speech signal is lost, the Figure 4 or Figure 5 The method shown reconstructs the target speech frame;
[0234] A speech signal is output based on the reconstructed target speech frame.
[0235] In one embodiment, one or more instructions in the computer readable storage medium are loaded by the processor 1101 and further perform the following steps:
[0236] Obtaining redundant information of a target speech frame;
[0237] When a target speech frame in a speech signal is lost, the target speech frame is reconstructed according to redundant information of the target speech frame;
[0238] If the target speech frame cannot be reconstructed based on the redundant information of the target speech frame, the Figure 4 or Figure 5 The method shown reconstructs the target speech frame.
[0239] In an embodiment of the present application, when a target voice frame in a VoIP voice signal is lost, an improved PLC technology can be used to reconstruct the target voice frame. The reconstruction process of the improved PLC technology is relatively simple and efficient, and is more suitable for communication scenarios with high real-time requirements. In addition, it supports continuous packet loss compensation, that is, when multiple consecutive voice frames are lost, it is possible to reconstruct multiple consecutive voice frames to ensure the quality of voice calls. Moreover, the improved PLC technology can also be used in combination with the FEC technology to avoid the adverse effects of sound quality damage in a relatively flexible combination manner.
[0240] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.
[0241] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.
Claims
1. A speech processing method, characterized in that: include: Determine a historical speech frame corresponding to a target speech frame to be processed; Acquire frequency domain features of the historical speech frame and time domain parameters of the historical speech frame; The first neural network in the network model is called to predict the frequency domain features of the historical speech frame to obtain the virtual frequency domain features of the target speech frame; the network model includes the first neural network and at least two second neural networks, and the second neural networks are subnetworks of the first neural network; the parameter set of the target speech frame includes at least two parameters, the number of the neural networks is determined according to the number of parameters in the parameter set, and one second neural network corresponds to one parameter in the parameter set; The virtual frequency domain features of the target speech frame and the time domain parameters of the historical speech frame are respectively input into the at least two second neural networks as input information for prediction processing to obtain at least two parameters in the parameter set; The target speech frame is reconstructed according to the parameter set.
2. The method according to claim 1, characterized in that The reconstructing the target speech frame according to the parameter set comprises: establishing a reconstruction filter according to the parameter set; Obtaining an excitation signal of a target speech frame; The reconstruction filter is used to filter the excitation signal of the target speech frame to obtain the target speech frame.
3. The method according to claim 2, characterized in that The step of obtaining the excitation signal of the target speech frame comprises: Acquiring an excitation signal of the historical speech frame; The excitation signal of the target speech frame is estimated according to the excitation signal of the historical speech frame.
4. The method according to claim 3, characterized in that The target voice frame refers to the nth voice frame in the voice signal transmitted via the VoIP system; The historical voice frames include t voice frames, from the nt frame to the n-1 frame, in total, in the voice signal transmitted via the VoIP system, where n and t are both positive integers.
5. The method according to claim 4, characterized in that The excitation signal of the historical speech frame includes the excitation signal of the n-1th speech frame; and estimating the excitation signal of the target speech frame according to the excitation signal of the historical speech frame includes: The excitation signal of the n-1th speech frame is determined as the excitation signal of the target speech frame.
6. The method according to claim 4, characterized in that The excitation signal of the historical speech frame includes the excitation signal of each speech frame from the nt frame to the n-1 frame; and estimating the excitation signal of the target speech frame according to the excitation signal of the historical speech frame includes: The excitation signal of the target speech frame is obtained by calculating the average value of the excitation signals of the t-th frame to the n-1-th frame, which is t frames in total.
7. The method according to claim 4, characterized in that The excitation signal of the historical speech frame includes the excitation signal of each speech frame from the nt frame to the n-1 frame; and estimating the excitation signal of the target speech frame according to the excitation signal of the historical speech frame includes: The excitation signals of the t-th frame to the n-1-th frame (t frames in total) are weighted summed to obtain the excitation signal of the target speech frame.
8. The method according to claim 1, characterized in that The acquiring the frequency domain features of the historical speech frames comprises: Performing short-time Fourier transform processing on the historical speech frame to obtain frequency domain coefficients corresponding to the historical speech frame; The amplitude spectrum is extracted from the frequency domain coefficients corresponding to the historical speech frame as the frequency domain feature of the historical speech frame.
9. The method according to claim 2, characterized in that If the target speech frame is an unvoiced frame, the time domain parameters of the historical speech frame include short-time correlation parameters of the historical speech frame, and the parameter set includes short-time correlation parameters of the target speech frame; the reconstruction filter includes a linear prediction coding filter; The target speech frame includes k sub-frames, and the short-time correlation parameters of the target speech frame include the line spectrum frequency and interpolation factor of the kth sub-frame of the target speech frame, where k is an integer greater than 1.
10. The method according to claim 2, characterized in that If the target speech frame is a voiced frame, the time domain parameters of the historical speech frame include a short-time correlation parameter and a long-time correlation parameter of the historical speech frame, and the parameter set includes the short-time correlation parameter of the target speech frame and the long-time correlation parameter of the target speech frame; the reconstruction filter includes a long-time prediction filter and a linear prediction coding filter; The target speech frame includes k sub-frames, and the short-time correlation parameters of the target speech frame include the line spectrum frequency and the interpolation factor of the kth sub-frame of the target speech frame, where k is an integer greater than 1; The target speech frame includes m subframes, and the long-term correlation parameters of the target speech frame include the pitch delay and the long-term prediction coefficient of each subframe of the target speech frame, and m is a positive integer.
11. The method according to claim 1, characterized in that The network model further includes a third neural network, and the third neural network and the first neural network are parallel networks; the time domain parameters of the historical speech frames include energy parameters of the historical speech frames; the method further includes: Calling the third neural network to perform prediction processing on the energy parameter of the historical speech frame to obtain the energy parameter of the target speech frame, where the energy parameter of the target speech frame belongs to a parameter set of the target speech frame; The target speech frame includes m subframes, and the energy parameter of the target speech frame includes a gain value of each subframe of the target speech frame.
12. A speech processing method, characterized in that: include: Receive voice signals transmitted via the VoIP system; When a target speech frame in the speech signal is lost, reconstructing the target speech frame using the method according to any one of claims 1 to 11; The speech signal is output based on the reconstructed target speech frame.
13. The method according to claim 12, characterized in that The method further comprises: Acquiring redundant information of the target speech frame; When a target speech frame in the speech signal is lost, reconstructing the target speech frame according to redundant information of the target speech frame; If reconstruction of the target speech frame according to the redundant information of the target speech frame fails, then the method according to any one of claims 1 to 11 is triggered to reconstruct the target speech frame.
14. A speech processing device, characterized in that: include: A determination unit, used to determine a historical speech frame corresponding to a target speech frame to be processed; An acquisition unit, used to acquire the frequency domain features of the historical speech frame and the time domain parameters of the historical speech frame; A processing unit is used to call the first neural network in the network model to predict the frequency domain features of the historical speech frame to obtain the virtual frequency domain features of the target speech frame; the network model includes the first neural network and at least two second neural networks, and the second neural networks are subnetworks of the first neural network; the parameter set of the target speech frame includes at least two parameters, the number of the neural networks is determined according to the number of parameters in the parameter set, and one second neural network corresponds to one parameter in the parameter set; The processing unit is further used to input the virtual frequency domain features of the target speech frame and the time domain parameters of the historical speech frame as input information into the at least two second neural networks for prediction processing to obtain at least two parameters in the parameter set; and The processing unit is further configured to reconstruct the target speech frame according to the parameter set.
15. A speech processing device, characterized in that: include: A receiving unit, used for receiving a voice signal transmitted via the VoIP system; A processing unit, configured to reconstruct the target speech frame by using the method according to any one of claims 1 to 11 when the target speech frame in the speech signal is lost; An output unit is used to output the speech signal based on the reconstructed target speech frame.
16. A speech processing device, characterized in that: The device comprises: a processor adapted to implement one or more instructions; and, A computer-readable storage medium storing one or more instructions, wherein the one or more instructions are suitable for being loaded by the processor and executing the speech processing method as described in any one of claims 1-11.
17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more instructions, and the one or more instructions are suitable for being loaded by a processor and executing the speech processing method according to any one of claims 1-11.
