Voice communication method, device and system, storage medium and electronic device

Abstract

The present disclosure provides a voice communication method, device and system, storage medium and electronic device, which relate to the field of voice communication technology. When the method is applied to a terminal, it includes: encoding the collected original voice signal, determining the first data stream and sending it to the server; in response to the second data stream returned by the server according to the first data stream, decoding the second data stream and determining the play voice signal; the bandwidth of the first data stream and the second data stream meet the bandwidth resource constraints of narrowband communication; using a trained deep learning model to optimize and reconstruct the play voice signal and determine the optimized voice signal. On the basis of meeting the bandwidth resource constraints of narrowband communication, the sending of the original language signal and the optimization of the play voice signal are realized, thereby completing high-quality voice communication.

Description

Background Art

[0002] With the development of communication technology, especially in the field of mobile communications, users have higher and higher requirements for voice call quality. In extreme scenarios such as high-orbit satellites and underwater submarines, narrowband communication networks are the basis for achieving high-quality voice communication. The bandwidth of narrowband communication networks is usually less than 1 Kbps, which is an ultra-low bit rate. The bit rate of traditional communication networks is generally tens of Kbps or even higher, which makes it difficult to ensure high-quality voice calls at ultra-low bit rates.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0004] The present disclosure provides a voice communication method, device and system, storage medium and electronic device, which at least to some extent overcome the problem of low voice quality in narrowband communication due to related technologies.

[0005] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0006] According to one aspect of the present disclosure, a voice communication method is provided, which is applied to a terminal, and the method includes:

[0007] Encode the collected original voice signal, determine the first data stream and send it to the server;

[0008] In response to the second data stream returned by the server according to the first data stream, the second data stream is decoded to determine to play the voice signal; the bandwidth of the first data stream and the second data stream meets the bandwidth resource constraint of narrowband communication;

[0009] The played voice signal is optimized and reconstructed using a trained deep learning model to determine an optimized voice signal.

[0010] In some embodiments, encoding the collected original voice signal, determining the first data stream and sending it to the server includes:

[0011] Collect original voice signals using voice collection driver;

[0012] Performing feature encoding on the original voice signal through a coding and decoding strategy to determine a first data stream;

[0013] The first data stream is sent to the server.

[0014] In some embodiments, feature encoding the original speech signal through a coding strategy to determine the first data stream includes:

[0015] Acquiring parameters of the original speech signal;

[0016] Extracting and quantizing the parameters of the original speech signal to determine the extracted and quantized parameters;

[0017] The extracted and quantized parameters are converted into a bit stream to determine a first data stream.

[0018] In some embodiments, obtaining the parameters of the original speech signal includes:

[0019] Determining a speech signal frame of the original speech signal;

[0020] Performing fundamental frequency estimation on the speech signal frame to determine the pitch;

[0021] Performing a fast Fourier transform on the time domain signal of the speech signal frame to determine a frequency domain signal;

[0022] Performing amplitude estimation on the pitch and the frequency domain signal to determine the amplitude of the sine wave;

[0023] Performing a mute analysis on the amplitude of the sine wave to determine a mute analysis result;

[0024] The pitch, the frequency domain signal, the sine wave amplitude and the silence analysis result are determined as parameters of the original speech signal.

[0025] In some embodiments, extracting and quantizing the parameters of the original speech signal to determine the extracted and quantized parameters includes:

[0026] Sampling the pitch, the frequency domain signal, the sine wave amplitude and the silence analysis result to reduce the bit rate, and determining the extracted parameters;

[0027] The extracted parameters are quantized to convert the bits of the original speech signal into a set number, and the extracted quantized parameters are determined.

[0028] In some embodiments, the first data stream has a target communication terminal;

[0029] The server returns a second data stream according to the first data stream, including:

[0030] The server sends the first data stream to the target communication terminal, so that the target communication terminal decodes the first data stream to obtain an original voice signal, and sends a second data stream generated in response to the original voice signal to the server;

[0031] The server returns the second data stream to the source of the first data stream.

[0032] In some embodiments, the second data stream is decoded to determine the voice signal to be played, including:

[0033] quantize and restore the second data stream to determine signal frame parameters;

[0034] amplifying the signal frame parameters to determine energy amplitude;

[0035] The energy amplitude is used to perform superposition synthesis to determine the playing voice signal.

[0036] In some embodiments, the signal frame parameters include: tone, spectrum amplitude, pitch;

[0037] Amplifying the signal frame parameters to determine the energy amplitude includes:

[0038] Interpolating the tone, spectrum amplitude and pitch to resample to a target bit rate, and determining an interpolation result;

[0039] Performing energy amplitude synthesis on the interpolation results to determine the energy amplitude;

[0040] The energy amplitude is filtered to determine the filtered energy amplitude.

[0041] In some embodiments, the energy amplitude is used for superposition synthesis to determine the playing voice signal, including:

[0042] Performing phase reconstruction on the energy amplitude to determine phase information;

[0043] Performing an inverse fast Fourier transform on the phase information and the filtered energy amplitude to determine a time domain signal;

[0044] The time domain signals are superimposed and synthesized to determine the playing voice signal.

[0045] In some embodiments, the trained deep learning model includes: a data processing layer, a reversible convolution layer, and an affine coupling layer;

[0046] Optimizing and reconstructing the played voice signal using a trained deep learning model to determine an optimized voice signal includes:

[0047] Input the played voice signal into the data processing layer, use the data processing layer to process the played voice signal into vector data and input the vector data into the reversible convolution layer;

[0048] Using the reversible convolution layer to perform reversible convolution calculation on the vector data, determine the convolution data and input it into the affine coupling layer;

[0049] The convolution data is affine coupled using the affine coupling layer to determine an optimized speech signal.

[0050] In some embodiments, performing affine coupling on the convolution data using the affine coupling layer to determine an optimized speech signal includes:

[0051] Convert the convolution data into a plurality of playback voice signal segments;

[0052] Dividing each played speech signal segment equally into a first sub-segment and a second sub-segment;

[0053] Inputting the first sub-segment and the Mel-spectrogram data into a trained deep neural network to determine a learned first sub-segment; the Mel-spectrogram data is calculated based on the played speech signal segment;

[0054] Inversely mapping the learned first sub-segment and the second sub-segment to determine an affine second sub-segment;

[0055] The first sub-segment and the affine second sub-segment are coupled in series to determine an optimized speech signal.

[0056] In some embodiments, the learned first sub-segment includes: a first calculation parameter, a second calculation parameter;

[0057] The step of inversely mapping the learned first sub-segment and the second sub-segment to determine an affine second sub-segment includes:

[0058] The first calculation parameter is matrix-dot-multiplied with the second sub-segment, and then added to the second calculation parameter to determine an affine second sub-segment.

[0059] In some embodiments, the deep learning model includes: a data processing layer, a reversible convolution layer, and an affine coupling layer; the affine coupling layer includes: a deep neural network;

[0060] The training process of the deep learning model includes:

[0061] Input the acquired training voice signal into the data processing layer, and use the data processing layer to process the playback voice signal into vector data and input the vector data into the reversible convolution layer;

[0062] Using the reversible convolution layer to perform reversible convolution calculation on the vector data, determine the convolution data and input it into the affine coupling layer;

[0063] The deep neural network is trained using the convolution data in the affine coupling layer, and when the optimization objective function of the deep neural network reaches a target value, a trained deep neural network is determined;

[0064] The affine coupling layer including the trained neural network as well as the data processing layer and the reversible convolution layer are determined as a trained deep learning model.

[0065] In some embodiments, the deep neural network is trained using the convolution data in the affine coupling layer, and when the optimization objective function of the deep neural network reaches a target value, the trained deep neural network is determined, including:

[0066] Dividing the training speech signal into a plurality of training speech signal segments;

[0067] Dividing each training speech signal segment equally into a first training sub-segment and a second training sub-segment;

[0068] Inputting the first training sub-segment and the training Mel-spectrogram data into a deep neural network to determine the training first sub-segment; the training Mel-spectrogram data is calculated based on the training speech signal segment;

[0069] Inversely mapping the first training sub-segment and the second training sub-segment to determine an affine training second sub-segment;

[0070] The first training sub-segment and the affine training second sub-segment are coupled in series to determine a coupled speech signal;

[0071] Re-inputting the coupled speech signal into the deep neural network to obtain an updated coupled speech signal, and monitoring whether the optimization objective function of the deep neural network reaches a target value;

[0072] If the optimization objective function of the deep neural network does not reach the target value, then continue to loop train the deep neural network;

[0073] If the optimization objective function of the deep neural network reaches the target value, a trained deep neural network is determined.

[0074] In some embodiments, before the original voice signal is collected, the method further includes:

[0075] Sending a vocoder acquisition request to a server to obtain a vocoder list of the server, wherein the vocoder list includes multiple vocoders, and the vocoders are used to encode speech signals and decode data streams;

[0076] According to the bandwidth resource constraint of the narrowband communication, a target vocoder is selected from the vocoder list of the server.

[0077] According to another aspect of the present disclosure, a voice communication method is also provided, which is applied to a server, and the method includes:

[0078] A first data stream sent by a receiving terminal; the first data stream is determined by encoding the collected original voice signal by the terminal;

[0079] Determine a second data stream according to the first data stream, and return the second data stream to the terminal, so that the terminal determines an optimized voice signal according to the second data stream; wherein, the terminal determines the optimized voice signal according to the second data stream, including: the terminal decodes the second data stream to determine a play voice signal, the terminal optimizes and reconstructs the play voice signal using a trained deep learning model to determine an optimized voice signal; the bandwidth of the first data stream and the second data stream meets the bandwidth resource constraints of narrowband communication.

[0080] According to another aspect of the present disclosure, a voice communication method is also provided, which is applied to a voice communication system, wherein the voice communication system includes a terminal and a server, and the method includes:

[0081] The terminal encodes the collected original voice signal, determines the first data stream and sends it to the server;

[0082] The server determines a second data stream according to the first data stream, and returns the second data stream to the terminal; the bandwidth of the first data stream and the second data stream conforms to the bandwidth resource constraint of narrowband communication;

[0083] The terminal decodes the second data stream and determines to play a voice signal;

[0084] The terminal optimizes and reconstructs the played voice signal using a trained deep learning model to determine an optimized voice signal.

[0085] According to another aspect of the present disclosure, there is also provided a voice communication device, applied to a terminal, the device comprising:

[0086] The encoding module is used to encode the collected original voice signal, determine the first data stream and send it to the server;

[0087] A decoding module, configured to decode the second data stream in response to the second data stream returned by the server according to the first data stream, and determine to play a voice signal; the bandwidth of the first data stream and the second data stream conform to the bandwidth resource constraint of narrowband communication;

[0088] The optimization module is used to optimize and reconstruct the playback voice signal using a trained deep learning model to determine an optimized voice signal.

[0089] According to another aspect of the present disclosure, there is also provided a voice communication device, which is applied to a server, and the device includes:

[0090] A first data stream receiving module, used for receiving a first data stream sent by a terminal; the first data stream is determined by encoding the collected original voice signal by the terminal;

[0091] A second data stream determination module is used to determine a second data stream based on the first data stream, and return the second data stream to the terminal, so that the terminal determines an optimized voice signal based on the second data stream; wherein, the terminal determines the optimized voice signal based on the second data stream, including: the terminal decodes the second data stream to determine a play voice signal, the terminal optimizes and reconstructs the play voice signal using a trained deep learning model to determine the optimized voice signal; the bandwidth of the first data stream and the second data stream meets the bandwidth resource constraints of narrowband communication.

[0092] According to another aspect of the present disclosure, there is also provided a voice communication system, the voice communication system comprising a terminal and a server;

[0093] The terminal is used to encode the collected original voice signal, determine the first data stream and send it to the server;

[0094] The server is configured to determine a second data stream according to the first data stream, and return the second data stream to the terminal; the bandwidths of the first data stream and the second data stream comply with bandwidth resource constraints of narrowband communication;

[0095] The terminal is also used for decoding the second data stream to determine a played voice signal; and optimizing and reconstructing the played voice signal using a trained deep learning model to determine an optimized voice signal.

[0096] According to another aspect of the present disclosure, an electronic device is also provided, which includes: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned voice communication methods by executing the executable instructions.

[0097] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the voice communication method described in any one of the above is implemented.

[0098] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the computer program implements any one of the above-mentioned voice communication methods.

[0099] The voice communication method, device and system, storage medium and electronic device provided in the embodiments of the present disclosure include: encoding the collected original voice signal, determining the first data stream and sending it to the server; in response to the second data stream returned by the server according to the first data stream, decoding the second data stream to determine the play voice signal; the bandwidth of the first data stream and the second data stream meet the bandwidth resource constraints of narrowband communication; using the trained deep learning model to optimize and reconstruct the play voice signal to determine the optimized voice signal. When the terminal performs language communication based on narrowband communication, the terminal collects the original language information for encoding, determines the first data stream that meets the bandwidth resource constraints of narrowband communication and sends the first data stream to the server, so as to realize the sending of the original language information under the narrowband communication conditions with limited bandwidth resources. The terminal performs voice communication through the server, and the dialogue end returns the second data stream to the server according to the first data stream. The server returns the second data stream to the terminal. The bandwidth of the second data stream also meets the bandwidth resource constraints of narrowband communication. The terminal decodes the second data stream to determine the play voice signal, so as to realize the reception and decoding of the play voice under narrowband communication, and uses the trained deep learning model. The playback voice signal is optimized and reconstructed to obtain an optimized voice signal. Since the data stream transmitted based on narrowband communication will lose too many voice signal features after encoding and decoding, the decoded playback voice information has the defect of low voice quality. Therefore, the playback voice signal is optimized and reconstructed through the trained deep learning model. The voice quality of the playback voice signal can be improved. On the basis of meeting the bandwidth resource constraints of narrowband communication, the original language signal is sent and the playback voice signal is optimized, thereby completing high-quality voice communication. In extreme narrowband communication scenarios such as high-orbit satellites and underwater submarines, high-real-time and high-quality voice call services can be completed.

[0100] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0101] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0102] Figure 1A schematic diagram showing the system structure of a voice communication method in an embodiment of the present disclosure.

[0103] Figure 2 A schematic diagram of a voice communication method in an embodiment of the present disclosure is shown.

[0104] Figure 3 A schematic diagram of an interaction of a voice communication method in an embodiment of the present disclosure is shown.

[0105] Figure 4 A service framework diagram of a voice communication method in a mobile communication network according to an embodiment of the present disclosure is shown.

[0106] Figure 5 An interactive diagram of a call establishment process based on a vocoder in a voice communication method in an embodiment of the present disclosure is shown.

[0107] Figure 6 A schematic diagram of the process of determining a first data stream and sending it to a server in a voice communication method according to an embodiment of the present disclosure is shown.

[0108] Figure 7 A structural diagram of a hybrid vocoder of a voice communication method in an embodiment of the present disclosure is shown.

[0109] Figure 8 A schematic diagram of a process of determining a first data stream in a voice communication method according to an embodiment of the present disclosure is shown.

[0110] Fig. 9 A schematic diagram of a process for obtaining parameters of an original voice signal in a voice communication method according to an embodiment of the present disclosure is shown.

[0111] Fig.10 A schematic diagram of a process of determining extracted and quantized parameters of a voice communication method in an embodiment of the present disclosure is shown.

[0112] Fig.11 A schematic diagram of an encoder encoding process of a voice communication method in an embodiment of the present disclosure is shown.

[0113] Fig.12 A schematic diagram showing a process in which a server of a voice communication method according to an embodiment of the present disclosure returns a second data stream based on a first data stream.

[0114] Fig.13 A schematic diagram showing the interaction between a terminal of a voice communication method and a target communication terminal through a service end in an embodiment of the present disclosure.

[0115] Fig.14 A schematic diagram of a process of determining to play a voice signal in a voice communication method according to an embodiment of the present disclosure is shown.

[0116] Fig.15A schematic diagram of a process of determining energy amplitude in a voice communication method in an embodiment of the present disclosure is shown.

[0117] Fig.16 A schematic diagram showing a specific process of determining to play a voice signal in a voice communication method according to an embodiment of the present disclosure is shown.

[0118] Fig.17 A schematic diagram of a decoding process of a decoder of a voice communication method in an embodiment of the present disclosure is shown.

[0119] Fig.18 A schematic diagram of a process of determining an optimized voice signal in a voice communication method according to an embodiment of the present disclosure is shown.

[0120] Fig.19 A schematic diagram showing a specific process of determining an optimized voice signal in a voice communication method according to an embodiment of the present disclosure is shown.

[0121] Fig. 20 A schematic diagram of a trained deep learning model of a voice communication method in an embodiment of the present disclosure is shown.

[0122] Fig.21 A schematic diagram showing the training process of a deep learning model of a voice communication method in an embodiment of the present disclosure is shown.

[0123] Fig. 22 A schematic diagram of a process of determining a trained deep neural network in a voice communication method in an embodiment of the present disclosure is shown.

[0124] Fig.23 A schematic diagram of a terminal encoding and decoding service process of a voice communication method in an embodiment of the present disclosure is shown.

[0125] Fig.24 A schematic diagram showing a voice communication method in an embodiment of the present disclosure applied to a server is shown.

[0126] Fig.25 A schematic diagram showing a voice communication method applied to a voice communication system in an embodiment of the present disclosure is shown.

[0127] Fig.26 A schematic diagram of a voice communication device in an embodiment of the present disclosure is shown.

[0128] Fig. 27 A schematic diagram of a voice communication device in another embodiment of the present disclosure is shown.

[0129] Fig.28 A structural block diagram of a computer device for a voice communication method in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0130] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0131] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0132] For ease of understanding, before introducing the embodiments of the present disclosure, several terms involved in the embodiments of the present disclosure are first explained as follows:

[0133] Vocoder: A vocoder is a device or algorithm used for voice compression. Its function is to convert the original voice signal into digital code for storage, transmission or broadcasting. The vocoder reduces the bit rate of the signal by removing redundant and irrelevant information in the voice signal, thereby compressing the voice signal. The vocoder is deployed on the terminal and the server.

[0134] Narrowband communication: Narrowband communication refers to a communication method in which the frequency band of the communication signal is narrow. In this method, the bandwidth of the signal is usually limited to a relatively small range, usually less than 1 Kbps, to reduce the transmission bandwidth and channel capacity, thereby improving the efficiency and reliability of signal transmission. In wireless communications, narrowband communication is often used for low-speed, long-distance wireless transmission.

[0135] Speech reconstruction: Speech reconstruction refers to the process of restoring compressed or encoded speech signals to their original state. This process usually involves decoding and decompression techniques to restore signals close to the original speech quality from the compressed digital code.

[0136] Fundamental frequency quantization: Fundamental frequency quantization refers to the process of quantizing the fundamental frequency (i.e. the vibration frequency of the sound wave) in the speech signal. In speech signal processing, fundamental frequency quantization is one of the important steps to achieve speech compression. By quantizing the fundamental frequency information, the amount of data can be reduced and the bit rate can be lowered.

[0137] Multi-band excitation: Multi-band excitation is a speech synthesis technology that decomposes speech signals into multiple frequency bands and processes and excites each frequency band separately to produce synthesized speech. This technology can better simulate the timbre and sound quality of human voice and provide a more natural and realistic speech synthesis effect.

[0138] IMS: IP Multimedia Subsystem, is an IP multimedia system, a new form of multimedia service, which can meet the needs of terminal customers for newer and more diversified multimedia services. IMS technology is an important way to solve the integration of mobile and fixed networks and introduce differentiated services such as triple integration of voice, data and video.

[0139] AMR: Adaptive Multi-Rate, Adaptive Multi-Rate speech coding, AMR was proposed by the European Communications Standards Committee and is the most widely used voice standard in mobile communication systems. MMS also uses this format as the voice standard.

[0140] G.711: G.711 is an audio coding method developed by the International Telecommunication Union (ITU-T), also known as ITU-TG.711. G.711 uses a bandwidth of 64Kbps and can convert 14 bits into 8 bits. Currently, G.711 has two coding methods, A-law and μ-law. In March 2008, the International Telecommunication Union officially released the latest broadband voice codec standard G.711.1.

[0141] The specific implementation of the embodiment of the present disclosure is described in detail below with reference to the accompanying drawings.

[0142] Figure 1 FIG. 1 shows an exemplary application system architecture diagram to which the voice communication method in the embodiment of the present disclosure can be applied. Figure 1 As shown, the system architecture may include a terminal device 101 , a network 102 and a server 103 .

[0143] The network 102 is a medium for providing a communication link between the terminal device 101 and the server 103, and can be a wired network or a wireless network.

[0144] Optionally, the wireless network or wired network described above uses standard communication technology and / or protocol. The network is usually the Internet, but it can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a dedicated network or any combination of a virtual private network). In some embodiments, technologies and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. are used to represent data exchanged through the network. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above data communication technologies.

[0145] The terminal device 101 can be various electronic devices, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, wearable devices, augmented reality devices, virtual reality devices, etc.

[0146] Optionally, the client of the application installed in different terminal devices 101 is the same, or the client of the same type of application based on different operating systems. Based on the different terminal platforms, the specific form of the client of the application can also be different, for example, the application client can be a mobile client, a PC client, etc.

[0147] The server 103 may be a server that provides various services, such as a background management server that provides support for the device operated by the user using the terminal device 101. The background management server may analyze and process the received request and other data, and feed back the processing results to the terminal device.

[0148] Optionally, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in this application.

[0149] Those skilled in the art will know that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration, and any number of terminal devices, networks and servers may be provided according to actual needs, and the embodiments of the present disclosure do not limit this.

[0150] Under the above system architecture, a voice communication method is provided in an embodiment of the present disclosure, and the method can be executed by any electronic device with computing and processing capabilities.

[0151] In some embodiments, the voice communication method provided in the embodiments of the present disclosure can be executed by a terminal device of the above-mentioned system architecture; in other embodiments, the voice communication method provided in the embodiments of the present disclosure can be executed by a server in the above-mentioned system architecture; in other embodiments, the voice communication method provided in the embodiments of the present disclosure can be implemented by the terminal device and the server in the above-mentioned system architecture through interaction.

[0152] The bandwidth of narrowband communication networks is usually less than 1 Kbps. The G.711 (64 Kbps) and AMR (4.75Kbps ~23.85Kbps) used in traditional communication networks can hardly guarantee high-quality voice calls at ultra-low bit rates in narrowband communications. The encoding and decoding capabilities of ultra-low-speed vocoders can provide high-quality voice communications that meet bandwidth resource constraints in narrowband communications. An ultra-low-speed vocoder is a type of vocoder that can encode voice signals to an ultra-low bit rate, for example, it can be encoded to a bandwidth of less than 0.5 Kbps. However, the existing technology has not implemented the deployment of ultra-low-speed vocoders on terminals, so high-quality voice calls in narrowband communications cannot be achieved.

[0153] Figure 2 A schematic diagram of a voice communication method in an embodiment of the present disclosure is shown. Figure 2 As shown, the voice communication method provided in the embodiment of the present disclosure includes the following steps:

[0154] Step S202: Encode the collected original voice signal, determine the first data stream and send it to the server;

[0155] Step S204: in response to the second data stream returned by the server according to the first data stream, decoding the second data stream and determining to play a voice signal; the bandwidth of the first data stream and the second data stream meet the bandwidth resource constraint of narrowband communication;

[0156] Step S206: Utilize the trained deep learning model to optimize and reconstruct the played voice signal to determine an optimized voice signal.

[0157] In the embodiment, when the terminal performs language communication based on narrowband communication, the terminal collects original language information for encoding, determines a first data stream that meets the bandwidth resource constraint of the narrowband communication, and sends the first data stream to the server, thereby realizing the sending of original language information under narrowband communication conditions with limited bandwidth resources. The dialogue end of the terminal performing voice communication through the server returns a second data stream to the server according to the first data stream, and the server returns the second data stream to the terminal, and the bandwidth of the second data stream also meets the bandwidth resource constraint of the narrowband communication. The terminal decodes the second data stream, determines the playback voice signal, and realizes the reception and decoding of the playback voice under narrowband communication, using the trained deep learning The training model optimizes and reconstructs the playback voice signal to obtain the optimized voice signal. Since the data stream transmitted based on narrowband communication will lose too many voice signal features after encoding and decoding, the decoded playback voice information has the defect of low voice quality. Therefore, the playback voice signal is optimized and reconstructed by the trained deep learning model. The voice quality of the playback voice signal can be improved, thereby achieving the sending of the original language signal and the optimization of the playback voice signal on the basis of meeting the bandwidth resource constraints of narrowband communication, thereby completing high-quality voice communication, and being able to complete high-real-time and high-quality voice call services in extreme narrowband communication scenarios such as high-orbit satellites and underwater submarines.

[0158] For the encoding, decoding and optimization of speech signals, one method is to use a parametric vocoder, which selects parameters such as the fundamental frequency, amplitude and phase of the speech signal for encoding and decoding based on the experience of acoustic experts. It has the advantage of consuming less computing resources, but it has the defect of low speech quality due to the loss of too many speech signal features. One way is to use a data-driven vocoder, which is obtained by training a neural network with a large amount of voice signal data; a neural network model is used to compress, encode, decode and reconstruct the voice signal, which has the advantages of low bit rate and high voice quality, but the encoding, decoding and optimization process of the entire voice signal by a single neural network model will have the problem of increased computing resource consumption. Therefore, this embodiment constructs a hybrid vocoder, which combines a parametric vocoder and a data-driven vocoder, and comprehensively considers the advantages of low computing resource requirements of the parametric vocoder and high voice reconstruction quality of the data-driven vocoder. The hybrid vocoder of this embodiment is an ultra-low-speed vocoder, and the bit rate of the data stream (such as the first data stream) encoded by the hybrid vocoder is lower than 1 Kbps. In an implementation example, the bit rate of the encoded first data stream is 450bps=0.45 Kbps, which meets the 1 Kbps bandwidth constraint of the narrowband communication network.

[0159] By deploying a hybrid vocoder on a terminal, the original voice signal is encoded using the parameter vocoder of the hybrid vocoder to obtain a first data stream that meets the bandwidth resource constraints of narrowband communication; when a second data stream that meets the bandwidth resource constraints of narrowband communication is received, the parameter vocoder of the hybrid vocoder is used to decode the second data stream to obtain a playback voice signal. The above uses the high efficiency and low computing power consumption of the parameter vocoder of the hybrid vocoder to achieve encoding and decoding, realizes narrowband communication between the first data stream and the second data stream between the terminal and the server, and realizes real-time voice calls under narrowband communication. Further, using the data-driven vocoder of the hybrid vocoder, the playback voice signal is optimized and reconstructed through a trained deep learning model, thereby improving the voice quality of the playback voice signal, obtaining an optimized voice signal, and realizing high-quality voice call services under narrowband communication. This embodiment can be applied to the voice call service system of the existing mobile communication network application, and can be adapted to a general mobile phone terminal to complete high-quality voice call services while meeting the narrowband bandwidth constraints.

[0160] The problem of ultra-low-rate high-quality voice calls under narrowband communications is solved. The possibility of low-rate voice calls is realized through the coordinated work of parameter vocoders and data-driven vocoders, providing a solution for scenarios with limited bandwidth. The present invention can fill the gap in the deployment of hybrid vocoders on general mobile phones to achieve high-real-time, high-quality voice call services in extreme narrowband communication scenarios such as high-orbit satellites and underwater submarines.

[0161] Figure 3 A schematic diagram of a voice communication method interaction in an embodiment of the present disclosure is shown. Figure 3 As shown, the voice communication method provided by the embodiment of the present disclosure is applied to a voice communication system, the voice communication system includes a terminal 31 and a server 32, and the method includes:

[0162] Step S301: The terminal encodes the collected original voice signal to determine a first data stream;

[0163] Step S302: The terminal sends the first data stream to the server;

[0164] Step S303: the server determines the second data stream according to the first data stream, and the bandwidths of the first data stream and the second data stream meet the bandwidth resource constraints of narrowband communication;

[0165] Step S304: the server sends the second data stream to the terminal;

[0166] Step S305: The terminal decodes the second data stream and determines to play a voice signal;

[0167] Step S306: The terminal optimizes and reconstructs the played voice signal using the trained deep learning model to determine the optimized voice signal.

[0168] This embodiment is deployed in a system of mobile communication network applications and can cope with a communication environment with limited bandwidth. Figure 4 A service framework diagram of a voice communication method in a mobile communication network according to an embodiment of the present disclosure is shown. Figure 4 As shown, the service framework of the mobile communication network includes at least a terminal and an IMS network for communication on both sides, and a gateway for communicating with the terminal in the IMS network; a hybrid vocoder is configured on both the terminal and the server, and the server is deployed on the gateway of the IMS network, so that the terminal and the server have both encoding and decoding capabilities. In order to make the transmission of the first data stream and the second data stream compatible with the existing IMS network, the gateway will convert the first data stream and the second data stream into traditional encoding, such as AMR, G.711 and other encoding methods, to achieve compatibility with narrowband communications and existing IMS networks. This embodiment provides a voice processing solution for a hybrid vocoder and a system for application in a mobile communication network, which is adapted to communication environments with limited bandwidth. The terminal negotiates with the edge gateway to select a hybrid vocoder, which has encoding and decoding capabilities. The server deployed on the gateway supports transcoding to traditional voice encoding to be compatible with the IMS network.

[0169] Figure 5The interactive diagram of the call establishment process based on the vocoder of a voice communication method in the embodiment of the present disclosure is shown. Before the terminal collects the original voice signal, it is necessary to establish a call connection between the terminal and the server, including: sending a vocoder acquisition request to the server to obtain a vocoder list of the server, the vocoder list includes a variety of vocoders, and the vocoder is used to encode the voice signal and decode the data stream; according to the bandwidth resource constraint of the narrowband communication, the target vocoder is selected from the vocoder list of the server, such as Figure 5 As shown in the figure, the specific interaction process includes:

[0170] Step S501: The terminal initiates a call request to the server;

[0171] Step S502: The terminal and the server configure wireless resources according to the call;

[0172] Step S503: the terminal sends a vocoder acquisition request to the terminal;

[0173] Step S504: the server sends a vocoder list to the terminal according to the vocoder capability request; the vocoder list includes multiple vocoders with corresponding priorities supported by the server, which are used to encode voice signals and decode data streams, and the vocoder list includes hybrid vocoders;

[0174] Step S505: the terminal selects a target hybrid vocoder from the vocoder list according to the bandwidth resource constraint of the narrowband communication, and determines the vocoder selection result;

[0175] Step S506: The terminal sends the vocoder selection result to the server;

[0176] Step S507: The server establishes a call connection with the terminal according to the vocoder selection result.

[0177] In the call establishment phase, the terminal initiating the call first requests the server deployed on the gateway to obtain a list of vocoders. The list includes hybrid vocoders with corresponding priorities supported by the server deployed on the gateway. The terminal selects the optimal hybrid vocoder supported by both parties based on the acquisition results. After the selection is completed, the terminal initiates a call establishment request, determines the hybrid vocoder, and establishes a call connection. In the call establishment phase, the terminal requests the gateway to obtain a list of supported hybrid vocoders, selects the optimal hybrid vocoder to establish a call connection, and adapts to the communication environment.

[0178] Figure 6 FIG. 4 shows a schematic diagram of a process of determining a first data stream and sending it to a server in a voice communication method according to an embodiment of the present disclosure. Figure 6 As shown, the collected original voice signal is encoded, a first data stream is determined and sent to the server, including:

[0179] Step S602: using a voice collection driver to collect original voice signals;

[0180] Step S604: feature encoding the original speech signal through a coding and decoding strategy to determine a first data stream;

[0181] Step S606: Send the first data stream to the server.

[0182] Figure 7 A hybrid vocoder structure diagram of a voice communication method in an embodiment of the present disclosure is shown. Figure 7 As shown, the first data stream is obtained by encoding the original voice signal using the encoder of the hybrid vocoder, and the playback voice signal is obtained by decoding the second data stream using the decoder of the hybrid encoder. The hybrid vocoder mainly includes: a codec strategy manager, an encoder and a decoder. The codec strategy manager is used to manage the working strategies of the encoder and the decoder, and use the codec strategy to decide when to use which codec method for decoding or encoding, so as to achieve the processing and optimization of the communication effect. The encoder is used to perform hybrid encoding on the collected original voice signal and convert it into a low-speed and low-bit-rate data stream to adapt to the limited bandwidth under narrowband communication, such as encoding the original voice signal into the first data stream. The decoder is used to decode the received low-speed and low-bit-rate data stream to restore the original language signal for playback, such as decoding the second data stream to obtain the playback voice signal.

[0183] Figure 8 FIG. 4 shows a schematic diagram of a process of determining a first data stream in a voice communication method according to an embodiment of the present disclosure. Figure 8 As shown, the original speech signal is feature-encoded by a coding and decoding strategy to determine a first data stream, including:

[0184] Step S802: Obtaining parameters of the original speech signal;

[0185] Step S804: extracting and quantizing the parameters of the original speech signal to determine the extracted and quantized parameters;

[0186] Step S806: Convert the extracted quantized parameters into a bit stream to determine a first data stream.

[0187] The coding and decoding strategy encodes the features of the original speech signal, which is mainly divided into three steps. The first step is to input the collected original speech signal into the encoder through the data interface to obtain the parameters of the original speech signal. The second step is to extract and quantize the parameters of the original speech signal and determine the extracted and quantized parameters. The bit rate can be reduced by extracting and quantizing so that it can be transmitted through a narrowband communication network. The third step is to convert the extracted and quantized parameters into a bit stream and determine the first data stream. The first data stream meets the bandwidth resource constraints of narrowband communication and can be efficiently transmitted in the narrowband communication network.

[0188] Fig. 9 FIG. 1 is a schematic diagram showing a process of obtaining parameters of an original voice signal in a voice communication method according to an embodiment of the present disclosure. Fig. 9 As shown, the parameters of the original speech signal are obtained, including:

[0189] Step S902: determining a speech signal frame of an original speech signal;

[0190] Step S904: performing fundamental frequency estimation on the speech signal frame to determine the pitch;

[0191] Step S906: performing fast Fourier transform on the time domain signal of the speech signal frame to determine the frequency domain signal;

[0192] Step S908: Estimating the amplitude of the pitch and frequency domain signals to determine the amplitude of the sine wave;

[0193] Step S9010: Performing a mute analysis on the amplitude of the sine wave to determine a mute analysis result;

[0194] Step S9012: Determine the pitch, frequency domain signal, sine wave amplitude and silence analysis results as parameters of the original speech signal.

[0195] The original speech signal is usually a speech with a certain duration. In order to speed up the coding efficiency, the original speech signal needs to be divided into frames to obtain the speech signal frames of the original speech signal. In this way, multiple speech signal frames can be encoded synchronously, which reduces the time consumed by encoding and satisfies the real-time nature of voice calls in narrowband communication. The speech signal frame is input into the fundamental frequency estimator for fundamental frequency estimation to determine the pitch. Fundamental frequency estimation refers to the fundamental frequency estimation of periodic signals, which plays an important role in many applications. For example, in Chinese speech recognition, since different tones correspond to different words, pitch is a very important feature. In addition, when identifying the speaker's emotions, the tone of speech also contains significant emotional tendencies; and in digital music applications, including music transcription, humming recognition, singing software, etc., there are many fundamental frequency estimation algorithms, including the pitch estimation model in the field of deep learning developed in recent years. This embodiment uses the YIN algorithm for fundamental frequency estimation. The fast Fourier transform (FFT) is used to perform fast Fourier transform (FFT) on the time domain signal of the speech signal frame to determine the frequency domain signal. The fast Fourier transform is used to convert the time domain signal into the frequency domain signal. The pitch and frequency domain signals are input into the amplitude estimator for amplitude estimation to obtain the sine wave amplitude of each speech signal frame. The amplitude estimator is used to obtain the amplitude of the signal. The pitch and frequency domain signals are both sine waves, so the sine wave amplitude of the speech signal frame can be obtained. The sine wave amplitude is input into the silence analyzer for silence analysis to determine whether the current frame is sounded or silent, and determine the silence analysis result. Finally, the pitch, frequency domain signal, sine wave amplitude and silence analysis result are determined as the parameters of the original speech signal.

[0196] Fig.10 FIG. 4 shows a schematic diagram of a process of determining extracted quantized parameters in a voice communication method according to an embodiment of the present disclosure. Fig.10 As shown, the parameters of the original speech signal are extracted and quantized, and the extracted and quantized parameters are determined, including:

[0197] Step S1002: sampling and decimating the pitch, frequency domain signal, sine wave amplitude and silence analysis result to reduce the bit rate, and determining the extracted parameters;

[0198] Step S1004: quantize the extracted parameters so that the bits of the original speech signal are converted into a set number, and determine the extracted and quantized parameters.

[0199] The bit rate can be reduced by extracting and quantizing the parameters. The specific process includes: inputting the pitch, frequency domain signal, sine wave amplitude and silence analysis results into the extraction quantizer, sampling and extracting the pitch, frequency domain signal, sine wave amplitude and silence analysis results to reduce the bit rate, determining the extracted parameters, quantizing the extracted parameters to convert the bits of the original speech signal into a set number, and determining the extracted quantized parameters. Quantization is one of the important steps to achieve speech signal compression, which can greatly reduce the amount of data to reduce the bit rate for channel transmission through narrowband communication networks.

[0200] Fig.11 A schematic diagram of the encoding process of an encoder of a voice communication method in an embodiment of the present disclosure is shown. Fig.11 As shown, the transmitter first determines the codec strategy according to the codec strategy manager, and then encodes the original voice data from the data interface through the parameter encoding module for characteristic parameters. The encoded data stream should meet the bandwidth resource constraints of the narrowband communication network. In the parameter encoding module part, the encoding process includes but is not limited to:

[0201] Input speech signal processing, including: Input speech: Input the collected speech signal frame to the encoder through the data interface. Estimation of pitch: Input the input speech signal frame to the fundamental frequency estimator for analysis to obtain the pitch. FFT estimation: Input the input speech signal frame (time domain signal) to the fast Fourier transformer for FFT to obtain the frequency domain signal. Estimation of amplitude: Input the pitch and frequency domain signal to the amplitude estimator to obtain the sine wave amplitude of each frame. Silence analysis: Input the amplitude to the silence analyzer to determine whether the current frame is voiced or silent, and obtain the silence analysis result.

[0202] Parameter processing, including: Extraction: The silence analysis results, amplitude, frequency domain signal and pitch are input into the extraction quantizer to sample and extract the parameters to reduce the bit rate. Quantization: Quantize the above parameters and convert them into a fixed number of bits for transmission through the channel.

[0203] Generate a bit stream, including: bit stream output: convert the quantized parameters into a bit stream in an extraction quantizer to determine a first data stream.

[0204] In the encoding part of the original speech signal, the characteristic parameters of the codec supported with the lowest bit rate include but are not limited to: 18-bit compression and quantization of the parameter characteristics of the 40ms speech signal segment, including 3 bits of energy parameter characteristics, 6 bits of pitch parameter characteristics, and 9 bits of frequency domain signal envelope parameter characteristics, totaling 3+6+9=18, so the bit rate of the parameter encoding can reach 18*1000 / 40=450 bps, which meets the 1 Kbps bandwidth constraint of the narrowband communication network.

[0205] Fig.12 FIG. 4 shows a schematic diagram of a process in which a server of a voice communication method according to an embodiment of the present disclosure returns a second data stream according to a first data stream. Fig.12 As shown, the first data stream has a target communication terminal; the server returns a second data stream according to the first data stream, including:

[0206] Step S1202: the server sends the first data stream to the target communication terminal, so that the target communication terminal decodes the first data stream to obtain an original voice signal, and sends a second data stream generated in response to the original voice signal to the server;

[0207] Step S1204: The server returns the second data stream to the source of the first data stream.

[0208] Voice communication is usually a call between two different terminals through the server, and the terminal's call object is the target communication terminal. Fig.13 A schematic diagram showing the interaction between the terminal and the target communication terminal through the server is shown in FIG. Fig.13 As shown, when the terminal 31 communicates with the target communication terminal 33 through the server 32, the step S303 executed by the original server is divided into multiple steps S3031-S3033, including:

[0209] Step S3031: the server sends the MC data stream to the target communication terminal;

[0210] Step S3032: The target communication terminal decodes the first data stream to obtain an original voice signal and plays it;

[0211] Step S3033: The target communication terminal responds to the original voice signal and generates a second data stream;

[0212] Step S3033: The target communication terminal sends the second data stream to the server.

[0213] Fig.14 FIG. 4 shows a schematic diagram of a process of determining to play a voice signal in a voice communication method according to an embodiment of the present disclosure. Fig.14 As shown, the second data stream is decoded to determine the voice signal to be played, including:

[0214] Step S1402: quantize and restore the second data stream to determine signal frame parameters;

[0215] Step S1404: amplify the signal frame parameters to determine the energy amplitude;

[0216] Step S1406: Use energy amplitude to perform superposition synthesis to determine the voice signal to be played.

[0217] The hybrid vocoder decodes the second data stream through a decoder, receives the second data stream through a data receiver, inputs the second data stream into a quantization recovery device for quantization recovery, and determines the signal frame parameters from the bit stream of the second data stream, including at least: tone, spectrum amplitude, and pitch. The signal frame parameters are amplified to increase the audio information contained in the voice signal to obtain the energy amplitude, and finally the energy amplitude is used for superposition synthesis to determine the voice signal to be played.

[0218] Fig.15 FIG. 4 shows a schematic diagram of a process of determining energy amplitude in a voice communication method according to an embodiment of the present disclosure. Fig.15 As shown, the signal frame parameters are amplified to determine the energy amplitude, including:

[0219] Step S1502: interpolating the tone, spectrum amplitude and pitch to resample to the target bit rate and determining the interpolation result;

[0220] Step S1504: performing energy amplitude synthesis on the interpolation result to determine the energy amplitude;

[0221] Step S1506: Filter the energy amplitude to determine the energy amplitude after filtering.

[0222] The tone, spectrum amplitude and pitch are input to the interpolator for interpolation processing, and resampled to the target bit rate, which is usually set to be consistent with the bit rate of the original voice signal, to obtain the interpolation result; the difference processor uses the average interpolation method for interpolation processing. After interpolation processing, the tone, spectrum amplitude and pitch of the voice signal can be restored to the level before encoding. The interpolation result is input to the energy amplitude synthesizer for energy amplitude synthesis to determine the energy amplitude; the energy amplitude is used to represent the intensity information of the voice signal, and the energy amplitude is input to the post-filter for filtering processing to determine the filtered energy amplitude. Filtering can filter noise and improve call quality.

[0223] Fig.16 FIG. 4 is a schematic diagram showing a specific process of determining to play a voice signal in a voice communication method according to an embodiment of the present disclosure. Fig.16 As shown, the energy amplitude is used for superposition synthesis to determine the played voice signal, including:

[0224] Step S1602: reconstruct the phase of the energy amplitude to determine the phase information;

[0225] Step S1604: Perform inverse fast Fourier transform on the phase information and the filtered energy amplitude to determine the time domain signal;

[0226] Step S1606: The time domain signals are superimposed and synthesized to determine the playing voice signal.

[0227] The energy amplitude is input into the phase reconstructor for phase reconstruction to determine the phase information; phase reconstruction can calculate the phase information from the energy amplitude. Sound is a waveform signal, and phase information is an important parameter used to describe the sound signal. The phase information and the filtered energy amplitude are subjected to inverse fast Fourier transform to determine the time domain signal, and the time domain signal is input into the superposition synthesizer for superposition synthesis to determine the playback voice signal.

[0228] Fig.17 A schematic diagram of a decoding process of a decoder of a voice communication method in an embodiment of the present disclosure is shown. Fig.17 As shown, the decoding process of the decoder includes but is not limited to:

[0229] Bitstream processing, including: Bitstream input: The decoder receives the bitstream from the data interface. Dequantization: The bitstream is input to the quantization recovery device to recover the quantized model parameters from the bitstream, including pitch, spectrum amplitude and pitch.

[0230] Parameter processing, including: interpolation: input the restored parameters into the interpolator and resample to the original bit rate. Interpolation adopts the average value interpolation method. Restore amplitude: input the interpolation result into the energy amplitude synthesizer to obtain the energy amplitude. Post-filtering: input the energy amplitude into the post-filter for post-filtering to obtain the filtered energy amplitude.

[0231] Synthesized speech, including: Synthesized phase: input the energy amplitude into the phase reconstructor to obtain phase information. Synthesized speech: input the phase information and the filtered energy amplitude into the inverse fast Fourier transformer to obtain the time domain signal, and then input the time domain signal into the superposition synthesizer to generate the playback speech signal.

[0232] Fig.18 FIG. 4 shows a schematic diagram of a process of determining an optimized voice signal in a voice communication method according to an embodiment of the present disclosure. Fig.18 As shown, the trained deep learning model includes: data processing layer, reversible convolution layer, affine coupling layer;

[0233] Use the trained deep learning model to optimize and reconstruct the playback voice signal and determine the optimized voice signal, including:

[0234] Step S1802: input the played voice signal into the data processing layer, use the data processing layer to process the played voice signal into vector data and input it into the reversible convolution layer;

[0235] Step S1804: using the reversible convolution layer to perform reversible convolution calculation on the vector data, determining the convolution data and inputting it into the affine coupling layer;

[0236] Step S1806: Use an affine coupling layer to perform affine coupling on the convolution data to determine the optimized speech signal.

[0237] The playback voice signal is also an audio with a certain duration. When optimizing and reconstructing, it is necessary to divide the playback voice signal into multiple segments, and input the multiple segments into the data processing layer for vectorization processing to form vectors, obtain vector data, and input them into the reversible convolution layer; the reversible convolution layer uses a reversible 1×1 convolution layer to perform 1×1 reversible convolution calculation on the vector data to obtain convolution data, and input it into the affine coupling layer; the convolution data is affinely coupled using the affine coupling layer to determine the optimized voice signal, which can filter out noise from the decoded playback voice signal, improve the caller's voice tone and continuity, and improve the voice call quality under narrowband communication.

[0238] The hybrid vocoder is configured in the operating system of the terminal device, and the operating system of the terminal device also includes a voice acquisition driver and a voice playback driver. The voice acquisition driver is responsible for collecting the original voice signal waveform of the terminal user, while the voice playback driver is used to output the decoded and optimized sound waveform to the audio system of the terminal device for playback. As a core component, the hybrid vocoder coordinates and manages the encoding and decoding strategy to achieve encoding and decoding. The hybrid vocoder also has a data interface, one port of which is connected to the voice acquisition driver and the voice playback driver to receive the collected original voice signal and output the decoded playback voice signal. Another port of the data interface is connected to the media stream interface of the terminal device to obtain the second data stream returned by the server. The data interface realizes the control and diversion of internal and external data through two ports. The hybrid encoder uses the voice acquisition driver to collect the original voice signal, uses the codec strategy manager to determine the codec strategy, performs feature encoding on the original voice signal through the codec strategy, determines the first data stream, sends the first data stream to the server through the data interface, and decodes the second data stream returned by the server to obtain the playback voice signal.

[0239] Fig.19 FIG. 4 shows a schematic diagram of a specific process of determining an optimized voice signal in a voice communication method according to an embodiment of the present disclosure. Fig.19 As shown, the convolution data is affine coupled using an affine coupling layer to determine an optimized speech signal, including:

[0240] Step S1902: Convolute the data into multiple playback voice signal segments;

[0241] Step S1904: Divide each playback voice signal segment into a first sub-segment and a second sub-segment;

[0242] Step S1906: inputting the first sub-segment and the Mel-spectrogram data into the trained deep neural network to determine the learned first sub-segment; the Mel-spectrogram data is calculated according to the played speech signal segment;

[0243] Step S1908: performing inverse mapping on the learned first sub-segment and the second sub-segment to determine an affine second sub-segment;

[0244] Step S19010: Couple the first sub-segment and the affine second sub-segment in series to determine an optimized speech signal.

[0245] Fig. 20 A schematic diagram of a trained deep learning model of a voice communication method in an embodiment of the present disclosure is shown, such as Fig. 20 As shown in the figure, the optimal reconstruction of the speech signal requires the optimization of the speech signal in parts and then the serial coupling. First, the convolution data needs to be divided into multiple playback speech signal segments, and each playback speech signal segment needs to be Divide into the first sub-segment and the second sub-fragment ; Mel-spectrogram data is calculated based on the played speech signal segment. Mel-spectrogram refers to the conversion of the signal into a spectrogram during audio and speech signal processing, and the data on the spectrogram is used as the signal feature. The horizontal axis x of the spectrogram is time, and the vertical axis y is frequency. The value corresponding to (x, y) represents the amplitude of frequency y at time x. The Mel-spectrogram data is input into a trained deep neural network to determine a learned first sub-segment; the learned first sub-segment and the second sub-segment are combined. Perform inverse mapping to determine the affine second sub-segment ; The first sub-fragment and the affine second subfragment Perform series coupling to determine the optimized speech signal.

[0246] In an embodiment, the learned first sub-segment includes: a first calculation parameter and a second calculation parameter; the learned first sub-segment and the second sub-segment are reversely mapped to determine the affine second sub-segment, including: performing matrix dot multiplication of the first calculation parameter and the second sub-segment and then adding the matrix multiplication to the second calculation parameter to determine the affine second sub-segment.

[0247] Fig.21 A schematic diagram of the training process of a deep learning model of a voice communication method in an embodiment of the present disclosure is shown. Fig.21 As shown, the deep learning model includes: a data processing layer, a reversible convolution layer, and an affine coupling layer; the affine coupling layer includes: a deep neural network;

[0248] The training process of a deep learning model includes:

[0249] Step S2102: input the acquired training voice signal into the data processing layer, use the data processing layer to process the playback voice signal into vector data and input it into the reversible convolution layer;

[0250] Step S2104: using the reversible convolution layer to perform reversible convolution calculation on the vector data, determining the convolution data and inputting it into the affine coupling layer;

[0251] Step S2106: training the deep neural network using the convolution data in the affine coupling layer, and determining the trained deep neural network when the optimization objective function of the deep neural network reaches the target value;

[0252] Step S2108: Determine the affine coupling layer, data processing layer and reversible convolution layer containing the trained neural network as a trained deep learning model.

[0253] When the deep learning model is trained, the steps are similar to those of the trained deep learning model when it is used. The main training process includes: obtaining a training voice signal, inputting the obtained training voice signal into the data processing layer, using the data processing layer to process the playback voice signal into vector data and inputting it into the reversible convolution layer; the reversible convolution layer uses a reversible 1×1 convolution layer, uses the reversible convolution layer to perform 1×1 reversible convolution calculation on the vector data, determines the convolution data and inputs it into the affine coupling layer; uses the convolution data in the affine coupling layer to train the deep neural network, and when the optimization objective function of the deep neural network reaches the target value, determines the trained deep neural network; the affine coupling layer containing the trained neural network, as well as the data processing layer and the reversible convolution layer are determined as the trained deep learning model.

[0254] The training process of a deep neural network is described by the following expression:

[0255]

[0256] (1)

[0257]

[0258] in, Represents: Speech signal segment (The real speech vector data from the training data) will be split into two parts, the left one is , the one on the right is ; Represents the specific implementation of a deep neural network. This formula expresses and by Calculated Mel spectrogram After inputting into the deep neural network operation, two outputs are obtained. and ; represents the Mel spectrogram calculated from the original speech signal segment. The third formula represents Obtained by affine transformation (essentially a linear mapping), that is, obtained in the previous step and Do the matrix dot multiplication and add the other output from the previous step get; is the matrix dot product; and After concatenation, this is the output of the affine coupling layer , and get the optimized speech signal.

[0259] Fig. 22 FIG. 4 shows a schematic diagram of a process of determining a trained deep neural network in a voice communication method according to an embodiment of the present disclosure. Fig. 22 As shown, the deep neural network is trained using convolution data in the affine coupling layer. When the optimization objective function of the deep neural network reaches the target value, the trained deep neural network is determined, including:

[0260] Step S2202: Divide the training speech signal into a plurality of training speech signal segments;

[0261] Step S2204: Divide each training speech signal segment into a first training sub-segment and a second training sub-segment;

[0262] Step S2206: inputting the deep neural network into the first training sub-segment and the training Mel-spectrogram data to determine the training first sub-segment; the training Mel-spectrogram data is calculated based on the training speech signal segment;

[0263] Step S2208: reverse mapping the first training sub-segment and the second training sub-segment to determine an affine training second sub-segment;

[0264] Step S22010: Couple the first training sub-segment and the affine training second sub-segment in series to determine a coupled speech signal;

[0265] Step S22012: re-input the coupled speech signal into the deep neural network to obtain an updated coupled speech signal, and monitor whether the optimization objective function of the deep neural network reaches the target value;

[0266] Step S22014: If the optimization objective function of the deep neural network does not reach the target value, continue to cyclically train the deep neural network;

[0267] Step S22016: If the optimization objective function of the deep neural network reaches the target value, the trained deep neural network is determined.

[0268] When training a deep neural network, a training speech signal is divided into a plurality of training speech signal segments; each training speech signal segment is evenly divided into a first training sub-segment and a second training sub-segment; the first training sub-segment and the training Mel-spectrogram data are input into the deep neural network to determine the training first sub-segment; the training Mel-spectrogram data is calculated based on the training speech signal segment; the first training sub-segment and the second training sub-segment are reversely mapped to determine the affine training second sub-segment; the first training sub-segment and the affine training second sub-segment are coupled in series to determine the coupled speech signal; the coupled speech signal is re-input into the deep neural network to obtain an updated coupled speech signal, and it is monitored whether the optimization objective function of the deep neural network reaches the target value; if the optimization objective function of the deep neural network does not reach the target value, then the steps S2206-S22012 are continuously executed to train the deep neural network; if the optimization objective function of the deep neural network reaches the target value, then the trained deep neural network is determined.

[0269] The optimization objective function of the deep neural network is described by the following expression:

[0270]

[0271] (2)

[0272] Among them, the model To describe the data The distribution of refers to the parameters of the model, represents a reversible transformation, is the Jacobian determinant, To find the determinant of a matrix. is a white noise input that conforms to a high-dimensional standard Gaussian distribution, with a mean of 0 and a variance matrix that is a unit matrix; Indicates that the entire network contains Second-rate Transformation.

[0273] The above expression describes the iterative goal of the deep neural network during training, that is, the following process: training given , to train a series of functions The inverse function of That is, the parameters in the network, and make each After this series of (ordered) networks, the vectors we end up with are close to the "Gaussian distribution" .

[0274] Fig.23A schematic diagram of a terminal encoding and decoding service process of a voice communication method in an embodiment of the present disclosure is shown. Fig.23 As shown, the hybrid vocoder provided by the present invention includes four parts: parameter coding, parameter decoding, deep learning model and model training. The sending terminal first performs feature coding on the original voice data through parameter coding. The encoded first data stream should meet the bandwidth resource constraints of the narrowband communication network. Secondly, the encoded first data stream is transmitted through the narrowband communication network. When the receiving terminal receives the first data stream, it decodes it through parameter decoding, and then optimizes and restores the decoded voice signal through the deep learning model to restore the optimized voice signal with minimum distortion. The model training module inputs the original voice data and the feature coding output by the parameter coding module in an offline (bypass) manner, iteratively trains its own neural network model, and pushes the trained optimization model to the deep learning model in an online manner, which is deployed in the terminal for optimized reconstruction of the voice signal. In the encoding and decoding process, first, the terminal as the transmitting end collects the sound waveform. Then, the sound waveform is efficiently converted into a low-rate code stream through the parameter encoder for hybrid coding. Subsequently, the data is packaged according to the voice stream format for transmission in communication. During the receiving process, the terminal receives the low-rate code stream encoded by the gateway. First, the low-rate code stream is decoded by the parameter decoder to obtain the original voice signal, and then input into the deep learning model for optimization, and the sound waveform is obtained through intelligent processing. Finally, the receiving end plays the decoded and optimized sound waveform to complete the entire encoding and decoding process.

[0275] The system service framework is built to provide a hybrid voice coder voice processing method and a system for application in mobile communication networks, which is adapted to the communication environment with limited bandwidth. The terminal negotiates with the edge gateway to select a hybrid voice coder with encoding and decoding capabilities. The gateway supports transcoding to traditional voice coding to be compatible with the IMS network.

[0276] The call establishment process of the hybrid vocoder includes the terminal requesting the gateway to obtain a list of supported hybrid vocoders during the call establishment phase, and selecting the optimal hybrid vocoder to establish a call connection to adapt to the communication environment.

[0277] The terminal encoding and decoding business process includes a hybrid vocoder including a parameter encoding module, a parameter decoding module, a deep learning model module and a model training module. The parameter encoding module encodes the characteristic parameters of the original voice data to meet the bandwidth resource constraints of the narrowband communication network. The parameter decoding module and the deep learning model module work together at the receiving end to optimize and restore the decoded voice signal to achieve high-quality voice calls.

[0278] The structure of the hybrid vocoder includes the hybrid vocoder as the core component of the terminal device operating system, including the codec strategy manager, encoder, decoder and data interface. The data interface is connected to the voice acquisition driver and the voice playback driver externally, and is connected to the encoder and decoder internally.

[0279] Detailed technical features of the encoder and decoder, including the encoder includes a parameter coding module, including but not limited to baseband estimation, excitation algorithm, linear prediction coding and other characteristic parameter coding methods, to meet bandwidth resource constraints. The decoder is composed of a parameter decoder and a deep learning generator, including but not limited to frequency domain transformation, magnitude recovery, phase synthesis and other methods, using a deep learning model to optimize and restore the speech signal.

[0280] The embodiments of the present disclosure have the following technical advantages:

[0281] Narrowband communication adaptability: Through the hybrid vocoder, high-quality voice calls can be made in a limited bandwidth communication environment. This solves the problem that high-speed vocoders in traditional communication networks are difficult to ensure real-time high-quality voice calls in narrowband communication networks.

[0282] Universal mobile terminal application: It fills the gap in the deployment of ultra-slow-speed vocoders on universal mobile phones, making the technology adaptable to universal mobile phone terminals, and expanding the application scope of real-time high-quality voice calls in extreme narrow-band scenarios such as high-orbit satellites and underwater submarines.

[0283] Comprehensive parameter and data-driven advantages: It combines the advantages of low computing resource requirements of parametric vocoders and high voice reconstruction quality of data-driven vocoders. Through the hybrid encoding and decoding strategy, low-rate voice calls are possible, providing a solution for scenarios with limited bandwidth.

[0284] The embodiments of the present disclosure provide a comprehensive and widely applicable solution to solve the problems of medium and low-speed vocoders in narrowband communication environments.

[0285] It should be noted that the acquisition, storage, use, and processing of data in the technical solution disclosed in this disclosure are in compliance with the relevant provisions of national laws and regulations, and various types of data such as personal identity data, operation data, behavioral data, etc. related to individuals, customers, and groups obtained in the embodiments of this disclosure have all been authorized.

[0286] Fig.24 A schematic diagram of a voice communication method in an embodiment of the present disclosure applied to a server is shown. Fig.24 As shown, when the voice communication method is applied to the server, it includes:

[0287] Step S2402: receiving a first data stream sent by a terminal; the first data stream is determined by encoding the collected original voice signal by the terminal;

[0288] Step S2404: determine the second data stream according to the first data stream, and return the second data stream to the terminal, so that the terminal determines the optimized voice signal according to the second data stream; wherein the terminal determines the optimized voice signal according to the second data stream, including: the terminal decodes the second data stream to determine the play voice signal, the terminal optimizes and reconstructs the play voice signal using a trained deep learning model to determine the optimized voice signal; the bandwidth of the first data stream and the second data stream meets the bandwidth resource constraints of narrowband communication.

[0289] Fig.25 FIG. 1 shows a schematic diagram of a voice communication method applied to a voice communication system in an embodiment of the present disclosure. Fig.25 As shown, when the voice communication method is applied to a voice communication system, it includes:

[0290] Step S2502: the terminal encodes the collected original voice signal, determines the first data stream and sends it to the server;

[0291] Step S2504: the server determines the second data stream according to the first data stream, and returns the second data stream to the terminal; the bandwidth of the first data stream and the second data stream meets the bandwidth resource constraint of narrowband communication;

[0292] Step S2506: The terminal decodes the second data stream and determines to play a voice signal;

[0293] Step S2508: The terminal optimizes and reconstructs the played voice signal using the trained deep learning model to determine the optimized voice signal.

[0294] A schematic diagram of a voice communication system in an embodiment of the present disclosure, the voice communication system includes a terminal and a server;

[0295] The terminal is used to encode the collected original voice signal, determine the first data stream and send it to the server;

[0296] The server is used to determine the second data stream according to the first data stream and return the second data stream to the terminal; the bandwidth of the first data stream and the second data stream conforms to the bandwidth resource constraint of the narrowband communication;

[0297] The terminal is also used for the terminal to decode the second data stream and determine the playback voice signal; use the trained deep learning model to optimize and reconstruct the playback voice signal to determine the optimized voice signal.

[0298] The speech processing method of the hybrid vocoder of the embodiment of the present invention and the system for its application in the mobile communication network are used to cope with the communication environment with limited bandwidth. It relates to the field of digital voice compression and can be applied in scenarios such as voice call services of high-orbit satellite communications and voice services of underwater acoustic communications in deep-sea exploration and scientific research. The hybrid vocoder system includes a parameter encoder and a generator. The former compresses the voice signal to a low rate through parameter encoding, and the latter realizes the recovery of high-quality sound waveforms through an AI-driven voice generation model. The parameter encoder also provides a universal voice signal compression method, including a bit allocation strategy for features such as pitch and energy. It covers the voice call service system transformation framework of the mobile communication network, including the deployment of terminals and gateways. Through differentiated user service processing, personalized service allocation for different users is realized according to terminal characteristics, thereby improving user experience. The difference between this application and the prior art is that through the hybrid vocoder system and intelligent optimization, the existing technical problems in low rate, high delay and differentiated services are effectively solved, providing telecom operators and satellite communications with a more flexible and efficient voice call solution.

[0299] The main advantages are: the advantages of comprehensive parameter-based and data-driven vocoders are combined, and the problems of traditional vocoders are overcome by combining the advantages of parameter-based and data-driven vocoders. The parameter-based vocoder has the advantage of low computing resource requirements, while the data-driven vocoder performs well in voice quality. The hybrid vocoder of the present invention not only achieves low resource requirements, but also can compress and reconstruct voice signals with high quality. Suitable for narrowband communication networks in extreme scenarios: Focusing on providing narrowband voice communications in extreme scenarios such as high-orbit satellites and underwater submarines, it solves the challenge of limited bandwidth in traditional communication networks. This is of key significance for real-time, high-quality voice calls in narrowband communication networks. A low-rate voice coding and decoding method of parameter coding and AI reconstruction is creatively proposed to achieve a high-quality voice call solution at an ultra-low bit rate. In view of the bandwidth constraints of narrowband communication networks, the hybrid vocoder optimizes the coding and decoding strategy, parameter coding and decoding technology, realizes efficient voice transmission under limited bandwidth, and realizes compatibility of the hybrid vocoder with the traditional IMS network through edge gateway transcoding, providing better adaptability for the business. Applicable to general mobile phone terminals: There is no relevant patent to deploy ultra-low-speed vocoders to general mobile phone terminals. The hybrid vocoder provides a solution for application on general mobile phone terminals. Solve the defects of existing technologies: It overcomes the defects of low voice quality of parameter-based vocoders and large computing resource requirements of data-driven vocoders, so that the hybrid vocoder can take into account both voice quality and computing efficiency in narrowband scenarios.

[0300] It is applied to voice call services in high-orbit satellite communications, voice services in underwater acoustic communications in deep-sea exploration and scientific research, and voice services in emergency rescue scenarios, etc., to improve service coverage. Since the patent focuses on providing narrowband voice communications in extreme scenarios, the technology is used to improve its service coverage. In scenarios with greater communication challenges such as high-orbit satellites and underwater submarines, operators can provide more stable and high-quality voice communication services to meet users' communication needs in special environments. Differentiated service advantages: By applying ultra-low-speed vocoders on general mobile phone terminals, differentiated voice call services can be provided to provide users with a high-quality voice call experience. Responding to challenges in extreme scenarios: In extreme scenarios, such as high-orbit satellite communications, limited bandwidth and high latency are often faced. The ultra-low-speed vocoder technology provided by this patent effectively responds to these challenges, providing China Telecom with the ability to provide voice communication services under extreme conditions, thereby increasing its competitiveness in special markets.

[0301] Based on the same inventive concept, the present disclosure also provides a voice communication device, such as the following embodiment. Since the principle of solving the problem in the device embodiment is similar to that in the above method embodiment, the implementation of the device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0302] Fig.26 A schematic diagram of a voice communication device in an embodiment of the present disclosure is shown, which is applied to a terminal, such as Fig.26 As shown, the device comprises:

[0303] The encoding module 2601 is used to encode the collected original voice signal, determine the first data stream and send it to the server;

[0304] The decoding module 2602 is used to decode the second data stream in response to the second data stream returned by the server according to the first data stream, and determine to play the voice signal; the bandwidth of the first data stream and the second data stream meets the bandwidth resource constraint of the narrowband communication;

[0305] The optimization module 2603 is used to optimize and reconstruct the played voice signal using the trained deep learning model to determine the optimized voice signal.

[0306] It should be noted that the encoding module 2601, decoding module 2602 and optimization module 2603 correspond to S202 to S206 in the method embodiment, and the examples and application scenarios implemented by the modules and corresponding steps are the same, but are not limited to the contents disclosed in the method embodiment. It should be noted that the modules as part of the device can be executed in a computer system such as a set of computer executable instructions.

[0307] Based on the same inventive concept, the present disclosure also provides a voice communication device, such as the following embodiment. Since the principle of solving the problem in the device embodiment is similar to that in the above method embodiment, the implementation of the device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0308] Fig. 27 A schematic diagram of a voice communication device in another embodiment of the present disclosure is shown, which is applied to a server, such as Fig. 27 As shown, the device comprises:

[0309] The first data stream receiving module 2701 is used to receive the first data stream sent by the terminal; the first data stream is determined by encoding the collected original voice signal by the terminal;

[0310] The second data stream determination module 2702 is used to determine the second data stream according to the first data stream, and return the second data stream to the terminal so that the terminal determines the optimized voice signal according to the second data stream; wherein the terminal determines the optimized voice signal according to the second data stream, including: the terminal decodes the second data stream to determine the play voice signal, the terminal uses the trained deep learning model to optimize and reconstruct the play voice signal, and determines the optimized voice signal; the bandwidth of the first data stream and the second data stream meets the bandwidth resource constraints of narrowband communication.

[0311] It should be noted that the first data stream receiving module 2701 and the second data stream determining module 2702 correspond to S2402 to S2406 in the method embodiment, and the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the contents disclosed in the above method embodiment. It should be noted that the above modules, as part of the device, can be executed in a computer system such as a set of computer executable instructions.

[0312] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0313] Refer to the following Fig.28 hereinafter describes an electronic device 2800 according to this embodiment of the present disclosure. Fig.28 The electronic device 2800 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0314] like Fig.28As shown, the electronic device 2800 is in the form of a general computing device. The components of the electronic device 2800 may include but are not limited to: at least one processing unit 2810, at least one storage unit 2820, and a bus 2830 connecting different system components (including the storage unit 2820 and the processing unit 2810).

[0315] Among them, the storage unit stores a program code, and the program code can be executed by the processing unit 2810, so that the processing unit 2810 performs the steps described in the above "Exemplary Method" section of this specification according to various exemplary embodiments of the present disclosure. For example, the processing unit 2810 can perform the following steps of the above method embodiment: encode the collected original voice signal, determine the first data stream and send it to the server; in response to the second data stream returned by the server according to the first data stream, decode the second data stream and determine the playback voice signal; the bandwidth of the first data stream and the second data stream meet the bandwidth resource constraints of narrowband communication; use the trained deep learning model to optimize and reconstruct the playback voice signal to determine the optimized voice signal.

[0316] The storage unit 2820 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 28201 and / or a cache storage unit 28202 , and may further include a read-only storage unit (ROM) 28203 .

[0317] The storage unit 2820 may also include a program / utility 28204 having a set (at least one) of program modules 28205, such program modules 28205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0318] Bus 2830 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0319] The electronic device 2800 may also communicate with one or more external devices 2840 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 2800, and / or communicate with any device that enables the electronic device 2800 to communicate with one or more other computing devices (e.g., routers, modems, etc.). This communication may be performed through an input / output (I / O) interface 2850. In addition, the electronic device 2800 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) through a network adapter 2860. As shown, the network adapter 2860 communicates with other modules of the electronic device 2800 through a bus 2830. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 2800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0320] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0321] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart may be implemented as a computer program product, which includes: a computer program, which implements the above voice communication method when executed by a processor.

[0322] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. A program product capable of implementing the above method of the present disclosure is stored thereon. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to cause the terminal device to execute the steps according to various exemplary implementations of the present disclosure described in the above “Exemplary Method” section of this specification.

[0323] More specific examples of computer-readable storage media in the present disclosure may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0324] In the present disclosure, a computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0325] Alternatively, the program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0326] In specific implementation, the program code for performing the operation of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0327] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0328] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0329] Through the description of the above implementation modes, it is easy for those skilled in the art to understand that the example implementation modes described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation mode of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation mode of the present disclosure.

[0330] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A voice communication method, characterized in that: Applied to a terminal, the method comprises: Encoding the collected original voice signal, determining a first data stream and sending it to the server; the first data stream has a target communication terminal; In response to the second data stream returned by the server according to the first data stream, the second data stream is decoded to determine to play the voice signal; the bandwidth of the first data stream and the second data stream meets the bandwidth resource constraint of narrowband communication; Utilizing a trained deep learning model to optimize and reconstruct the playback voice signal, and determine an optimized voice signal; The server returns the second data stream according to the first data stream, including: The server sends the first data stream to the target communication terminal, so that the target communication terminal decodes the first data stream to obtain an original voice signal, and sends a second data stream generated in response to the original voice signal to the server; The server returns the second data stream to the source of the first data stream.

2. The voice communication method according to claim 1, characterized in that: Encoding the collected original voice signal, determining the first data stream and sending it to the server, including: Collect original voice signals using voice collection driver; Performing feature encoding on the original voice signal through a coding and decoding strategy to determine a first data stream; The first data stream is sent to the server.

3. The voice communication method according to claim 2, characterized in that: The method of encoding the original speech signal by a coding and decoding strategy to determine a first data stream includes: Acquiring parameters of the original speech signal; Extracting and quantizing the parameters of the original speech signal to determine the extracted and quantized parameters; The extracted and quantized parameters are converted into a bit stream to determine a first data stream.

4. The voice communication method according to claim 3, characterized in that: Acquiring parameters of the original speech signal includes: Determining a speech signal frame of the original speech signal; Performing fundamental frequency estimation on the speech signal frame to determine the pitch; Performing a fast Fourier transform on the time domain signal of the speech signal frame to determine a frequency domain signal; Performing amplitude estimation on the pitch and the frequency domain signal to determine the amplitude of the sine wave; Performing a mute analysis on the amplitude of the sine wave to determine a mute analysis result; The pitch, the frequency domain signal, the sine wave amplitude and the silence analysis result are determined as parameters of the original speech signal.

5. The voice communication method according to claim 4, characterized in that: Extracting and quantizing the parameters of the original speech signal to determine the extracted and quantized parameters includes: Sampling the pitch, the frequency domain signal, the sine wave amplitude and the silence analysis result to reduce the bit rate, and determining the extracted parameters; The extracted parameters are quantized to convert the bits of the original speech signal into a set number, and the extracted quantized parameters are determined.

6. The voice communication method according to claim 1, characterized in that: The second data stream is decoded to determine the voice signal to be played, including: quantize and restore the second data stream to determine signal frame parameters; amplifying the signal frame parameters to determine energy amplitude; The energy amplitude is used to perform superposition synthesis to determine the playing voice signal.

7. The voice communication method according to claim 6, characterized in that: The signal frame parameters include: tone, spectrum amplitude, and pitch; Amplifying the signal frame parameters to determine the energy amplitude includes: Interpolate the tone, spectrum amplitude and pitch to resample to a target bit rate, and determine an interpolation result; Performing energy amplitude synthesis on the interpolation results to determine the energy amplitude; The energy amplitude is filtered to determine the filtered energy amplitude.

8. The voice communication method according to claim 7, characterized in that: The energy amplitude is used for superposition synthesis to determine the playing voice signal, including: Reconstructing the phase of the energy amplitude to determine phase information; Performing an inverse fast Fourier transform on the phase information and the filtered energy amplitude to determine a time domain signal; The time domain signals are superimposed and synthesized to determine the playing voice signal.

9. The voice communication method according to claim 1, characterized in that: The trained deep learning model includes: a data processing layer, a reversible convolution layer, and an affine coupling layer; Optimizing and reconstructing the played voice signal using a trained deep learning model to determine an optimized voice signal includes: Input the played voice signal into the data processing layer, use the data processing layer to process the played voice signal into vector data and input the vector data into the reversible convolution layer; Using the reversible convolution layer to perform reversible convolution calculation on the vector data, determine the convolution data and input it into the affine coupling layer; The convolution data is affine coupled using the affine coupling layer to determine an optimized speech signal.

10. The voice communication method according to claim 9, characterized in that: The convolution data is affine coupled by using the affine coupling layer to determine an optimized speech signal, including: Convert the convolution data into a plurality of playback voice signal segments; Dividing each played speech signal segment equally into a first sub-segment and a second sub-segment; Inputting the first sub-segment and the Mel-spectrogram data into a trained deep neural network to determine a learned first sub-segment; the Mel-spectrogram data is calculated based on the played speech signal segment; Inversely mapping the learned first sub-segment and the second sub-segment to determine an affine second sub-segment; The first sub-segment and the affine second sub-segment are coupled in series to determine an optimized speech signal.

11. The voice communication method according to claim 10, characterized in that: The learned first sub-segment includes: a first calculation parameter and a second calculation parameter; The step of inversely mapping the learned first sub-segment and the second sub-segment to determine an affine second sub-segment includes: The first calculation parameter is matrix-dot-multiplied with the second sub-segment, and then added to the second calculation parameter to determine an affine second sub-segment.

12. The voice communication method according to claim 1, characterized in that: The deep learning model includes: a data processing layer, a reversible convolution layer, and an affine coupling layer; the affine coupling layer includes: a deep neural network; The training process of the deep learning model includes: Input the acquired training voice signal into the data processing layer, and use the data processing layer to process the playback voice signal into vector data and input the vector data into the reversible convolution layer; Using the reversible convolution layer to perform reversible convolution calculation on the vector data, determine the convolution data and input it into the affine coupling layer; The deep neural network is trained using the convolution data in the affine coupling layer, and when the optimization objective function of the deep neural network reaches a target value, a trained deep neural network is determined; The affine coupling layer including the trained neural network as well as the data processing layer and the reversible convolution layer are determined as a trained deep learning model.

13. The voice communication method according to claim 12, characterized in that: The deep neural network is trained using the convolution data in the affine coupling layer, and when the optimization objective function of the deep neural network reaches a target value, a trained deep neural network is determined, including: Dividing the training speech signal into a plurality of training speech signal segments; Dividing each training speech signal segment equally into a first training sub-segment and a second training sub-segment; Inputting the first training sub-segment and the training Mel-spectrogram data into a deep neural network to determine the training first sub-segment; the training Mel-spectrogram data is calculated based on the training speech signal segment; Inversely mapping the first training sub-segment and the second training sub-segment to determine an affine training second sub-segment; The first training sub-segment and the affine training second sub-segment are coupled in series to determine a coupled speech signal; Re-inputting the coupled speech signal into the deep neural network to obtain an updated coupled speech signal, and monitoring whether the optimization objective function of the deep neural network reaches a target value; If the optimization objective function of the deep neural network does not reach the target value, then continue to loop train the deep neural network; If the optimization objective function of the deep neural network reaches the target value, a trained deep neural network is determined.

14. The voice communication method according to claim 1, characterized in that: Before the original speech signal is collected, it also includes: Sending a vocoder acquisition request to a server to obtain a vocoder list of the server, wherein the vocoder list includes multiple vocoders, and the vocoders are used to encode speech signals and decode data streams; According to the bandwidth resource constraint of the narrowband communication, a target vocoder is selected from the vocoder list of the server.

15. A voice communication method, characterized in that: Applied to the server, the method includes: A first data stream sent by a receiving terminal; the first data stream is determined by encoding the collected original voice signal by the terminal; the first data stream has a target communication terminal; Determine a second data stream according to the first data stream, and return the second data stream to the terminal, so that the terminal determines an optimized voice signal according to the second data stream; wherein the terminal determines the optimized voice signal according to the second data stream, including: the terminal decodes the second data stream to determine a play voice signal, and the terminal optimizes and reconstructs the play voice signal using a trained deep learning model to determine an optimized voice signal; the bandwidth of the first data stream and the second data stream meets the bandwidth resource constraints of narrowband communication; Among them, determining the second data stream based on the first data stream and returning the second data stream to the terminal includes: sending the first data stream to the target communication terminal so that the target communication terminal decodes the first data stream to obtain an original voice signal, and sending the second data stream generated in response to the original voice signal to the server; the server returns the second data stream to the source terminal of the first data stream.

16. A voice communication method, characterized in that: Applied to a voice communication system, the voice communication system includes a terminal and a server, and the method includes: The terminal encodes the collected original voice signal, determines a first data stream and sends it to the server; the first data stream has a target communication terminal; The server determines a second data stream according to the first data stream, and returns the second data stream to the terminal; the bandwidth of the first data stream and the second data stream conforms to the bandwidth resource constraint of narrowband communication; The terminal decodes the second data stream and determines to play a voice signal; The terminal optimizes and reconstructs the played voice signal using a trained deep learning model to determine an optimized voice signal; Among them, the server determines the second data stream based on the first data stream and returns the second data stream to the terminal, including: the server sends the first data stream to the target communication terminal so that the target communication terminal decodes the first data stream to obtain the original voice signal, and sends the second data stream generated in response to the original voice signal to the server; the server returns the second data stream to the source terminal of the first data stream.

17. A voice communication device, characterized in that: Applied to a terminal, the device comprises: The encoding module is used to encode the collected original voice signal, determine the first data stream and send it to the server; the first data stream has a target communication terminal; A decoding module, configured to decode the second data stream in response to the second data stream returned by the server according to the first data stream, and determine to play a voice signal; the bandwidth of the first data stream and the second data stream conform to the bandwidth resource constraint of narrowband communication; An optimization module, used to optimize and reconstruct the played voice signal using a trained deep learning model to determine an optimized voice signal; Among them, the server returns the second data stream according to the first data stream, including: the server sends the first data stream to the target communication terminal, so that the target communication terminal decodes the first data stream to obtain the original voice signal, and sends the second data stream generated in response to the original voice signal to the server; the server returns the second data stream to the source of the first data stream.

18. A voice communication device, characterized in that: Applied to the server, the device comprises: A first data stream receiving module is used to receive a first data stream sent by a terminal; the first data stream is determined by encoding the collected original voice signal by the terminal; the first data stream has a target communication terminal; A second data stream determination module is used to determine a second data stream according to the first data stream, and return the second data stream to the terminal, so that the terminal determines an optimized voice signal according to the second data stream; wherein, the terminal determines the optimized voice signal according to the second data stream, including: the terminal decodes the second data stream to determine a play voice signal, and the terminal optimizes and reconstructs the play voice signal using a trained deep learning model to determine the optimized voice signal; the bandwidth of the first data stream and the second data stream conform to the bandwidth resource constraint of narrowband communication; wherein, determining the second data stream according to the first data stream and returning the second data stream to the terminal includes: sending the first data stream to the target communication terminal, so that the target communication terminal decodes the first data stream to obtain an original voice signal, and sends the second data stream generated in response to the original voice signal to the server; the server returns the second data stream to the source terminal of the first data stream.

19. A voice communication system, characterized in that: The voice communication system includes a terminal and a server; The terminal is used to encode the collected original voice signal, determine the first data stream and send it to the server; the first data stream has a target communication terminal; The server is configured to determine a second data stream according to the first data stream, and return the second data stream to the terminal; the bandwidths of the first data stream and the second data stream comply with bandwidth resource constraints of narrowband communication; The terminal is further used for the terminal to decode the second data stream to determine a play voice signal; optimize and reconstruct the play voice signal using a trained deep learning model to determine an optimized voice signal; Among them, the server is specifically used for: the server sends the first data stream to the target communication terminal, so that the target communication terminal decodes the first data stream to obtain the original voice signal, and sends the second data stream generated in response to the original voice signal to the server; the server returns the second data stream to the source terminal of the first data stream.

20. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the voice communication method described in any one of claims 1 to 14 or claim 15 or claim 16 by executing the executable instructions.

21. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the voice communication method described in any one of claims 1 to 14 or claim 15 or claim 16 is implemented.

22. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the voice communication method described in any one of claims 1 to 14 or claim 15 or claim 16 is implemented.

Images