Voice data processing method, storage medium and electronic equipment
By processing and encoding the pronunciation characteristics of the voice signal in frame-by-frame processing and encoding the pronunciation characteristics of the voice signal at the voice transmitting end, it is transmitted to the voice receiver for acoustic synthesis, which solves the high-definition bandwidth and processing capabilities of voice data processing, and realizes high-quality voice under narrowband communication call.
Patent Information
- Application Number
- CN202411181974.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2024-12-06
AI Technical Summary
In the prior art, voice data processing methods have high requirements for communication bandwidth and terminal processing capabilities, and it is difficult to realize high-quality voice calls under the conditions of limited bandwidth or insufficient terminal resources.
By performing frame processing and pronunciation feature extraction on the voice signal at the voice sending end, encoding pronunciation units and pronunciation information, and transmitting it to the voice receiving end for acoustic synthesis, avoiding text recognition and linguistic processing, and reducing encoding coding rate and processing resources Require.
Effectively transmit all semantic and subsemantic information of speech at low encoding rate, realizing high-quality voice calls in narrowband communication scenarios, reducing the requirements for terminal resources.
Smart Images

Figure CN119094506A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of communications, and in particular to a method for processing voice data, a storage medium and an electronic device. Background Art
[0002] Voice can convey three parts of content, including semantics (text content), subsemantics (rhythm and intonation), and timbre (personal characteristics), and can express more accurate and comprehensive information than simple text, such as questions, antonyms, urgency, emotions, etc. Therefore, voice calls are the most important form of emergency communication and cannot be completely replaced by text communication.
[0003] However, voice calls have certain requirements on communication bandwidth and terminal processing capabilities, and cannot be realized under conditions where the communication bandwidth is narrow or the communication terminal processing capabilities are limited. For example, in application scenarios such as high-orbit satellite communications, high-speed frequency-hopping secure communications, and underwater personnel communications, the communication bandwidth is very limited, so it is difficult to achieve voice call functions with basic sound quality.
[0004] It can be seen that the voice data processing method in the related art has the problem of high requirements on communication bandwidth and terminal processing capabilities. Summary of the invention
[0005] The embodiments of the present application provide a method for processing voice data, a storage medium, and an electronic device, so as to at least solve the problem that the method for processing voice data in the related art has high requirements on communication bandwidth and terminal processing capability.
[0006] According to one aspect of an embodiment of the present application, a method for processing speech data is provided, which is applied to a speech sending end, and the method includes: acquiring a speech signal to be sent, performing frame processing on the speech signal to be sent, and obtaining a group of target speech frames; converting the group of target speech frames into a group of pronunciation units, and performing rhythmic feature extraction on the group of target speech frames to obtain target rhythmic information; encoding the target rhythmic information and the group of pronunciation units to obtain first speech data; and sending the first speech data to a speech receiving end so that the speech receiving end performs acoustic synthesis based on the first speech data.
[0007] According to another aspect of an embodiment of the present application, a method for processing voice data is also provided, which is applied to a voice receiving end, and the method includes: receiving first voice data sent by a voice sending end, wherein the first voice data packet contains encoded target prosody information and an encoded group of pronunciation units; decoding the encoded target prosody information and the encoded group of pronunciation units to obtain the target prosody information and the group of pronunciation units; acoustically synthesizing the group of pronunciation units based on the target prosody information to obtain a first voice signal, and playing the obtained first voice signal.
[0008] According to another aspect of the embodiments of the present application, a terminal device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the above-mentioned voice data processing method through the computer program.
[0009] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned method for processing speech data when running.
[0010] According to another aspect of the embodiment of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above method for processing voice data.
[0011] In an embodiment of the present application, a method of transmitting pronunciation unit encoding carrying prosody information is adopted to obtain a voice signal to be sent, and the voice signal to be sent is framed to obtain a group of target voice frames; a group of target voice frames is converted into a group of pronunciation units, and prosody features are extracted for a group of target voice frames to obtain target prosody information; the target prosody information and a group of pronunciation units are encoded to obtain first voice data; the first voice data is sent to a voice receiving end so that the voice receiving end performs acoustic synthesis based on the first voice data. Since the voice waveform is converted into the corresponding pronunciation unit for encoding and transmission at the voice sending end, no text recognition, linguistic processing or text-based acoustic processing is performed, and the waveform and timbre information is not recorded, while the encoding bit rate is greatly reduced, the requirements for processing resources are greatly reduced. At the same time, the acoustic prosody information in the voice waveform is added to the encoding transmission at the voice sending end, and all semantic and sub-semantic valid information contained in the voice is effectively transmitted under the premise of low encoding bit rate, and the high requirements for terminal resources are also avoided, thereby solving the problem that the voice data processing method in the related art has high requirements for communication bandwidth and terminal processing capability. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0013] Figure 1 is a schematic diagram of an application environment of an optional method for processing voice data according to an embodiment of the present application;
[0014] Figure 2 is a flowchart of an optional method for processing voice data according to an embodiment of the present application;
[0015] Figure 3 is a schematic diagram of an optional method for processing voice data according to an embodiment of the present application;
[0016] Figure 4 is a schematic diagram of another optional method for processing voice data according to an embodiment of the present application;
[0017] Figure 5 is a schematic diagram of another optional method for processing voice data according to an embodiment of the present application;
[0018] Figure 6 is a flowchart of another optional method for processing voice data according to an embodiment of the present application;
[0019] Figure 7 is a schematic diagram of another optional method for processing voice data according to an embodiment of the present application;
[0020] Figure 8 is a schematic diagram of another optional method for processing voice data according to an embodiment of the present application;
[0021] Fig. 9 It is a flowchart of another optional method for processing voice data according to an embodiment of the present application. DETAILED DESCRIPTION
[0022] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.
[0023] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0024] According to one aspect of the embodiments of the present application, a method for processing voice data is provided. Optionally, as an optional implementation, the method for processing voice data can be but is not limited to being applied to: Figure 1 In the hardware environment composed of the terminal device 102 and the server 104 shown in FIG. Figure 1 As shown, the server 104 is connected to the terminal device 102 via a network, and can be used to provide services for the terminal device or a client installed on the terminal device. A database can be set on the server or independently of the server to provide data storage services for the server 104. Cloud computing and / or edge computing services can be configured on the server or independently of the server to provide data computing services for the server 104.
[0025] The above network may include but is not limited to at least one of the following: wired network, wireless network. The above wired network may include but is not limited to at least one of the following: wide area network, metropolitan area network, local area network, and the above wireless network may include but is not limited to at least one of the following: WIFI (Wireless Fidelity), Bluetooth. The terminal device 102 may include but is not limited to a PC, a mobile phone, or a tablet computer, and the terminal device 102 may include a voice transmitter and a voice receiver.
[0026] The method for processing voice data in the embodiment of the present application may be executed by the server 104 , or may be executed jointly by the terminal device 102 and the server 104 , which is not limited in the present embodiment.
[0027] The voice data processing method in this embodiment can be applied to the scenario where signals are transmitted between terminal devices through satellites as relay stations to achieve voice calls, that is, satellite communications, or extremely low bandwidth scenarios under narrowband private networks, which have low requirements on the processing capabilities of terminal devices. Optionally, the terminal device may be a communication device that has certain limitations in processing data, information or signals and cannot handle large or complex tasks. For example, the processor performance is insufficient, the processor speed of the communication device is slow, and it cannot quickly process large amounts of data or complex tasks; the memory is insufficient, the memory capacity of the communication device is limited, and it cannot store large amounts of data or run multiple tasks at the same time; the bandwidth is limited, the bandwidth of the communication device is limited, and it cannot quickly transmit large amounts of data or signals, etc.
[0028] Voice can convey three parts of content, including semantics (text content), sub-semantics (rhythm and tone), and timbre (personal characteristics), and can express more accurate and comprehensive information than simple text, such as questions, antonyms, urgency, emotions, etc. Therefore, voice calls are the most important form of emergency communication and cannot be completely replaced by text communications. However, voice calls have certain requirements for communication bandwidth and terminal processing capabilities, and cannot be achieved under conditions where the communication bandwidth is extremely limited or the communication terminal processing capabilities are limited. In some scenarios, such as applications such as high-orbit satellite communications, high-speed frequency hopping secure communications, and underwater personnel communications, the communication bandwidth is very limited, so it is difficult to achieve voice call functions with basic sound quality.
[0029] The speech coding technology in the related art is to sample, extract, compress and encode the speech waveform. In order to ensure that the speech waveform is basically restored, the encoding bit rate is relatively high, and as the encoding bit rate decreases, the sound quality also deteriorates.
[0030] At present, another approach can be adopted to realize speech coding, that is, using ASR (Automatic Speech Recognition) and TTS (Text To Speech, i.e., speech synthesis) technology. Through this technology, waveform information is no longer transmitted, but speech recognition results are transmitted, and the rhythm information of the original speech is combined and resynthesized at the receiving end. This new speech coding idea has the characteristics of low coding bit rate and high sound quality.
[0031] However, currently, the methods for speech encoding using ASR and TTS technologies have not disassembled the ASR and TTS processes, and have completely retained their respective linguistic processing parts. First, using text as the conversion medium can only process specific language information, and cannot process multi-language or non-language human voices; second, using the rhythm information predicted by linguistics as a reference, it cannot truly restore the original voice rhythm of the speaker, which does not meet the actual needs of voice intercom; third, the linguistic processing part has very high requirements for terminal resources. For example, taking a mature product in the market as an example, the function operation requires a CPU with a 2-core A53 1.5GHz main frequency, a ROM resource of 1.8G, and a runtime RAM of 500MB, which cannot be met by narrowband communication terminals.
[0032] In order to at least partially solve the above technical problems, this application proposes a speech encoding method with extremely low bit rate and low resources that carries complete semantic and sub-semantic information. Acoustic feature extraction and acoustic model are used at the speech sending end to output pronunciation unit codes with additional prosodic information; at the speech receiving end, the pronunciation unit codes carrying prosodic information are directly input into the acoustic synthesis system to restore the speech with complete semantic and sub-semantic information. By adding prosodic annotation at the speech unit level, it is possible to retain the real sub-semantic information while maintaining the advantages of extremely low coding bit rate and high sound quality, while also avoiding high requirements on terminal resources.
[0033] Optionally, Figure 2 is a flow chart of an optional method for processing voice data according to an embodiment of the present application, such as Figure 2 As shown, the process of the method may include the following steps:
[0034] Step S202, obtaining a voice signal to be sent, performing frame processing on the voice signal to be sent, and obtaining a group of target voice frames.
[0035] As an optional solution, before the voice signal to be sent is framed and processed to obtain a group of target voice frames, the above method also includes: performing voice activity detection on the initial voice signal collected by the voice collection component of the voice sending end to obtain the voice start time point and the voice end time point; and determining the voice signal in the initial voice signal that is between the voice start time point and the voice end time point as the voice signal to be sent.
[0036] Here, the voice collection component may include a microphone or other components for capturing sound signals. The target pronunciation object may be a terminal user object corresponding to the voice transmitter. VAD (Voice activity detection) may be to analyze and process the input voice signal according to a certain algorithm to determine whether the voice signal exists. Its function is to determine the effective part of the voice signal, that is, to determine the time point when the voice starts and ends, so that only the effective voice part is analyzed and recognized in the subsequent processing, reducing the amount of calculation and improving the recognition accuracy.
[0037] As an optional solution, the voice signal to be sent is processed by frame processing to obtain a group of voice frames, including: sampling the voice signal to be sent according to a preset sampling interval and digitizing each sampling point obtained by sampling to obtain a digitized voice signal; framing the digitized voice signal according to a specified sliding window of a specified time length to obtain a group of initial voice frames; vectorizing each initial voice frame in a group of initial voice frames to obtain a group of target voice frames.
[0038] In order to facilitate tasks such as digital signal processing and speech recognition, in this embodiment, the continuous speech signal can be sampled at a certain time interval, the analog signal can be converted into a digital signal, the sampled speech signal can be divided into small time periods, and a window of a certain length is usually used for framing operations, so that each time period can be subsequently processed, and the speech signal of each time period is feature extracted and converted into a feature vector representation, and MFCC (Mel Frequency Cepstral Coefficients) and other methods are usually used for vectorization processing. Through such processing, the speech signal can be converted into a digital signal, and useful feature vectors can be extracted to provide basic data for subsequent speech processing tasks.
[0039] For example, in this embodiment, the speech signal is divided into small segments, and the commonly used frame length is 20-30 milliseconds. The speech signal of each frame is windowed, and a window function such as a Hamming window or a Hanning window is usually used. Each frame of the speech signal is converted into a spectrum feature based on STFT (Short-time Fourier Transform). The spectrum feature is processed by a Mel filter bank and its logarithm is taken. Finally, a discrete cosine transform is performed to obtain MFCC coefficients, and the MFCC coefficients are normalized to eliminate the influence of factors such as different speakers and different environments on the features.
[0040] Step S204: converting a group of target speech frames into a group of pronunciation units, and extracting prosodic features from the group of target speech frames to obtain target prosodic information.
[0041] In this embodiment, after completing signal processing and feature extraction of the speech waveform, the waveform can be converted into a state vector with a sliding window segment of about 20 ms, and then the maximum probability of the corresponding phoneme is calculated through the acoustic model to obtain a series of pronunciation units.
[0042] As an optional solution, a group of target speech frames is converted into a group of pronunciation units, including: extracting state features from each target speech frame in the group of target speech frames to obtain a group of state features; inputting the group of state features into a target acoustic model to obtain an output result of the target acoustic model, wherein the output result of the target acoustic model is used to indicate the probability that continuous speech frames in a group of target speech frames are combined into a pronunciation unit; based on the output result of the target acoustic model, converting the group of target speech frames into a group of pronunciation units, wherein each pronunciation unit is one of the following: phoneme, diphone, syllable, symbol.
[0043] After sampling and framing the speech signal to be transmitted to obtain a group of target speech frames, state features are extracted for each target speech frame in the group of target speech frames to obtain a group of state features. By inputting the state features into the target acoustic model, the observation / transition probability of the corresponding pronunciation unit of the audio can be calculated, that is, the probability that consecutive speech frames in a group of target speech frames are combined into a pronunciation unit, thereby converting a group of target speech frames into a group of pronunciation units, each pronunciation unit in a group of pronunciation units includes at least one target speech frame in a group of target speech frames, for example, 1 phoneme includes 3 speech frames and so on.
[0044] The input of the target acoustic model is the feature vector sequence of the speech signal after feature extraction, that is, the state feature, which usually uses Mel Frequency Cepstral Coefficients (MFCC) as features. These feature vectors are used to build an acoustic model, usually using a hidden Markov model (HMM) to model the speech signal. The output of the acoustic model is a sequence of pronunciation units corresponding to the feature vector sequence of the input acoustic model, such as a phoneme sequence. This process is usually called decoding, and its purpose is to infer the corresponding sequence of pronunciation units based on the input acoustic feature vector sequence.
[0045] Pronunciation units include phonemes, biphones, syllables, and symbols. Phonemes are the smallest units of speech in linguistics. Biphones are units of speech composed of two phonemes. Biphones are used to analyze continuous phoneme combinations in speech signals to improve recognition accuracy. Syllables are the basic prosodic units in speech, consisting of one or more phonemes, usually a vowel and one or more consonants. For example, in English, the word "cat" contains 3 phonemes. and Two syllables respectively.
[0046] For example, taking the pronunciation unit as phoneme, after extracting the features of the audio signal and obtaining a series of feature vectors, the Gaussian mixture model can be used to model the observation probability of each phoneme. The Gaussian mixture model can model the feature vector of each phoneme and obtain the probability distribution of the phoneme. In order to calculate the transition probability, suppose there is a language model, and the model indicates: A is usually connected to B, M is usually connected to N... Using this information, the transition probability can be calculated, that is, the probability of transferring from one phoneme to another. Combining the observation probability and the transition probability, the Viterbi algorithm can be used to calculate the optimal decoding result of the phoneme sequence corresponding to the speech signal to be sent.
[0047] At the same time, by performing prosodic feature extraction on a group of target speech frames, the target prosodic information can be obtained. That is, in the feature extraction part, the acoustic prosodic information such as sound length, energy, pitch, tone, pause, stress, rhythm, etc. in the speech waveform emitted by the target pronunciation object (actual speaker) is extracted.
[0048] Step S206: Encode the target prosody information and a group of pronunciation units to obtain first speech data.
[0049] The extracted acoustic prosody information and pronunciation units are annotated together, and then the pronunciation units carrying the actual prosody information are encoded as communication transmission content.
[0050] In this embodiment, semantic information can be expressed by pronunciation units such as phonemes, diphones, syllables, symbols, etc. The annotation of pronunciation units can use common methods such as the International Phonetic Alphabet IPA or the CMU pronunciation dictionary, etc.; sub-semantic information is mainly represented by prosodic states. The annotation of prosodic states can use internationally common annotation methods, such as the International Phonetic Alphabet IPA or the Speech Synthesis Markup Language SSML, etc., or customized annotation rules.
[0051] For example, combined with Figure 3Taking the custom encoding method of recording phonemes + rhythm as an example, a Chinese character has a maximum of 4 phonemes, and each phoneme has two bytes, plus one byte for length, intensity, pitch, tone, and pause status. Each Chinese character uses 7-28 bytes to identify complete semantic and sub-semantic information; based on the calculation of about 8-10 phonemes per second of fast broadcast, the encoding bit rate is 0.55kbps. If several rhythmic states share the same byte, the encoding bit rate can be further reduced, and this encoding method can accurately record any language or non-language human voice pronunciation information.
[0052] Through this embodiment, the problem of achieving real-time voice calls in extremely low bandwidth scenarios under satellite communication or narrowband private networks is solved. At the voice sending end, the voice waveform is converted into corresponding pronunciation units such as phonemes, semiphones, syllables, and symbols for encoding and transmission, without any text recognition, linguistic processing or text-based acoustic processing, and waveform and timbre information is not recorded. While greatly reducing the encoding bit rate, the requirements for processing resources are greatly reduced, so that the solution can be applied to embedded devices and can process all human voices such as multilingual and non-linguistic voices; at the same time, at the voice sending end, the acoustic rhythmic information such as sound length, energy, pitch, tone, pause, stress, rhythm, etc. in the voice waveform is added to the encoding transmission, and all semantic and sub-semantic effective information contained in the voice is effectively transmitted under the premise of low encoding bit rate, and real-time calls can be achieved in narrowband communication modes such as satellites and clusters.
[0053] Step S208: sending the first voice data to the voice receiving end, so that the voice receiving end performs acoustic synthesis based on the first voice data.
[0054] Similar to the above-mentioned embodiment, the first voice data includes the encoded target prosody information and a group of encoded pronunciation units. The pronunciation units carrying the actual prosody information are encoded as the communication transmission content and directly sent to the acoustic synthesis system for speech synthesis at the voice receiving end, so that high-quality speech with complete sub-semantic information can be obtained, but the timbre (pronunciation frequency) characteristics of the target pronunciation object are not included. The synthesized speech output in this way can accurately convey the actual intention of the target pronunciation object.
[0055] Through the present application, a voice signal to be sent is acquired, and the voice signal to be sent is processed into frames to obtain a group of target voice frames; a group of target voice frames is converted into a group of pronunciation units, and rhythmic features are extracted from a group of target voice frames to obtain target rhythmic information; the target rhythmic information and a group of pronunciation units are encoded to obtain first voice data; the first voice data is sent to a voice receiving end so that the voice receiving end performs acoustic synthesis based on the first voice data, thereby solving the problem that the voice data processing method in the related art has high requirements on the communication bandwidth and terminal processing capability.
[0056] As an optional solution, sending the first voice data to the voice receiving end includes:
[0057] S11, the voice sending end detects the current communication bandwidth;
[0058] S12, when the current communication bandwidth is less than or equal to the specified bandwidth value, sending the first voice data to the voice receiving end; or,
[0059] S13, when the current communication bandwidth is greater than or equal to the specified bandwidth value, obtaining voice feature information of the target pronunciation object, wherein the target pronunciation object is the user who generates the first voice data, and the voice feature information includes at least one of the following: timbre information, signal amplitude, signal fundamental frequency, and signal harmonics; sending the first voice data and the voice feature information of the target pronunciation object to the voice receiving end.
[0060] Optionally, in this embodiment, the transmission data content of the two parties in the call may be dynamically selected based on the current communication bandwidth.
[0061] When the current communication bandwidth is less than or equal to the specified bandwidth value, only the first voice data may be sent to the voice receiving end;
[0062] When the current communication bandwidth is greater than or equal to the specified bandwidth value, voice feature information of a target pronunciation object is obtained, wherein the target pronunciation object is a user who generates first voice data, and the voice feature information includes at least one of the following: timbre information, signal amplitude, signal fundamental frequency, and signal harmonics; the first voice data and the voice feature information of the target pronunciation object are sent to a voice receiving end.
[0063] Similar to the aforementioned embodiment, the first voice data is obtained by encoding the target prosody information and a group of pronunciation units. In the case of limited communication bandwidth, the voice sending end can only send the first voice data to the voice receiving end. Based on this, the voice receiving end can perform acoustic synthesis based only on the first voice data.
[0064] When the communication bandwidth is sufficient, the voice transmitting end can send the first voice data and the voice feature information of the target pronunciation object to the voice receiving end, based on which the voice receiving end can achieve timbre-level acoustic synthesis.
[0065] Optionally, in this embodiment, the timbre information of multiple target pronunciation objects can be sent to the receiving end for pre-storage in advance when the communication bandwidth is greater than or equal to the specified bandwidth value. When the current communication bandwidth is less than or equal to the specified bandwidth value, after the first voice data is sent to the voice receiving end, the voice receiving end can perform acoustic synthesis based on the first voice data and the pre-stored timbre information.
[0066] Optionally, the pre-stored timbre information may carry an object identifier, and the first voice data may also carry an object identifier. The receiving end may extract the timbre information matching the object identifier carried in the first voice data through the object identifier for acoustic synthesis.
[0067] Optionally, by utilizing the characteristics of extremely low bit rates, communication terminals can uniformly encode and transmit voice information on multiple wide-band and narrow-band networks at the same time, avoiding multiple encodings and improving efficiency.
[0068] As an optional solution, the voice sending end is a dual-mode terminal, including a narrowband module and a broadband module, and sends the first voice data to the voice receiving end, including:
[0069] S21, acquiring speech feature information of a target pronunciation object, and sending the speech feature information to a speech receiving end through a broadband channel by a broadband module of the speech transmitting end, wherein the target pronunciation object is a user who generates first speech data, and the speech feature information includes at least one of the following: timbre information, signal amplitude, signal fundamental frequency, and signal harmonics;
[0070] S22, the narrowband module of the voice sending end sends the first voice data to the voice receiving end through the narrowband channel.
[0071] The individual pronunciation characteristics of a specific speaker (i.e., voice feature information) are mainly composed of two parts: voice style (rhythm, emphasis, punctuation, and personalized voice) and timbre characteristics. Since the voice style information is already included in the aforementioned rhythmic pronunciation unit encoding method, it is only necessary to extract the timbre characteristics at the voice sending end, train the corresponding acoustic model in an offline or online manner, and then transmit the personalized acoustic model characteristics to the voice receiving end through a broadband channel, and apply them to the acoustic system for speech synthesis at the voice receiving end, so that the realistic original speaker's voice can be restored in a narrowband call based on the aforementioned rhythmic pronunciation unit encoding information.
[0072] Here, the acoustic system used for speech synthesis converts phonemes into pronunciation under standard linguistic rhythm through various methods (concatenating synthesis, parameter synthesis such as formant / LPC (Linear Predictive Coding) / PSOLA (Pitch Synchronous Overlap and Add), end-to-end synthesis of various neural network models, etc.).
[0073] As an optional solution, when the voice receiving end is a dual-mode terminal, the broadband module of the voice transmitting end sends the voice feature information of the target pronunciation object to the voice receiving end through the broadband channel;
[0074] When the voice receiving end is a single-mode terminal, the voice feature information of the target pronunciation object is sent to the relay terminal through the broadband channel between the voice sending end and the relay terminal, so that the relay terminal sends the voice feature information of the target pronunciation object to the voice receiving end through near field communication.
[0075] Since the aforementioned personalized acoustic model features (voice feature information of the target pronunciation object) only need to be transmitted once and have no strong correlation with the call service, they can be transmitted on the broadband side of the dual-mode terminal and handed over to the narrowband side for use.
[0076] like Figure 4 As shown, in this embodiment, in a multi-mode terminal or a multi-mode network, for example, the voice receiving end is a dual-mode terminal, and under the condition that broadband communication can occasionally be achieved, the voice feature information of the corresponding target pronunciation object can be transmitted separately and once through a broadband channel (also called a broadband channel), and the complete voice of the corresponding target pronunciation object can be restored at the voice receiving end through a voice synthesis model.
[0077] Here, a multimode terminal may refer to a terminal device that supports multiple communication modes or network formats, for example, both broadband and narrowband formats. A multimode network may refer to a network that supports a variety of different communication technologies and protocols. These communication technologies and protocols may include wired and wireless communications, Internet Protocol, Ethernet, Wi-Fi, Bluetooth, LTE (Long-Term Evolution), 5G, etc. Multimode networks can provide more flexible, efficient and reliable communication services, suitable for different application scenarios and environments.
[0078] like Figure 5 As shown, when the receiving terminal has only narrowband format, but can interact with nearby broadband / dual-mode terminals (relay terminals) through near-field communication methods such as Wi-Fi, Bluetooth, StarFlash, NFC (NearField Communication), it can also be received by the nearby broadband / dual-mode terminal (relay terminal) through a broadband data channel (also called a broadband channel) and forwarded to the receiving terminal through near-field communication. Here, near-field communication is a short-range wireless communication technology that enables close-range data transmission and interaction between devices.
[0079] The narrowband module of the voice transmitting end sends the first voice data to the voice receiving end through the narrowband channel. The voice coding bit rate of the current mainstream narrowband standard is at the level of several kbps, such as 3.6 kbps of the DMR (Digital Mobile Radio) standard and 4.57 kbps of the Tetra standard, and the sound quality has been significantly distorted and reduced.
[0080] In order to adapt to communication scenarios with extremely limited bandwidth, such as high-orbit satellite communications, high-speed frequency-hopping secure communications, and underwater personnel communications, in this embodiment, a narrowband channel (also called a narrowband channel) can be used for narrowband communication. Narrowband communication refers to the system communicating within a narrower frequency band. Narrowband communication has a smaller bandwidth and can achieve reliable communication when spectrum resources are limited. Generally speaking, the bandwidth range of narrowband communication is between several kilohertz and several megahertz.
[0081] For extremely narrowband communication scenarios, such as Tiantong satellite terminal communication (uplink about 1.2kbps, downlink about 2.2kbps) and underwater personal communication terminals (about 1kbps or less), this application can achieve real-time voice call function with better sound quality.
[0082] Through this embodiment, by transmitting effective information of voice calls with extremely low bandwidth, it is possible to reduce the bandwidth requirement of voice calls, increase channel capacity, and improve sound quality.
[0083] As an optional solution, before acquiring the speech feature information of the target pronunciation object, the method further includes:
[0084] S31, when the priority of the target pronunciation object is not lower than the receiving object of the voice receiving end, determining that the target pronunciation object needs to obtain voice feature information during the voice communication process; and / or,
[0085] S32, when the target pronunciation object is in a group call and when the priority of the target pronunciation object in the object group to which the target pronunciation object belongs is not lower than a preset priority, determining that the target pronunciation object needs to obtain voice feature information during voice communication.
[0086] In a communication system, different priorities can be assigned to different users or user types so that the service quality of certain users can be prioritized when communication resources such as bandwidth are limited, and whether timbre is required can be determined based on identity information such as the priority of the target pronunciation object.
[0087] When the priority of the target pronunciation object is not lower than that of the receiving object of the voice receiving end, it is determined that the target pronunciation object needs to obtain voice feature information during the voice communication.
[0088] Correspondingly, when the priority of the target pronunciation object is lower than the receiving object of the voice receiving end, it is determined that the target pronunciation object does not need to obtain voice feature information during the voice communication process.
[0089] Group calling allows a user to talk to multiple people at the same time. In this call mode, the user who initiates the call can communicate with a predefined group of people at once, rather than having a one-to-one call with each person individually.
[0090] When the target pronunciation object is in a group call and the priority of the target pronunciation object in the object group to which the target pronunciation object belongs is not lower than a preset priority, it is determined that the target pronunciation object needs to obtain voice feature information during voice communication.
[0091] Through this embodiment, a user with a high priority may obtain more preferential resource allocation, more stable connection, lower delay and higher data transmission rate during a call.
[0092] As an optional solution, a set of target speech frames is subjected to prosodic feature extraction to obtain target prosodic information, including:
[0093] S41, extracting rhythm features of each target speech frame in a group of target speech frames according to a preset first rhythm element to obtain first rhythm information, wherein the first rhythm element includes at least one of the following: energy, pitch, tone, pause, stress, rhythm, and harmonic features;
[0094] S42, extracting rhythmic features of each pronunciation unit in a group of pronunciation units according to a preset second rhythmic element to obtain second rhythmic information, wherein the second rhythmic element includes at least one of the following: sound length, energy, pitch, tone, pause, stress, rhythm, and harmonic features.
[0095] Taking into account the rapid fluctuation of acoustic prosodic information such as sound length, energy, pitch, tone, pause, stress, and rhythm, in this embodiment, prosodic feature extraction can be performed based on each target speech frame in a group of target speech frames, for example, the energy, pitch and other information of each target speech frame can be extracted. Furthermore, considering that a pronunciation unit may correspond to one or more target speech frames, the prosodic elements related to the pronunciation unit can be extracted based on the complete pronunciation unit, for example, information such as sound length.
[0096] Through this embodiment, the completeness and accuracy of sub-semantic information can be improved by extracting acoustic prosody information in layers.
[0097] As an optional solution, encoding the target prosodic information and a set of pronunciation units, the method includes:
[0098] S51, encoding part of the target prosody information and / or a group of pronunciation units using a first encoding method, and encoding the other part using a second encoding method, wherein the first encoding method and the second encoding method are different encoding methods, and at least one of them is an encrypted encoding method.
[0099] The choice of encryption or non-encryption depends on the specific application requirements and the requirements for data security. Encryption refers to the encryption of data during the encoding process to protect the confidentiality and integrity of information. Encryption can prevent unauthorized access and eavesdropping during data transmission, but encryption usually requires more computing resources.
[0100] For example, in this embodiment, in the scenario of confidential communication, only a small amount of the basic part of the encoded content (i.e., the symbolized part of the basic audio unit) can be encrypted at the source to ensure the security of the overall encoded content. At the same time, the encrypted content is very small, thereby greatly accelerating the encryption speed, reducing latency, and improving concurrent encryption efficiency. Other prosody parameters and characteristic parameters of speech waveforms do not contain any semantic information, and will not disclose the conversation content if not encrypted. They can be encoded in an unencrypted manner or encrypted to achieve better confidentiality.
[0101] Through this embodiment, in the scenario of confidential communication, only a small basic part of the encoded content (i.e., the pronunciation unit) can be source encrypted to ensure the security of the overall encoded content. At the same time, the encrypted content is extremely small, thereby greatly speeding up the encryption speed, reducing latency, and improving concurrent encryption efficiency.
[0102] As an optional solution, the above method also includes:
[0103] S61, using a third encoding method to encode the voice signal to be transmitted to obtain second voice data to be transmitted, wherein the encoding rate of the second voice data is greater than the encoding rate of the first voice data;
[0104] S62, when the current communication bandwidth is greater than or equal to the specified bandwidth value, the second voice data is sent to the voice receiving end, or the second voice data is sent to the voice receiving end through the broadband channel based on the broadband module, so that the voice receiving end uses the first voice data and / or voice feature information to perform information compensation on the voice signal obtained by acoustic synthesis based on the second voice data.
[0105] Different encoding methods can have a significant impact on call quality, bandwidth usage, latency, and power consumption. When choosing an encoding method, you need to balance the specific application scenario, network conditions, device capabilities, and voice quality requirements. For example, in a bandwidth-constrained environment, you may need to use a lower bit rate encoding to save bandwidth, while in applications with higher voice quality requirements, you may need to use a higher bit rate encoding method, such as the third encoding method.
[0106] For services that transmit via narrowband, it can perform low bit rate transmission; for services that support broadband transmission, it can assist in transmitting voice feature information.
[0107] Optionally, extremely low bit rate coded data (e.g., first voice data) can be used as supplementary information for standard coded data of other communication formats (e.g., second voice data). In the case of poor network conditions, severe packet loss or bit errors, the characteristic of the basic audio unit lasting multiple voice frames can be utilized to infer the pronunciation of the lost voice frame, the characteristic parameters of the excitation function, and the characteristic parameters of the all-pole linear filter based on the first voice data and / or voice feature information in the unlost related voice frame, including the rhythmic pronunciation unit, and the amplitude, fundamental frequency, harmonics and other characteristic parameters of the carried waveform, based on the acoustic model, thereby performing information compensation and voice restoration for the voice frames lost in the standard coding of other communication formats to maintain the fluency of the call voice.
[0108] For example, in this embodiment, the symbols and rhythmic information of the pronunciation units, including the frequency domain feature parameters of the waveform such as amplitude, fundamental frequency, harmonics, etc., are combined with the data before and after the second voice data, and based on the acoustic model features, the lost frames of the second voice data are restored.
[0109] As an optional solution, Figure 6 is a flow chart of another optional voice data processing method according to an embodiment of the present application, which is applied to a voice receiving end, such as Figure 6 As shown, the process of the method may include the following steps:
[0110] S602, receiving first voice data sent by a voice sending end, wherein the first voice data contains encoded target prosody information and a group of encoded pronunciation units;
[0111] S604, decoding the encoded target prosody information and the encoded group of pronunciation units to obtain the target prosody information and the group of pronunciation units;
[0112] S606: acoustically synthesize a group of pronunciation units based on the target prosody information to obtain a first speech signal, and play the obtained first speech signal.
[0113] In this embodiment, combined with Figure 7, only the acoustic synthesis system is used at the voice receiving end, and the voice unit code is directly input into the acoustic synthesis system to restore high-quality voice with complete semantics (text content) and sub-semantics (rhythm and tone) information. The voice sending end sends the pronunciation unit code carrying the actual rhythmic information to the voice receiving end as the communication transmission content, and directly sends it to the acoustic synthesis system at the voice receiving end for voice synthesis, so as to obtain high-quality voice that completely retains the sub-semantic information, but does not contain the timbre (pronunciation frequency) characteristics of the speaker. The synthesized voice (voice signal to be played) output in this way can accurately convey the actual intention of the speaker, and then the obtained voice signal to be played can be played through the voice playing components such as the speaker at the voice receiving end.
[0114] Through this embodiment, the voice receiving end directly takes the rhythmic pronunciation unit as input, decodes and plays the voice based on splicing synthesis, parameter synthesis or end-to-end synthesis, does not involve any linguistic processing, and greatly reduces the requirements for processing resources under the premise of low encoding bit rate and high sound quality restoration of all semantic and sub-semantic effective information contained in the voice. It can be applied in embedded devices and can make real-time calls in narrowband communication modes such as satellite and cluster.
[0115] As an optional solution, the above method also includes:
[0116] S71, obtaining voice feature information of a target pronunciation object, wherein the target pronunciation object is a user who generates first voice data, and the voice feature information includes at least one of the following: timbre information, signal amplitude, signal fundamental frequency, and signal harmonics;
[0117] S72, acoustically synthesizing a group of pronunciation units based on the target prosody information and the speech feature information of the target pronunciation object to obtain a second speech signal, and playing the obtained second speech signal.
[0118] Combination Figure 8 , through the assistance of a one-time broadband communication, the speech feature information of the target pronunciation object at the speech sending end can be transmitted to the receiving terminal and added to the acoustic synthesis process, so that the speech receiving end can completely restore the human voice timbre of the target pronunciation object at the speech sending end, so that the final synthesized output speech is highly similar to the actual speech sent thereafter by the target pronunciation object at the narrowband speech sending end.
[0119] Here, the timbre information can be sent before the voice data is sent, or it can be sent at the same time as the voice data. This is not limited in the present embodiment. If the timbre information is sent after the voice data, after the voice receiving end receives the timbre information corresponding to the target pronunciation object, when it receives the voice data corresponding to the target pronunciation object sent by the voice sending end, it can perform acoustic synthesis with the previously received timbre information corresponding to the target pronunciation object.
[0120] Optionally, in this embodiment, the timbre information of multiple target pronunciation objects can be sent to the receiving end for pre-storage in advance when the communication bandwidth is greater than or equal to the specified bandwidth value. When the target pronunciation object is determined, the corresponding speech feature information and target rhythm information can be selected from the pre-stored timbre information to perform acoustic synthesis on a group of pronunciation units.
[0121] Optionally, in this embodiment, when both the voice transmitting end and the voice receiving end are dual-mode terminals including a narrowband module and a broadband module, the voice receiving end can receive the voice feature information of the target pronunciation object through the broadband module.
[0122] Optionally, in this embodiment, when the bandwidth is sufficient (for example, when the current communication bandwidth is greater than or equal to the specified bandwidth value), the voice sending end may send the voice feature information of the target pronunciation object to the voice receiving end.
[0123] Through this embodiment, a method for assisting in restoring timbre in extremely narrowband calls is proposed. Under narrowband communication conditions, all semantic and sub-semantic valid information contained in the speech can be effectively transmitted without retaining the timbre information; in a multi-mode terminal or multi-mode network, under the condition that broadband communication can occasionally be achieved, the timbre information of the corresponding caller can also be transmitted separately and at one time through the broadband channel, and the complete voice of the corresponding caller can be restored through a speech synthesis model on the output side.
[0124] As an optional solution, the speech feature information of the target pronunciation object is obtained, including:
[0125] S81, when both the voice receiving end and the voice sending end are dual-mode terminals, the broadband module of the voice receiving end receives voice feature information sent by the broadband module of the voice sending end through a broadband channel, wherein the dual-mode terminal includes a narrowband module and a broadband module;
[0126] S82, receiving voice feature information sent by a voice sending end or a relay terminal when the current communication bandwidth meets a preset condition.
[0127] Similar to the aforementioned embodiment, the voice feature information of the corresponding caller (target pronunciation object) can be transmitted separately and at one time through the broadband channel. When the voice receiving end is a dual-mode terminal, the broadband side of the dual-mode terminal is used for transmission and handed over to the narrowband side for use. When the voice receiving end is a single-mode terminal, that is, when the receiving terminal has only a narrowband format, but can interact with nearby broadband / dual-mode terminals through near-field communication methods such as WIFI, Bluetooth, NFC, and Star Flash, it can also be received by nearby broadband / dual-mode terminals through a broadband data channel and forwarded to the final receiving terminal through near-field communication methods.
[0128] Therefore, in the case where the voice receiving end is a dual-mode terminal, the voice characteristic information sent by the voice sending end through the broadband channel between the voice sending end and the voice receiving end is received;
[0129] When the voice receiving end is a single-mode terminal, the voice feature information is sent by the receiving relay terminal via near field communication. Here, the target timbre information is sent by the voice sending end to the relay terminal via the broadband channel between the voice sending end and the relay terminal.
[0130] When the bandwidth is sufficient, the voice transmitter can directly send the voice feature information to the voice receiver; when the bandwidth is insufficient, the relay terminal can send the voice feature information to the voice receiver through near-field communication, and there is no restriction on the transmission path between the relay terminal and the voice transmitter.
[0131] As an optional solution, before receiving the first voice data sent by the voice sending end, the method further includes:
[0132] S91, displaying timbre selection prompt information to the user, wherein the timbre selection prompt information is used to prompt the user of the voice receiving end to select whether to use the timbre information during the voice communication;
[0133] S92, obtaining a timbre selection result returned by the user in response to the timbre selection prompt information;
[0134] S93: When the timbre selection result indicates that timbre information needs to be obtained, a timbre information request is sent to the voice sending end, so that the voice sending end sends the timbre information according to the timbre information request.
[0135] The voice receiving end can display the tone selection prompt information on the terminal display screen, and the user can make the selection; the specific selection method (physical button or virtual button) is not limited.
[0136] The timbre selection prompt information is used to prompt the user at the voice receiving end to select whether to use the timbre information during the voice communication process.
[0137] Optionally, in this embodiment, acquiring the timbre information includes but is not limited to at least one of the following:
[0138] Acquire a timbre selection result returned by the user in response to the timbre selection prompt information, the timbre selection result including required timbre information, indication information of the type of timbre information used, or indication information that timbre information is not required;
[0139] When the timbre selection result is that the timbre information needs to be obtained, a timbre information request is sent to the voice sending end, so that the voice sending end sends the timbre information according to the timbre information request.
[0140] As an optional solution, the above method also includes:
[0141] S101, receiving second voice data, wherein the second voice data is obtained by encoding a voice signal to be transmitted by a voice transmitting end using a third encoding method, and the encoding bit rate of the second voice data is greater than the bit rate of the first voice data;
[0142] S102: When there are lost voice frames in the second voice data, use the first voice data to perform information compensation on a voice signal obtained by acoustic synthesis based on the second voice data.
[0143] During voice communication and data transmission, voice frames may be lost due to various reasons (such as network congestion, signal interference, hardware failure, etc.).
[0144] Lost voice frames refer to one or more frames of voice data that fail to reach the receiving end during the transmission process. A voice frame is a set of samples of a voice signal within a specific time interval, usually in milliseconds.
[0145] Information compensation refers to taking a series of measures to restore or simulate lost voice frames to reduce the impact on call quality. Information compensation here may refer to synthesizing lost frames of second voice data using first voice data. Here, the encoding method of the first voice data and the second voice data is similar to the above-mentioned embodiment and will not be described in detail here.
[0146] The following is an explanation of the method for processing speech data in the embodiment of the present application in conjunction with an optional example. In this optional example, the pronunciation unit includes a phoneme.
[0147] In the related art, there are some methods for processing speech data. In order to solve the problem that the code rate of waveform compression coding is high and speech conversion does not carry sub-semantic information, there are some application ideas that use ASR technology to identify speech content, transmit small data such as text, waveform template or synthesis model number, and then use TTS technology to restore speech. In the speech conversion scenario, there are also some ideas for adding prosodic information to improve the conversion effect. However, these ideas and methods have some problems and are not applicable to narrowband handheld devices and ultra-narrowband communication scenarios:
[0148] 1. The code rate of waveform-based narrowband speech signal transcoding must reach at least 2.4-9.6kbps, which cannot meet the satellite communication scenario of 1.2kbps;
[0149] 2. In addition to the acoustic model, speech recognition ASR and speech synthesis TTS technologies also require the use of linguistic models and dictionaries for linguistic audio processing and text recognition. This process has high requirements for terminal resources and cannot be implemented in real time on embedded devices with limited processor and RAM resources.
[0150] 3. If only the recognized text content is transmitted, important sub-speech information is missing; and adding text alignment and rhythm processing will place high demands on the pre-training of the models on both sides and the processing capabilities of the terminals on both sides; at the same time, it is impossible to truly express multilingual and non-linguistic speech information;
[0151] 4. If the restored timbre features need to be transmitted at the same time, higher requirements will be placed on the transmission bandwidth, or communication pre-training and pre-setting of the voice model of the call initiator will be required.
[0152] In order to at least partially solve the above technical problems, the present application proposes a method for transmitting call voice with low resources and low bit rate, which solves the problem of achieving real-time voice calls under extremely narrow bandwidth conditions. Figure 6 At the voice input end, the voice waveform is converted into corresponding pronunciation units such as phonemes, semiphones, syllables, etc. for encoding and transmission. No text recognition, linguistic processing or text-based acoustic processing is performed, and the waveform and timbre information is not recorded. While greatly reducing the encoding bit rate, the requirements for processing resources are greatly reduced, so that the solution can be applied to embedded devices; at the same time, the acoustic rhythmic information such as sound length, energy, pitch, tone, pause, stress, rhythm, etc. in the voice waveform is added to the encoding transmission at the voice input end, and all semantic and sub-semantic valid information contained in the voice is effectively transmitted under the premise of low encoding bit rate, and real-time communication can be realized in narrowband communication modes such as satellites and clusters; accordingly, the voice receiving end can decode and play the voice based on parameter synthesis or end-to-end synthesis, without involving any linguistic processing. Under the premise of low encoding bit rate and restoring all semantic and sub-semantic valid information contained in the voice, the requirements for processing resources are greatly reduced, and it can be applied in embedded devices and make real-time calls in narrowband communication modes such as satellites and clusters;
[0153] Optionally, the present application also proposes a solution for assisting in restoring the timbre in extremely narrowband calls. Figure 8 Under narrowband communication conditions, all semantic and sub-semantic valid information contained in the speech can be effectively transmitted without retaining the timbre information; in multi-mode terminals or multi-mode networks, under the condition that broadband communication can occasionally be achieved, the timbre information of the corresponding caller can also be transmitted separately through the broadband channel at one time, and the complete voice of the corresponding caller can be restored through the speech synthesis model on the output side.
[0154] Specifically, combined Fig. 9 The method for processing voice data in the present application may include the following steps:
[0155] Step 1: Voice collection and input;
[0156] Step 2, VAD processing;
[0157] Step 3, digital signal processing;
[0158] Step 4, sampling and framing;
[0159] Step 5, extracting state features, inputting the state features into the acoustic model to calculate the acoustic probability, and encoding the pronunciation unit; extracting the rhythm features, and encoding the rhythm information;
[0160] Step 6, fault tolerance and interleaving, interleaving can refer to rearranging or reorganizing data during data transmission to enhance fault tolerance and reduce the error rate in data transmission. By interleaving the data at the sending end before sending it, and then deinterleaving the data at the receiving end, errors and interference in the data transmission process can be effectively reduced. Fault tolerance and interleaving can be used to ensure the reliability and stability of data transmission;
[0161] Step 7, modulation and communication demodulation;
[0162] Step 8, decoding the pronunciation units carrying prosodic information;
[0163] Step 9, speech synthesis;
[0164] Step 10, high-quality speech containing semantic and sub-semantic information.
[0165] Compared with traditional methods, the present application has the advantages of semantic integrity, extremely low bit rate, low resources, and timbre restoration in narrowband voice transmission. It can realize effective real-time voice calls under extremely low bandwidth communication (such as satellite communication, narrowband communication) and embedded narrowband terminal conditions.
[0166] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.
[0167] According to another aspect of the embodiment of the present application, a terminal device for implementing the above-mentioned voice data processing method is also provided, which is applied to a voice sending end. The terminal device may include a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to perform the following steps through the computer program:
[0168] S1, obtaining a voice signal to be sent, dividing the voice signal to be sent into frames, and obtaining a group of target voice frames;
[0169] S2, converting a group of target speech frames into a group of pronunciation units, and extracting prosodic features from the group of target speech frames to obtain target prosodic information;
[0170] S3, encoding the target prosody information and a group of pronunciation units to obtain first speech data;
[0171] S4, sending the first voice data to the voice receiving end, so that the voice receiving end performs acoustic synthesis based on the first voice data.
[0172] Optionally, the processor may also be configured to execute any step in the above embodiments through a computer program, which has been explained and will not be repeated here.
[0173] According to another aspect of the embodiment of the present application, a terminal device for implementing the above-mentioned voice data processing method is also provided, which is applied to a voice receiving end. The terminal device may include a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to perform the following steps through the computer program:
[0174] S1, receiving first voice data sent by a voice sending end, wherein the first voice data contains encoded target prosody information and a group of encoded pronunciation units;
[0175] S2, decoding the encoded target prosody information and the encoded group of pronunciation units to obtain the target prosody information and the group of pronunciation units;
[0176] S3, acoustically synthesizing a group of pronunciation units based on the target prosody information to obtain a first speech signal, and playing the obtained first speech signal.
[0177] Optionally, the processor may also be configured to execute any step in the above embodiments through a computer program, which has been explained and will not be repeated here.
[0178] According to another aspect of the present application, a computer-readable storage medium is also provided, and a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the voice data processing method provided in the above-mentioned various optional implementations.
[0179] Optionally, in this embodiment, the computer-readable storage medium may be configured to store a computer program for performing the following steps:
[0180] S1, obtaining a voice signal to be sent, dividing the voice signal to be sent into frames, and obtaining a group of target voice frames;
[0181] S2, converting a group of target speech frames into a group of pronunciation units, and extracting prosodic features from the group of target speech frames to obtain target prosodic information;
[0182] S3, encoding the target prosody information and a group of pronunciation units to obtain first speech data;
[0183] S4, sending the first voice data to the voice receiving end, so that the voice receiving end performs acoustic synthesis based on the first voice data.
[0184] Optionally, in this embodiment, the computer-readable storage medium may be configured to store a computer program for performing the following steps:
[0185] S1, receiving first voice data sent by a voice sending end, wherein the first voice data contains encoded target prosody information and a group of encoded pronunciation units;
[0186] S2, decoding the encoded target prosody information and the encoded group of pronunciation units to obtain the target prosody information and the group of pronunciation units;
[0187] S3, acoustically synthesizing a group of pronunciation units based on the target prosody information to obtain a first speech signal, and playing the obtained first speech signal.
[0188] Optionally, in this embodiment, a person of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing hardware related to the terminal device through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.
[0189] According to another aspect of the present application, a computer program product is also provided, the computer program product includes a computer program / instruction, and the computer program / instruction contains a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part, and / or installed from a removable medium. When the computer program is executed by the central processing unit, various functions provided by the embodiment of the present application are executed.
[0190] The computer system includes a central processing unit (CPU), which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) or the program loaded from the storage part to the random access memory (RAM). In the random access memory, various programs and data required for system operation are also stored. The central processing unit, the read-only memory and the random access memory are connected to each other through a bus. The input / output interface (I / O interface) is also connected to the bus.
[0191] The following components are connected to the input / output interface: an input part including a keyboard, a mouse, etc.; an output part including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage part including a hard disk, etc.; and a communication part including a network interface card such as a local area network card, a modem, etc. The communication part performs communication processing via a network such as the Internet. A drive is also connected to the input / output interface as needed. Removable media, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., are installed on the drive as needed so that the computer program read therefrom is installed into the storage part as needed.
[0192] In particular, according to an embodiment of the present application, the process described in each method flow chart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer readable medium, and the computer program contains a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part, and / or installed from a removable medium. When the computer program is executed by a central processing unit, various functions defined in the system of the present application are executed.
[0193] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.
[0194] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application.
[0195] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.
[0196] In the several embodiments provided in the present application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.
[0197] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0198] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0199] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
Claims
1. A method for processing speech data, It is characterized in that Applied to the voice sending end, including: Acquire a voice signal to be sent, perform frame processing on the voice signal to be sent, and obtain a group of target voice frames; Converting the set of target speech frames into a set of pronunciation units, and extracting prosodic features from the set of target speech frames to obtain target prosodic information; Encoding the target prosody information and the group of pronunciation units to obtain first speech data; The first voice data is sent to a voice receiving end so that the voice receiving end performs acoustic synthesis based on the first voice data.
2. The method according to claim 1, It is characterized in that The sending the first voice data to the voice receiving end includes: The voice sending end detects the current communication bandwidth; When the current communication bandwidth is less than or equal to the specified bandwidth value, sending the first voice data to the voice receiving end; or, When the current communication bandwidth is greater than or equal to the specified bandwidth value, acquiring voice feature information of a target pronunciation object, wherein the target pronunciation object is a user who generates the first voice data, and the voice feature information includes at least one of the following: timbre information, signal amplitude, signal fundamental frequency, and signal harmonics; The first voice data and the voice feature information of the target pronunciation object are sent to the voice receiving end.
3. The method according to claim 1, It is characterized in that The voice sending end is a dual-mode terminal, including a narrowband module and a broadband module, and sends the first voice data to the voice receiving end. include: Acquire speech feature information of a target pronunciation object, and send the speech feature information to the speech receiving end through a broadband channel by a broadband module of the speech transmitting end, wherein the target pronunciation object is a user who generates the first speech data, and the speech feature information includes at least one of the following: timbre information, signal amplitude, signal fundamental frequency, and signal harmonics; The narrowband module of the voice sending end sends the first voice data to the voice receiving end through a narrowband channel.
4. The method according to claim 2 or 3, It is characterized in that Before acquiring the speech feature information of the target pronunciation object, the method further includes: When the priority of the target pronunciation object is not lower than that of the receiving object of the voice receiving end, determining that the target pronunciation object needs to obtain the voice feature information during voice communication; and / or, When the target pronunciation object is in a group call and the priority of the target pronunciation object in the object group to which the target pronunciation object belongs is not lower than a preset priority, it is determined that the target pronunciation object needs to obtain the voice feature information during voice communication.
5. The method according to claim 1, It is characterized in that Before the voice signal to be sent is subjected to frame processing to obtain a group of target voice frames, the method further includes: Performing voice activity detection on the initial voice signal collected by the voice collection component of the voice sending end to obtain a voice start time point and a voice end time point; The speech signal between the speech start time point and the speech end time point in the initial speech signal is determined as the speech signal to be sent.
6. The method according to claim 1, It is characterized in that The extracting prosodic features of the group of target speech frames to obtain target prosodic information includes: Extracting the rhythm feature of each target speech frame in the group of target speech frames according to a preset first rhythm element to obtain first rhythm information, wherein the first rhythm element includes at least one of the following: energy, pitch, tone, pause, stress, rhythm, and harmonic feature; According to the preset second prosodic element, the prosodic feature of each pronunciation unit in the group of pronunciation units is extracted to obtain the second prosodic information, wherein the second prosodic element includes at least one of the following: sound length, energy, pitch, tone, pause, stress, rhythm, and harmonic feature.
7. The method according to claim 1, It is characterized in that Encoding the target prosody information and the group of pronunciation units, the method comprising: Part of the target prosody information and / or the group of pronunciation units is encoded using a first encoding method, and the other part is encoded using a second encoding method, wherein the first encoding method and the second encoding method are different encoding methods, and at least one of them is an encrypted encoding method.
8. The method according to claim 2 or 3, It is characterized in that The method further comprises: Encoding the voice signal to be sent using a third encoding method to obtain second voice data to be sent, wherein the encoding bit rate of the second voice data is greater than the bit rate of the first voice data; When the current communication bandwidth is greater than or equal to the specified bandwidth value, the second voice data is sent to the voice receiving end, or the second voice data is sent to the voice receiving end through a broadband channel based on a broadband module, so that the voice receiving end uses the first voice data and / or the voice feature information to perform information compensation on the voice signal obtained by acoustic synthesis based on the second voice data.
9. A method for processing speech data, It is characterized in that Applied to the voice receiving end, including: Receiving first voice data sent by a voice transmitting end, wherein the first voice data contains encoded target prosody information and a group of encoded pronunciation units; Decoding the encoded target prosody information and the encoded group of pronunciation units to obtain the target prosody information and the group of pronunciation units; The group of pronunciation units are acoustically synthesized based on the target prosody information to obtain a first speech signal, and the obtained first speech signal is played.
10. The method according to claim 9, It is characterized in that The method further comprises: Acquire voice feature information of a target pronunciation object, wherein the target pronunciation object is a user who generates the first voice data, and the voice feature information includes at least one of the following: timbre information, signal amplitude, signal fundamental frequency, and signal harmonics; The group of pronunciation units is acoustically synthesized based on the target prosody information and the speech feature information of the target pronunciation object to obtain a second speech signal, and the obtained second speech signal is played.
11. The method according to claim 10, It is characterized in that The step of obtaining the speech feature information of the target pronunciation object includes: In the case where both the voice receiving end and the voice sending end are dual-mode terminals, the broadband module of the voice receiving end receives the voice feature information sent by the broadband module of the voice sending end through a broadband channel, wherein the dual-mode terminal includes a narrowband module and a broadband module; When the current communication bandwidth meets the preset condition, the voice feature information sent by the voice sending end or the relay terminal is received.
12. The method according to claim 10 or 11, It is characterized in that Before receiving the first voice data sent by the voice sending end, the method further includes: Displaying timbre selection prompt information to the user, wherein the timbre selection prompt information is used to prompt the user of the voice receiving end to select whether to use the timbre information during the voice communication; Acquiring a timbre selection result returned by the user in response to the timbre selection prompt information; When the timbre selection result is that the timbre information needs to be obtained, a timbre information request is sent to the voice sending end, so that the voice sending end sends the timbre information according to the timbre information request.
13. The method according to claim 9, It is characterized in that The method further comprises: receiving second voice data, wherein the second voice data is obtained by encoding the voice signal to be transmitted by the voice transmitting end using a third encoding method, and the encoding bit rate of the second voice data is greater than the bit rate of the first voice data; In the case where there are lost voice frames in the second voice data, the first voice data is used to perform information compensation on a voice signal obtained by acoustic synthesis based on the second voice data.
14. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, wherein the computer program implements the method of any one of claims 1 to 8 or 9 to 13 when executed by a processor.
15. An electronic device, It is characterized in that The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 8 or 9 to 13 when executing the computer program.
Citation Information
Cited By
Speech data processing method, storage medium, and electronic device
WO2026040810A1
