Speech synthesis method suitable for off-line scene of Internet of Things

Abstract

The invention discloses a speech synthesis method suitable for an off-line scene of the Internet of Things, and relates to the technical field of speech data processing. The method comprises the following steps: acquiring an original human voice syllable sample and a to-be-synthesized target syllable sequence; calculating short-time root-mean-square energy of each frame of the original voice, and performing endpoint detection, framing windowing and FFT (Fast Fourier Transform) to obtain a frequency domain spectrum; extracting an amplitude higher than a preset multiple of the average amplitude in each frame of spectrum and a corresponding frequency as a characteristic wave peak, and packaging the characteristic wave peak into a compressed characteristic data packet to be stored locally; extracting a compressed data packet corresponding to the target syllable from the local, taking the characteristic wave crest of each frame as a node, complementing the frequency spectrum amplitude between nodes through linear interpolation, and adjusting the amplitude according to the short-time energy of each frame to obtain a time domain voice frame; and carrying out frame splicing and tone adjustment, and outputting high-quality human audio voice. According to the invention, while the low storage requirement of speech synthesis is met, the requirements of instantaneity and clear tone quality are met.

Description

Technical Field

[0001] This application relates to the field of speech data processing technology, and in particular to a speech synthesis method suitable for offline scenarios in the Internet of Things. Background Technology

[0002] Internet of Things (IoT) devices have been widely used in smart homes (such as smart switches and thermostats), industrial control (such as sensors and small controllers), and smart wearables (such as low-power wristbands). Offline voice interaction is one of the core functions. In most scenarios, devices do not have a stable network (such as industrial workshops and devices in remote areas), and need to complete voice synthesis independently without online computing power. In addition, devices generally have hardware limitations such as small storage capacity (mostly MB-level Flash) and low computing power (mostly low-power MCUs).

[0003] In the existing technology, current IoT offline speech synthesis technology is mainly divided into two categories. One is the pre-storage splicing scheme, which pre-records commonly used speech segments, compresses them using audio compression algorithms (such as MP3, AAC) or stores them uncompressed in the device, and retrieves the corresponding segments as needed during synthesis, generating complete speech through frame splicing and smoothing. The other is the traditional parametric synthesis scheme, which extracts parameters such as linear prediction coefficients (LPC), fundamental frequency (F0), and formants from the speech signal, stores the parameters instead of the complete audio, and reconstructs the speech waveform through LPC inverse filtering during synthesis.

[0004] However, pre-stored splicing solutions require storing a large number of complete voice segments, with each short sentence being about 5-10KB. Covering common scenarios requires several MB of storage, far exceeding the KB-level storage redundancy of some IoT devices (such as smart sensors). Furthermore, they can only synthesize combinations of pre-stored segments and cannot dynamically generate content that is not pre-stored. Traditional parametric synthesis solutions suffer from the difficulty of balancing storage and sound quality, and have high latency, failing to simultaneously meet the needs of IoT devices for low storage, immediacy, and clear sound quality.

[0005] Therefore, there is an urgent need for a speech synthesis method suitable for offline IoT scenarios, which can meet the requirements of low storage requirements, immediacy, and clear sound quality. Summary of the Invention

[0006] Therefore, it is necessary to provide a speech synthesis method suitable for offline IoT scenarios to address the aforementioned technical problems.

[0007] The present invention adopts the following technical solution: This invention provides a speech synthesis method suitable for offline scenarios in the Internet of Things (IoT), comprising: Obtain raw human speech syllable samples and speech syllable sequences to be synthesized, covering the target language categories; The short-time root-mean-square energy (SRMSE) of each frame of raw human speech syllable sample is determined. Endpoint detection, framing and windowing, and FFT transformation are performed sequentially on the raw human speech syllable samples to obtain the frequency domain speech spectrum of each frame. Amplitudes greater than 1.5 times the average amplitude of each frame's frequency domain speech spectrum are extracted and sorted in descending order. The top 5-8 amplitudes and their corresponding frequency values ​​are selected as feature peaks. The SRMSE and feature peaks are packaged into a compressed feature data package and imported into local Flash storage. The compressed feature data for each syllable includes multiple speech frames, the SRMSE corresponding to each speech frame, and multiple feature peaks corresponding to each speech frame. Extract compressed feature data packets corresponding to each syllable in the speech syllable sequence to be synthesized from local Flash storage; using the feature peaks of each frame in the compressed feature data packets corresponding to each syllable as nodes, complete the frequency domain speech spectrum amplitude between nodes through linear interpolation, and assign a fixed phase offset within the range of 0 to π / 2 to the completed frequency domain speech spectrum amplitude, while maintaining a linear phase transition between adjacent nodes, to obtain the completed frequency domain speech spectrum; adjust the completed frequency domain speech spectrum according to the short-time root mean square energy of each frame in the compressed feature data packets corresponding to each syllable and perform inverse FFT transformation to obtain the time domain speech frame of the speech syllable sequence to be synthesized; The temporal speech frames of the speech syllable sequence to be synthesized are sequentially spliced ​​and their pitches adjusted to obtain the human audio speech of the speech syllable sequence to be synthesized.

[0008] Optionally, the sampling frequency of the original human speech syllable samples is 16 kHz.

[0009] Optionally, endpoint detection is performed on the original human speech syllable samples, specifically including: Calculate the short-time root mean square energy of each 10ms time window in the original human speech syllable sample, distinguish between silent segments and speech segments in the original human speech syllable sample and segment them into multiple syllable signals; The number of times the amplitude signal crosses zero within each 10ms time window in the original human speech syllable sample is calculated, and voiceless and voiced segments are distinguished and divided into multiple syllable signals.

[0010] Optionally, the original human speech syllable samples are framed and windowed, specifically including: Set the frame shift to 1 / 2 of the frame length and then frame the segmented syllable signal. Apply a Hamming window to each syllable signal after framing, using the following formula: ; In the formula, For the first Hamming window for syllable signals The total frame length for each syllable signal.

[0011] Optionally, the original human speech syllable samples are subjected to FFT transformation, specifically including: Each frame of time-domain signal in each windowed syllable signal undergoes a 1024-point FFT transformation to convert the original human speech syllable sample in the time domain into a speech spectrum in the frequency domain; the 1024-point FFT can cover the 0kHz-8kHz frequency band at a sampling rate of 16kHz, with a spectral resolution of 15.6Hz. If the computing power of the IoT device cannot support a 1024-point FFT transformation, then the 1024-point FFT will be replaced with a 512-point FFT.

[0012] Optionally, the completed frequency domain speech spectrum is adjusted based on the short-time root mean square energy of each frame in the compressed feature data packet corresponding to each syllable, specifically including: The short-time root mean square energy of each frame in the compressed feature data packet corresponding to each syllable is multiplied by the amplitude of the completed frequency domain speech spectrum to adjust the completed frequency domain speech spectrum.

[0013] Optionally, the temporal speech frames of the syllable sequence to be synthesized are sequentially spliced ​​and their pitches adjusted, specifically including: Since the frame shift is half the frame length, there is a 50% overlap between adjacent speech frames of each syllable in the speech syllable sequence to be synthesized. The overlapping regions of the next frame and the overlapping regions of the previous frame are linearly weighted and superimposed to splice the temporal speech frames of the speech syllable sequence to be synthesized. Based on the user's preset speech synthesis requirements, the fundamental frequency of the spliced ​​time-domain speech frames is adjusted, including: The fundamental frequency of the spliced ​​time-domain speech frames was extracted using an autocorrelation algorithm; If the user's preset speech synthesis requirement is to raise the pitch, then the fundamental frequency of the speech frame is multiplied by 1.2 to 1.5. If the user's preset speech synthesis requirement is to lower the pitch, then the fundamental frequency of the speech frame is multiplied by 0.7 to 0.9.

[0014] This invention provides a speech synthesis device suitable for offline scenarios in the Internet of Things (IoT), comprising: The data acquisition module acquires raw human speech syllable samples and speech syllable sequences to be synthesized, covering the target language categories. The syllable extraction, compression, and storage module is used to determine the short-time root-mean-square energy (SRM) of each frame of original human speech syllable samples; it sequentially performs endpoint detection, framing and windowing, and FFT transformation on the original human speech syllable samples to obtain the frequency domain speech spectrum of each frame of original human speech syllable samples; it extracts amplitudes in each frame of frequency domain speech spectrum that are greater than 1.5 times the average amplitude of each frame of frequency domain speech spectrum and sorts them in descending order, selecting the top 5-8 amplitudes and the frequency values ​​corresponding to each amplitude as feature peaks; it packages the SRM and the feature peaks into a compressed feature data package and imports it into local Flash storage; wherein, the compressed feature data corresponding to each syllable includes multiple speech frames, the SRM corresponding to each speech frame, and multiple feature peaks corresponding to each speech frame; The speech synthesis module extracts compressed feature data packets corresponding to each syllable in the speech syllable sequence to be synthesized from local Flash storage; using the feature peaks of each frame in the compressed feature data packets corresponding to each syllable as nodes, it completes the frequency domain speech spectrum amplitude between nodes through linear interpolation, and assigns a fixed phase offset within the range of 0 to π / 2 to the completed frequency domain speech spectrum amplitude, while maintaining a linear phase transition between adjacent nodes, thus obtaining the completed frequency domain speech spectrum; based on the short-time root mean square energy of each frame in the compressed feature data packets corresponding to each syllable, it adjusts the completed frequency domain speech spectrum and performs an inverse FFT transform to obtain the time domain speech frames of the speech syllable sequence to be synthesized; the time domain speech frames of the speech syllable sequence to be synthesized are sequentially spliced ​​and pitch adjusted to obtain the human audio speech of the human audio speech sequence of the speech syllable sequence to be synthesized.

[0015] The present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the aforementioned speech synthesis method suitable for offline scenarios in the Internet of Things.

[0016] The present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the above-mentioned speech synthesis method suitable for offline scenarios of the Internet of Things.

[0017] The above-mentioned at least one technical solution adopted in this invention can achieve the following beneficial effects: In the speech synthesis method for offline IoT scenarios provided by this invention, a limited number of key feature peaks and frame energy from each frame of speech spectrum are extracted and stored, replacing the traditional approach of storing complete speech segments or a large number of linear prediction parameters. This compresses the storage amount of a single syllable, significantly reducing storage requirements compared to traditional pre-storage schemes (MB-level) and LPC schemes (hundreds of bytes), fundamentally solving the problem of extremely limited storage resources in IoT devices. The offline dynamic synthesis process based on syllable-level compressed feature data packets rapidly expands the spectrum through linear interpolation, calibrates the amplitude using frame energy, and then reconstructs the time-domain signal through inverse FFT transformation. This achieves the ability to dynamically generate speech based on arbitrary syllable sequences without pre-storing complete sentences, overcoming the limitations of poor flexibility in pre-storage schemes. The synthesis process is computationally simple, avoiding the complex matrix operations of traditional parameter synthesis schemes, and can achieve millisecond-level latency synthesis on low-power MCUs, meeting the real-time response requirements of IoT scenarios. By controlling the number of feature peaks and performing frame splicing smoothing and dynamic pitch adjustment during the reconstruction stage, this invention significantly reduces storage requirements while maintaining high clarity and naturalness in synthesized speech, effectively resolving the traditional contradiction between storage and sound quality in speech synthesis. This method does not rely on networks or cloud computing power, making it widely applicable to various resource-constrained IoT terminal devices without network access, achieving a comprehensive performance improvement characterized by low storage, high flexibility, low latency, and high clarity.

[0018] In summary, this invention addresses the prominent problems of existing IoT offline speech synthesis solutions, such as large storage requirements, poor flexibility, high synthesis latency, and difficulty in balancing sound quality and storage. By employing a collaborative compression and reconstruction mechanism of feature peaks and frame energy, it achieves high-quality, real-time speech synthesis with extremely low resource consumption. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0020] Figure 1 This is a flowchart illustrating a speech synthesis method suitable for offline IoT scenarios provided by the present invention.

[0021] Figure 2 The present invention provides an operation flowchart of a speech synthesis method suitable for offline scenarios in the Internet of Things. Figure 3 A schematic diagram of a speech synthesis device suitable for offline scenarios in the Internet of Things provided by the present invention; Figure 4 A computer device diagram for implementing a speech synthesis method suitable for offline scenarios in the Internet of Things, provided by the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments in the specification without creative effort are within the scope of protection of this application.

[0023] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0024] Figure 1 This is a schematic diagram of a speech synthesis method applicable to offline IoT scenarios according to the present invention, which specifically includes the following steps: S101: Obtain original human speech syllable samples and speech syllable sequences to be synthesized, covering the target language.

[0025] Specifically, the original speech time-series data covers original human speech syllable samples of commonly used syllables in the target language (such as 400 commonly used syllables in Mandarin Chinese, with each syllable recorded 3-5 times to obtain the best sample, with a sampling rate of 16kHz and 16-bit quantization).

[0026] S102: Determine the short-time root-mean-square energy of each frame in each syllable of the original human speech syllable sample; sequentially perform endpoint detection, framing and windowing, and FFT transformation on the original human speech syllable sample to obtain the frequency domain speech spectrum of each frame in each syllable of the original human speech syllable sample; extract the amplitudes in the frequency domain speech spectrum of all frames in each syllable that are greater than a preset multiple of the average amplitude of the frequency domain speech spectrum of each frame and sort them in descending order, and select a specified number of the top-ranked amplitudes and the frequency values ​​corresponding to each amplitude as feature peaks; and package the short-time root-mean-square energy and the feature peaks into a compressed feature data package corresponding to each syllable and import it into local Flash storage; wherein, the compressed feature data package corresponding to each syllable includes multiple speech frames, the short-time root-mean-square energy corresponding to each speech frame, and multiple feature peaks corresponding to each speech frame.

[0027] Optionally, endpoint detection is performed on the original human speech syllable samples, specifically including: calculating the short-time root mean square energy of each 10ms time window in the original human speech syllable samples, distinguishing between silent segments and speech segments in the original human speech syllable samples and segmenting them into multiple syllable signals; calculating the number of times the amplitude signal crosses zero within each 10ms time window in the original human speech syllable samples, distinguishing between unvoiced segments and voiced segments and segmenting them into multiple syllable signals.

[0028] Specifically, the short-time energy of the speech signal (root mean square energy per 10ms window) is calculated to distinguish between silence segments and speech segments; the zero-crossing rate (the number of times the amplitude signal crosses zero within each 10ms window) is calculated to help distinguish between unvoiced segments (high zero-crossing rate) and voiced segments (low zero-crossing rate).

[0029] Optionally, the original human speech syllable samples are framed and windowed, specifically including: shifting the frame to half the frame length, and framing the segmented syllable signals; applying a Hamming window to each framed syllable signal, with the formula: ; In the formula, For the first Hamming window for syllable signals The total frame length for each syllable signal.

[0030] Specifically, the signal of a single syllable is processed by framing, with the frame length set to 20ms (corresponding to 320 sampling points at a sampling rate of 16kHz) and the frame shift set to 10ms (160 sampling points) – the frame shift is half the frame length, to ensure the continuity of information between frames and avoid audio dropouts during synthesis.

[0031] Specifically, a Hamming window is applied to each frame of signal to reduce "spectral leakage" caused by inter-frame truncation and ensure the spectral accuracy of subsequent FFT transformations.

[0032] Optionally, the original human speech syllable samples are subjected to FFT transformation, specifically including: performing a 1024-point FFT transformation on each frame of the time-domain signal in each windowed syllable signal to convert the original human speech syllable samples in the time domain into a frequency-domain speech spectrum; wherein, the 1024-point FFT can cover the 0kHz-8kHz frequency band at a sampling rate of 16kHz, and the spectral resolution is 15.6Hz; if the computing power of the IoT device cannot support the 1024-point FFT transformation, then the 1024-point FFT is replaced with a 512-point FFT.

[0033] Specifically, FFT (time-domain to frequency-domain): A 1024-point FFT is performed on each frame of the windowed time-domain signal to convert the time-domain signal into a frequency-domain spectrum. The 1024-point FFT can cover the 0-8kHz frequency band at a sampling rate of 16kHz (the main energy of human speech is concentrated in 0-4kHz), with a spectral resolution of about 15.6Hz (8000Hz / 512, the FFT result takes the first half), which meets the requirements for speech feature extraction. If the computing power of the IoT device is extremely low, it can be replaced with a 512-point FFT (resolution of about 31.2Hz) without affecting the core feature extraction.

[0034] Specifically, feature peak extraction: Amplitude analysis is performed on the frequency domain spectrum (0-8kHz) after each frame of FFT to screen out candidate peaks with amplitudes greater than 1.5 times the average amplitude of the spectrum; the candidate peaks are sorted in descending order of amplitude, and the top 5-8 peaks (≤10, the specific number can be adjusted according to storage requirements) are selected. The frequency value (Hz) and amplitude (relative value) of each peak are recorded. This is the core feature of speech (reflecting key information such as formants and fundamental harmonics).

[0035] When extracting feature peaks, if the number of candidate peaks (peaks with amplitude greater than 1.5 times the average amplitude of the spectrum) is less than 10 (e.g., the unvoiced segment spectrum is relatively flat with few peaks), then all candidate peaks are directly selected (at least 3 are retained) to ensure that the core spectral information of the speech is not lost.

[0036] Frame energy extraction: Calculate the "short-time root mean square energy" of the time-domain signal for each frame, using the following formula: ; in, x ( n ) represents the intra-frame temporal sampling point, and N is the frame length; the frame energy reflects the intensity of the speech in that frame and is used for amplitude calibration during subsequent spectrum expansion.

[0037] The characteristic peaks (frequency + amplitude) of all frames corresponding to a single syllable, plus the energy of all frames, are packaged into a compressed characteristic data packet for that syllable. See Table 1 for a format example (taking the “bā” syllable with 10 frames as an example).

[0038] Table 1 Examples of Compressed Feature Data Packets The compressed feature data packet for a single syllable is approximately 170 bytes (10 frames × 8 peaks per frame × 2 parameters + 10 frames of energy), and is imported into the local Flash storage of the IoT device (e.g., 400 syllables require only 80KB of storage). To achieve compact storage, the "frequency" of the feature peaks is stored using 1 byte (quantization rule: 0-255 corresponds to 0-8000Hz, with a precision of 31.25Hz), the "amplitude" is stored using 1 byte (storing relative values ​​0-255), and the frame energy is also stored using 1 byte (storing relative values ​​0-255).

[0039] S103: Extract the compressed feature data packet corresponding to each syllable in the speech syllable sequence to be synthesized from the local Flash storage; take the feature peak of each frame in the compressed feature data packet corresponding to each syllable as the node, and complete the frequency domain speech spectrum amplitude between the nodes by linear interpolation, and assign a fixed phase offset in the range of 0 to π / 2 to the completed frequency domain speech spectrum amplitude, and keep the phase between adjacent nodes linearly transitioned to obtain the completed frequency domain speech spectrum; adjust the completed frequency domain speech spectrum according to the short time root mean square energy of each frame in the compressed feature data packet corresponding to each syllable and perform inverse FFT transformation to obtain the time domain speech frame of the speech syllable sequence to be synthesized.

[0040] Specifically, the minimum phase reconstruction method is used to recover phase information, which satisfies the causal stability characteristics of speech signals and adapts to the low computing power requirements of IoT devices. A fixed phase offset within the range of 0-π / 2 is assigned to the completed frequency domain amplitude spectrum, and the phase between adjacent characteristic peaks maintains a linear transition, ensuring that the time domain signal is undistorted after the inverse FFT transformation.

[0041] Optionally, the completed frequency domain speech spectrum is adjusted based on the short-time root-mean-square energy of each frame in the compressed feature data packet corresponding to each syllable. Specifically, this includes multiplying the short-time root-mean-square energy of each frame in the compressed feature data packet corresponding to each syllable with the amplitude of the completed frequency domain speech spectrum to adjust the completed frequency domain speech spectrum.

[0042] Specifically, IoT devices receive voice synthesis commands from users or systems (such as "turn on the living room lights" or "temperature 28℃"), and use the device's built-in "text-syllable mapping module" to parse the corresponding syllable sequence. Based on the parsed syllable sequence, the device retrieves the "compressed feature data packet" corresponding to each syllable (such as retrieving data packets for syllables like "dǎ", "kāi", and "kè") from local Flash memory. This process requires no network dependency and has a retrieval latency of ≤1ms.

[0043] The compressed feature data packets for each syllable are processed to restore the frequency domain spectrum of each frame: using the characteristic peak frequency of each frame as a node, the spectral amplitude between nodes is supplemented by "linear interpolation" (e.g., if peak 1 is 300Hz / 0.8 and peak 2 is 1050Hz / 0.6, then the amplitude between 300-1050Hz decreases from 0.8 to 0.6 according to a linear law); the overall spectral amplitude is adjusted according to the "frame energy" of the frame - if the frame energy is 0.7 (relative value), then all interpolated spectral amplitudes are multiplied by 0.7 to ensure that the spectral energy is consistent with the original speech frame and to avoid fluctuations in the synthesized speech volume.

[0044] S104: Sequentially perform frame splicing and pitch adjustment on the time-domain speech frames of the speech syllable sequence to be synthesized to obtain the human audio speech of the speech syllable sequence to be synthesized.

[0045] Optionally, the temporal speech frames of the speech syllable sequence to be synthesized are sequentially spliced ​​and pitch adjusted, specifically including: since the frame shift is 1 / 2 of the frame length, there is a 50% overlap between adjacent speech frames of each syllable in the speech syllable sequence to be synthesized; the overlap of the next frame is linearly weighted and superimposed with the overlap of the previous frame, and the temporal speech frames of the speech syllable sequence to be synthesized are spliced.

[0046] Based on the user's preset speech synthesis requirements, the fundamental frequency of the spliced ​​time-domain speech frame is adjusted, including: extracting the fundamental frequency of the spliced ​​time-domain speech frame using an autocorrelation algorithm; if the user's preset speech synthesis requirement is to raise the pitch, the fundamental frequency of the speech frame is multiplied by 1.2 to 1.5; if the user's preset speech synthesis requirement is to lower the pitch, the fundamental frequency of the speech frame is multiplied by 0.7 to 0.9.

[0047] Specifically, the base frequency adjustment factor needs to be limited to between 0.7 and 1.5 times. If the factor is less than 0.7 times, the synthesized speech will sound muffled and distorted; if the factor is greater than 1.5 times, the synthesized speech will sound sharp and distorted, neither of which can meet the needs of IoT voice interaction.

[0048] Specifically, an inverse FFT is performed on the frequency domain spectrum (1024 points) of each frame after expansion to convert the frequency domain signal back to the time domain speech frame. After the inverse FFT, the real part of the signal needs to be taken (to remove the imaginary part error) to obtain the time domain waveform.

[0049] Overlap-Add is used to splice all temporal speech frames: Since the frame shift is 10ms (1 / 2 of the frame length) during framing, there is a 50% overlap between adjacent frames; the overlap area of ​​the next frame is linearly weighted and superimposed with the overlap area of ​​the previous frame (the weight of the overlap area of ​​the previous frame decreases from 1 to 0, and increases from 0 to 1 in the next frame) to avoid the "clicking" noise generated by inter-frame splicing.

[0050] Adjust the fundamental frequency of the speech according to the synthesis requirements (such as "mild tone", "alarm tone", "different user tone preferences"): use autocorrelation method to extract the fundamental frequency of the current speech (voiced fundamental frequency range: male 80-150Hz, female 150-250Hz); adjust the fundamental frequency through "temporal stretching and compression algorithm": if the tone needs to be raised (such as alarm), multiply the fundamental frequency by 1.2-1.5 times; if the tone needs to be lowered (such as mild prompt), multiply by 0.7-0.9 times. Keep the speech duration unchanged during adjustment to avoid abnormal speech rate.

[0051] The final processed time-domain speech signal (16kHz sampling rate, 16-bit quantization) is output to the audio DAC module of the IoT device to drive the speaker and generate clear and recognizable simulated human speech.

[0052] In addition, this embodiment also provides a specific application scenario and a comparison with the effects of existing technical solutions to further illustrate the method of the present invention. See the flowchart of the operation of the present invention. Figure 2 .

[0053] Storage size comparison: This invention (8 peaks): covers 400 syllables of Chinese with a storage size of 80KB; Traditional LPC scheme (12 coefficients): covers 400 Chinese syllables with a storage requirement of 160KB, which is 50% less storage compared to the present invention; Pre-storage splicing scheme (500 short sentences): covers common speech scenarios with a storage requirement of 2.5-5MB, which is 97% lower than the storage requirement of this invention.

[0054] Synthetic delay comparison (test platform: ESP32MCU, 240MHz main frequency): This invention: The delay for synthesizing 10-syllable speech (approximately 2 seconds) is 10ms; Traditional LPC schemes have a latency of 215ms for synthesizing 10-syllable speech (approximately 2 seconds), which is 95% lower than the latency of this invention.

[0055] Audio quality test: The synthesized speech of 10 commonly used IoT commands (such as "turn on the air conditioner", "humidity 65%", "fault alarm") was recognized. Among them, the synthesized speech of this invention was clear, natural and distinguishable.

[0056] 2. Typical application scenario examples (1) Smart home scenario: When the device is in a network environment, after the smart switch receives the user's "turn on the light" command, it synthesizes the voice "the living room light is on" through this technology and provides real-time feedback on the operation result; (2) Industrial Internet of Things scenario: When the temperature sensor in the workshop detects "temperature 35℃ (exceeding the standard)", it can directly synthesize the voice "equipment A temperature exceeds the standard, please stop the machine for inspection" offline through this technology without the need for network connection, so as to promptly remind the staff to deal with the abnormality; (3) Smart wearable scenario: After receiving an offline text message, the low-power bracelet uses this technology to synthesize the voice of "Text message from a friend: How are you doing lately?", so that users can obtain the key content of the text message even when there is no network. (4) Agricultural Internet of Things scenario: When the soil moisture sensor in the greenhouse detects "soil moisture 20%" (drought threshold), the technology synthesizes the voice "the soil in the cucumber greenhouse is dry and needs watering" to guide the growers to carry out irrigation operations.

[0057] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this invention.

[0058] The above describes a speech synthesis method suitable for offline IoT scenarios, provided by one or more embodiments of the present invention. Based on the same idea, the present invention also provides a corresponding speech synthesis device suitable for offline IoT scenarios, such as... Figure 3 As shown.

[0059] Figure 3 This invention provides a schematic diagram of a speech synthesis device suitable for offline IoT scenarios. The device includes: Data acquisition module 301 is used to acquire original human speech syllable samples and speech syllable sequences to be synthesized, covering the target language. The syllable extraction, compression, and storage module 302 is used to determine the short-time root-mean-square energy of each frame of original human speech syllable samples; perform endpoint detection, framing and windowing, and FFT transformation on the original human speech syllable samples sequentially to obtain the frequency domain speech spectrum of each frame of original human speech syllable samples; extract amplitudes in each frame of frequency domain speech spectrum that are greater than 1.5 times the average amplitude of each frame of frequency domain speech spectrum and sort them in descending order, and select the top 5-8 amplitudes and the frequency values ​​corresponding to each amplitude as feature peaks; package the short-time root-mean-square energy and the feature peaks into a compressed feature data package and import it into local Flash storage; wherein, the compressed feature data corresponding to each syllable includes multiple speech frames, the short-time root-mean-square energy corresponding to each speech frame, and multiple feature peaks corresponding to each speech frame; The speech synthesis module 303 is used to extract compressed feature data packets corresponding to each syllable in the speech syllable sequence to be synthesized from local Flash storage; using the feature peaks of each frame in the compressed feature data packets corresponding to each syllable as nodes, the frequency domain speech spectrum amplitude between nodes is completed by linear interpolation, and a fixed phase offset within the range of 0 to π / 2 is assigned to the completed frequency domain speech spectrum amplitude, while the phase between adjacent nodes maintains a linear transition, to obtain the completed frequency domain speech spectrum; according to the short-time root mean square energy of each frame in the compressed feature data packets corresponding to each syllable, the completed frequency domain speech spectrum is adjusted and inverse FFT transformation is performed to obtain the time domain speech frame of the speech syllable sequence to be synthesized; the time domain speech frames of the speech syllable sequence to be synthesized are sequentially spliced ​​and pitch adjusted to obtain the human audio speech of the human audio speech sequence of the speech syllable sequence to be synthesized.

[0060] The present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the aforementioned speech synthesis method suitable for offline scenarios in the Internet of Things.

[0061] The present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the above-mentioned speech synthesis method suitable for offline scenarios of the Internet of Things.

[0062] Specific limitations regarding a speech synthesis device suitable for offline IoT scenarios can be found in the above-described limitations of a speech synthesis method suitable for offline IoT scenarios, and will not be repeated here. The modules in the aforementioned speech synthesis device suitable for offline IoT scenarios can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0063] The present invention also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 1 This paper presents a speech synthesis method suitable for offline scenarios in the Internet of Things (IoT).

[0064] The present invention also provides Figure 4 The schematic diagram of the computer device shown is as follows: Figure 4 As shown, at the hardware level, this computer device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then executes it to achieve the above. Figure 1 This paper presents a speech synthesis method suitable for offline scenarios in the Internet of Things (IoT).

[0065] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

Claims

1. A speech synthesis method suitable for offline scenarios in the Internet of Things, characterized in that, include: Obtain raw human speech syllable samples and speech syllable sequences to be synthesized, covering the target language categories; Determine the short-time root mean square energy of each frame of original human speech syllable sample; Endpoint detection, framing and windowing, and FFT transformation are performed sequentially on the original human speech syllable samples to obtain the frequency domain speech spectrum of each frame of original human speech syllable samples; the amplitudes of each frame of frequency domain speech spectrum that are greater than 1.5 times the average amplitude of each frame of frequency domain speech spectrum are extracted and sorted in descending order, and the top 5-8 amplitudes and the frequency values ​​corresponding to each amplitude are selected as feature peaks. The short-time root mean square energy and the feature peaks are packaged into a compressed feature data package and imported into local Flash storage; wherein, the compressed feature data corresponding to each syllable includes multiple speech frames, the short-time root mean square energy corresponding to each speech frame, and multiple feature peaks corresponding to each speech frame; Extract compressed feature data packets corresponding to each syllable in the speech syllable sequence to be synthesized from local Flash storage; using the feature peaks of each frame in the compressed feature data packets corresponding to each syllable as nodes, complete the frequency domain speech spectrum amplitude between nodes through linear interpolation, and assign a fixed phase offset within the range of 0 to π / 2 to the completed frequency domain speech spectrum amplitude, while maintaining a linear phase transition between adjacent nodes, to obtain the completed frequency domain speech spectrum; adjust the completed frequency domain speech spectrum according to the short-time root mean square energy of each frame in the compressed feature data packets corresponding to each syllable and perform inverse FFT transformation to obtain the time domain speech frame of the speech syllable sequence to be synthesized; The temporal speech frames of the speech syllable sequence to be synthesized are sequentially spliced ​​and their pitches adjusted to obtain the human audio speech of the speech syllable sequence to be synthesized.

2. The speech synthesis method for offline IoT scenarios as described in claim 1, characterized in that, The sampling frequency of the original human speech syllable samples is 16 kHz.

3. The speech synthesis method suitable for offline IoT scenarios as described in claim 1, characterized in that, Endpoint detection is performed on the original human speech syllable samples, specifically including: Calculate the short-time root mean square energy of each 10ms time window in the original human speech syllable sample, distinguish between silent segments and speech segments in the original human speech syllable sample and segment them into multiple syllable signals; The number of times the amplitude signal crosses zero within each 10ms time window in the original human speech syllable sample is calculated, and voiceless and voiced segments are distinguished and divided into multiple syllable signals.

4. The speech synthesis method suitable for offline IoT scenarios as described in claim 3, characterized in that, The original human speech syllable samples are framed and windowed, specifically including: Set the frame shift to 1 / 2 of the frame length and then frame the segmented syllable signal. Apply a Hamming window to each syllable signal after framing, using the following formula: ； In the formula, For the first Hamming window for syllable signals The total frame length for each syllable signal.

5. A speech synthesis method suitable for offline IoT scenarios as described in claim 2 or 4, characterized in that, Performing an FFT transform on the original human speech syllable samples specifically includes: Each frame of time-domain signal in each windowed syllable signal undergoes a 1024-point FFT transformation to convert the original human speech syllable sample in the time domain into a speech spectrum in the frequency domain; the 1024-point FFT can cover the 0kHz-8kHz frequency band at a sampling rate of 16kHz, with a spectral resolution of 15.6Hz. If the computing power of the IoT device cannot support a 1024-point FFT transformation, then the 1024-point FFT will be replaced with a 512-point FFT.

6. The speech synthesis method suitable for offline IoT scenarios as described in claim 1, characterized in that, The step of adjusting the completed frequency domain speech spectrum based on the short-time root mean square energy of each frame in the compressed feature data packet corresponding to each syllable specifically includes: The short-time root mean square energy of each frame in the compressed feature data packet corresponding to each syllable is multiplied by the amplitude of the completed frequency domain speech spectrum to adjust the completed frequency domain speech spectrum.

7. The speech synthesis method suitable for offline IoT scenarios as described in claim 4, characterized in that, The process of sequentially splicing frames and adjusting the pitch of the temporal speech frames of the speech syllable sequence to be synthesized specifically includes: Since the frame shift is half the frame length, there is a 50% overlap between adjacent speech frames of each syllable in the speech syllable sequence to be synthesized. The overlapping regions of the next frame and the overlapping regions of the previous frame are linearly weighted and superimposed to splice the temporal speech frames of the speech syllable sequence to be synthesized. Based on the user's preset speech synthesis requirements, the fundamental frequency of the spliced ​​time-domain speech frames is adjusted, including: The fundamental frequency of the spliced ​​time-domain speech frames was extracted using an autocorrelation algorithm; If the user's preset speech synthesis requirement is to raise the pitch, then the fundamental frequency of the speech frame is multiplied by 1.2 to 1.

5. If the user's preset speech synthesis requirement is to lower the pitch, then the fundamental frequency of the speech frame is multiplied by 0.7 to 0.9.

Images