Chinese speech signal segmentation method, device and equipment and storage medium

Abstract

This disclosure provides a method, apparatus, device, and storage medium for segmenting Chinese speech signals. The method includes: sampling a target audio signal containing speech corresponding to a target Chinese text to obtain signal amplitudes corresponding to multiple sampling points; performing speech endpoint detection on the target audio data based on the signal amplitudes corresponding to the multiple sampling points to obtain multiple speech segments in the target audio data; determining the vowel position sequence corresponding to the target speech segment based on the formant energy of the speech signal in the target speech segment; determining syllable segmentation points and initial / final segmentation points of the target speech segment based on the signal amplitudes of sampling points between two adjacent vowel positions in the vowel position sequence and the initials and finals of the corresponding Chinese text segment; and segmenting the target speech segment based on the syllable segmentation points and initial / final segmentation points to obtain segmented speech primitives. This disclosure improves the accuracy of speech signal segmentation.

Description

Technical Field

[0001] This disclosure relates to artificial intelligence technology, speech recognition technology, speech synthesis technology, and in particular, a method, apparatus, device, and storage medium for segmenting Chinese speech signals. Background Technology

[0002] In speech recognition and speech synthesis technologies, large-scale training corpora are frequently required. Currently, many corpus resources are available, but these corpora need further processing. Both speech recognition and speech synthesis technologies process speech at the unit level of speech molecules. Therefore, accurate segmentation of initials and finals in Chinese speech to obtain accurate speech molecules is essential for building Chinese corpora. Achieving automatic and accurate initial and final segmentation is crucial for speech recognition and speech synthesis, as it directly affects the accuracy of speech recognition and the naturalness of speech synthesis.

[0003] Currently, various existing Chinese initial and final segmentation algorithms are based on multiple acoustic features for segmentation. However, continuous speech signals contain a lot of noise, and the methods used in related technologies are easily affected by noise when segmenting continuous speech, resulting in low accuracy of initial and final segmentation. Manual correction is required, and when the corpus is large, the workload of manual correction is very large, making it impossible to efficiently segment the initials and finals of continuous speech. Summary of the Invention

[0004] This disclosure provides a method, apparatus, device, and storage medium for segmenting Chinese speech signals, in order to at least partially solve the problems in related technologies.

[0005] One aspect of this disclosure provides a method for segmenting Chinese speech signals, comprising:

[0006] The target audio signal containing the speech corresponding to the target Chinese text is sampled to obtain the signal amplitude corresponding to multiple sampling points;

[0007] Based on the signal amplitude corresponding to the multiple sampling points, speech endpoint detection is performed on the target audio signal to obtain multiple speech segments in the target audio signal, wherein different speech segments correspond to different Chinese text segments, and the Chinese text segments belong to the target Chinese text;

[0008] Each of the multiple speech segments is taken as the target speech segment. Based on the formant energy of the speech signal in the target speech segment, the vowel position sequence corresponding to the target speech segment is determined. The vowel position sequence contains the vowel position of each vowel in the target speech segment.

[0009] Based on the signal amplitude of the sampling points between two adjacent vowel positions in the vowel position sequence and the initials and finals of the corresponding Chinese text segment of the target speech segment, the syllable segmentation points and initials and finals segmentation points of the target speech segment are determined.

[0010] Based on the syllable segmentation points and initial and final segmentation points of the target speech segment, the target speech segment is segmented to obtain segmented speech primitives, which are basic units for labeling the target audio signal.

[0011] In an exemplary embodiment, the step of performing speech endpoint detection on the target audio signal based on the signal amplitude corresponding to the plurality of sampling points to obtain multiple speech segments in the target audio signal includes:

[0012] Calculate the average value of the signal amplitude corresponding to the sampling point in each frame of the audio signal to obtain the frame average amplitude of each frame of the audio signal;

[0013] If the average amplitude of K consecutive audio frames is less than the amplitude threshold, the audio frames contained in the target audio signal are traversed starting from the end of the K consecutive audio frames, where K is a positive integer and K is a preset value. The amplitude threshold is calculated based on the average amplitude of the signals at the multiple sampling points.

[0014] During the traversal, if the noise energy of each audio signal in M ​​consecutive frames of audio signals is greater than the energy threshold, and the number retention rate of each audio signal in M ​​consecutive frames of audio signals is greater than the number retention rate threshold, then the M consecutive frames of audio signals are determined to be the speech segment, where M is a positive integer and M is a preset value. The energy threshold is calculated based on the average noise energy of the K consecutive frames of audio signals, and the number retention rate threshold is calculated based on the average number retention rate of the K consecutive frames of audio signals.

[0015] In an exemplary embodiment, determining the vowel position sequence corresponding to the target speech segment based on the formant energy of the speech signal in the target speech segment includes:

[0016] Windowing and framing are applied to the target speech segment to obtain a short-time signal;

[0017] The frequency domain signal is obtained by performing a Fourier transform on the short-time signal;

[0018] Calculate the formant energy of each frame of audio signal in the target speech segment based on the frequency domain signal;

[0019] The target speech segment is searched by traversing through the formant energy of each frame of audio signal.

[0020] During the traversal search process, if the formant energy of the consecutive S frames of audio signals is greater than the energy threshold, the sampling point in the consecutive S frames of audio signals is determined as the vowel position, where S is a preset positive integer.

[0021] In an exemplary embodiment, determining the syllable segmentation points and initial-final segmentation points of the target speech segment based on the signal amplitude of sampling points between two adjacent vowel positions in the vowel position sequence and the initials and finals of the corresponding Chinese text segment of the target speech segment includes:

[0022] Based on the initials and finals of the corresponding Chinese text segment of the target speech segment, determine the initial consonant type of the Pth syllable of the Pth vowel position point in the vowel position sequence, where P is a positive integer;

[0023] Based on the signal amplitude of each sampling point of the P-1 vowel position point in the vowel position sequence and the P vowel position point, calculate the second-order difference value of consecutive frames between the P-1 vowel position point and the P vowel position point.

[0024] Based on the initial and final vowel type of the Pth syllable and the second-order difference value, determine the initial and final vowel segmentation point and the syllable segmentation point between the P-1th vowel position point and the Pth vowel position point.

[0025] In an exemplary embodiment, the step of calculating the second-order difference value between the (P-1)th vowel position point and the Pth vowel position point based on the signal amplitude of each sampling point in the vowel position sequence includes:

[0026] Based on the signal amplitude of each sampling point at the P-1 vowel position point and the P vowel position point in the vowel position sequence, the speech segments at the P-1 vowel position point and the P vowel position point are framed to obtain multiple speech frames.

[0027] Perform correlation calculations on adjacent frames in the plurality of speech frames to obtain the correlation coefficients of adjacent frames;

[0028] The correlation coefficients of three consecutive speech frames are subjected to second-order difference to obtain the second-order difference values ​​of multiple consecutive three speech frames.

[0029] In an exemplary embodiment, the step of segmenting the speech segment at the (P-1)th vowel position point and the Pth vowel position point into multiple speech frames based on the amplitude values ​​of each sampling point at the Pth vowel position point in the vowel position sequence includes:

[0030] The search for the target zero-crossing point begins from the P-1th vowel position point. The target zero-crossing point refers to a sampling point where the signal amplitude crosses zero and the future trend of the signal amplitude changes towards a positive value.

[0031] During the search process, the i-th voice frame is determined with the i-th target zero-crossing point as the starting point, and the (i+1)-th target zero-crossing point is searched with the end point of the i-th voice frame as the starting point, where i is a positive integer.

[0032] In an exemplary embodiment, determining the initial-final segmentation point and syllable segmentation point between the (P-1)th vowel position point and the Pth vowel position point based on the initial consonant type of the Pth syllable and the second-order difference value includes:

[0033] When the initial consonant type of the Pth syllable is zero, the middle frame among the three consecutive speech frames corresponding to the maximum second-order difference value is determined as the syllable segmentation point.

[0034] When the initial consonant type of the P-th syllable is a voiceless initial consonant, the middle frame of the three consecutive speech frames corresponding to the largest second-order difference value is determined as the first segmentation point, and the middle frame of the three consecutive speech frames corresponding to the second largest second-order difference value is determined as the second segmentation point; the first segmentation point and the second segmentation point are determined as the syllable segmentation point, and the last segmentation point of the first segmentation point and the second segmentation point are determined as the initial and final segmentation point;

[0035] When the initial consonant type of the Pth syllable is a voiced initial consonant, the first T maximum values ​​of the second-order difference are obtained, and the intermediate frames between the first T maximum values ​​are combined to obtain multiple combined frame segments; based on the average zero-crossing rate of the multiple combined frame segments, the syllable segmentation point and the initial-final segmentation point are determined, where T is a preset positive integer.

[0036] In an exemplary embodiment, the plurality of combined frame segments include the minimum speech segment between each pair of adjacent points among the first T maximum value points and the combined segment obtained by combining the minimum speech segments;

[0037] The determination of the syllable segmentation point and the initial-final segmentation point based on the average zero-crossing rate of multiple combined frame segments includes:

[0038] Calculate the average zero-crossing rate of the combined segments and the average zero-crossing rate of each of the minimum speech segments;

[0039] If the deviation between the average zero-crossing rate of the combined segment and the average zero-crossing rate of the smallest speech segment contained in the combined segment is greater than a preset deviation threshold, the combined segment in the combined frame segment will be filtered out.

[0040] Determine the maximum average zero-crossing rate among the remaining combined frame segments, and determine the combined frame segment corresponding to the maximum average zero-crossing rate as the segmentation segment;

[0041] The left endpoint of the segment is determined as the syllable segmentation point, and the right endpoint of the segment is determined as the initial and final segmentation point.

[0042] Another aspect of the embodiments provides a Chinese speech signal segmentation device, comprising:

[0043] The sampling module is used to sample the target audio signal containing the speech corresponding to the target Chinese text, and obtain the signal amplitude corresponding to multiple sampling points;

[0044] The endpoint detection module is used to perform speech endpoint detection on the target audio signal based on the signal amplitude corresponding to the multiple sampling points, to obtain multiple speech segments in the target audio signal, wherein different speech segments correspond to different Chinese text segments, and the Chinese text segments belong to the target Chinese text;

[0045] The vowel detection module is used to determine the vowel position sequence corresponding to the target speech segment based on the formant energy of the speech signal in the target speech segment, with each of the plurality of speech segments as the target speech segment; the vowel position sequence contains the vowel position of each vowel in the target speech segment.

[0046] The segmentation point determination module is used to determine the syllable segmentation point and the initial and final segmentation point of the target speech segment based on the signal amplitude of the sampling point between two adjacent vowel positions in the vowel position sequence and the initial and final sounds of the corresponding Chinese text segment of the target speech segment.

[0047] The segmentation module is used to segment the target speech segment based on the syllable segmentation points and initial and final segmentation points to obtain segmented speech primitives, wherein the speech primitives are basic units for labeling the target audio signal.

[0048] In another aspect of this embodiment, an electronic device is provided, comprising:

[0049] Memory, used to store computer programs;

[0050] The processor is configured to execute a computer program stored in the memory, and when the computer program is executed, to implement the Chinese speech signal segmentation method described in any of the above embodiments.

[0051] In another aspect of this embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the Chinese speech signal segmentation method described in any of the above embodiments.

[0052] In this embodiment, for the target audio signal to be segmented, firstly, speech endpoint detection is performed on the target audio signal based on the audio amplitude values ​​of several sampling points in the target audio signal to obtain the effective speech segments contained in the target audio signal. Speech endpoint detection can eliminate noise or silent segments in the target audio signal, resulting in a more accurate speech signal. Then, taking advantage of the characteristic that the formant energy of vowels is higher than that of other sounds, the formant energy of the speech signal in the speech segment is calculated, thereby determining the position of each vowel contained in the speech segment based on the formant energy, and obtaining a vowel position sequence, which can improve the accuracy of vowel position extraction. Since the vowel position sequence has been determined, combined with the initials and finals corresponding to the Chinese text, the position segments where the initials and finals segmentation points and syllable segmentation points are located in the speech segment can be obtained. Then, combined with the amplitude values ​​of each sampling point in the position segment, the specific initials and finals segmentation points and syllable segmentation points are determined, and the target audio signal is segmented based on the initials and finals segmentation points and syllable segmentation points. In this embodiment, the target audio signal is first inspected for noise and silence to obtain speech segments. These segments are then segmented, which helps improve segmentation accuracy. Furthermore, since formant energy detection is less susceptible to noise interference, using formant energy to detect vowel position sequences improves vowel position detection accuracy. Based on vowel positions and the actual initials and finals of the Chinese text, the location segments of each segmentation point are determined. Determining the required segmentation points from these segments further improves the accuracy of initials, finals, and syllable segmentation, enhancing the accuracy of segmentation for continuous speech signals. This, in turn, improves the accuracy of speech primitive segmentation. Accurate annotation of speech primitives improves the accuracy of speech primitives in the corpus and the accuracy of feature extraction based on speech primitives. Moreover, the automatic initials and finals segmentation method improves the efficiency of segmenting large amounts of audio signals, contributing to the efficiency of building large-scale corpora. By improving the accuracy of speech primitives in the corpus, speech synthesis using these speech primitives can improve the naturalness of the synthesized speech; and speech recognition using these speech primitives can improve the accuracy of the speech recognition.

[0053] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0054] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0055] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein:

[0056] Figure 1 A flowchart illustrating a method for segmenting Chinese speech signals provided as an exemplary embodiment of this disclosure;

[0057] Figure 2 A flowchart illustrating the segmentation process of a Chinese speech signal provided as an exemplary embodiment of this disclosure;

[0058] Figure 3 A schematic diagram of segmentation points in a speech segment is provided for an exemplary embodiment of this disclosure;

[0059] Figure 4 A schematic diagram of the structure of a Chinese speech signal segmentation device provided as an exemplary embodiment of this disclosure;

[0060] Figure 5 This is a schematic diagram of the structure of an application embodiment of the electronic device disclosed herein. Detailed Implementation

[0061] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0062] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0063] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0064] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0065] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0066] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0067] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0068] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0069] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0070] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0071] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0072] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0073] Speech segmentation refers to the process of dividing a speech signal into basic speech units (i.e., speech primitives involved in the embodiments of this disclosure). Speech primitives can be syllables or smaller phonemes such as initials and finals. In the embodiments of this disclosure, initials and finals are used as speech primitives.

[0074] In speech synthesis technology, synthesis is performed on a unit-by-unit basis: speech primitives. The process of building a corpus for speech synthesis first requires segmenting and labeling the speech in the corpus. Segmentation yields speech primitives, which are then feature-annotated to create the final corpus for speech synthesis. Speech synthesis based on accurate speech primitives from the corpus improves the naturalness of the synthesized speech.

[0075] In speech recognition technology, recognition is performed on a unit-by-unit basis: speech primitives. In constructing a speech corpus for speech recognition, the speech in the corpus needs to be segmented to obtain the speech primitives contained in each speech instance. Subsequent feature extraction and annotation are then performed based on these primitives to construct the speech recognition corpus. Speech recognition based on accurate speech primitives from the corpus can improve the accuracy of speech recognition.

[0076] To address the problems in related technologies, this disclosure provides a method for segmenting Chinese speech signals. Figure 1 This is a flowchart illustrating an exemplary embodiment of the present disclosure providing a method for segmenting Chinese speech signals. The method provided in this embodiment can be used in electronic devices, such as... Figure 1 As shown, it includes the following steps:

[0077] Step 101: Sample the target audio signal containing the speech corresponding to the target Chinese text to obtain the signal amplitude corresponding to multiple sampling points.

[0078] The target audio signal is the speech signal to be segmented, which includes the speech corresponding to the target Chinese text. Optionally, the target Chinese text can be a sentence, paragraph, or article, and the target audio signal is the continuous speech signal corresponding to the target Chinese text. Optionally, the target audio signal can be a sample speech signal obtained during the corpus construction process.

[0079] Indicatively, the target Chinese text could be: "Emperor Renzong of the Northern Song Dynasty established Bianliang. The emperor was righteous and the ministers were virtuous, and the people were at peace. But the Western Xia invaded, causing the Central Plains to be engulfed in flames. In March, all things revive, and the Qin River Valley stretches for eight hundred miles, with rows of green willows."

[0080] The signal amplitude of an audio signal represents the intensity or energy of the corresponding sound. In this embodiment, the target audio signal is segmented based on its signal amplitude. First, the target audio signal is sampled at a preset sampling rate to obtain multiple sampling points and their corresponding signal amplitudes. Then, the target audio signal is framed using these sampling points. A preset unit frame length can be used to segment the target audio signal, combining the sampling points within the preset unit frame length into one frame. For example, the preset unit frame length can be 20ms, meaning one frame is 20ms.

[0081] Based on the above example, with a preset sampling rate of 22050Hz, sampling the target audio signal corresponding to the Chinese text in the above example yields 992250 sampling points. The total duration of the target audio signal is 45s, and each frame is 20ms, resulting in 2250 frames, each containing 441 sampling points.

[0082] Step 102: Based on the signal amplitude corresponding to multiple sampling points, perform speech endpoint detection on the target audio signal to obtain multiple speech segments in the target audio signal. Among them, different speech segments correspond to different Chinese text segments, and the Chinese text segments belong to the target Chinese text.

[0083] The target audio signal consists of continuous speech audio corresponding to the target Chinese text. It contains silent segments or noise, such as pauses within paragraphs. To improve detection accuracy, silent or noisy segments are first identified in the target audio signal. Speech endpoint detection refers to detecting the presence or absence of speech in an audio signal within a noisy environment. It is used to accurately detect the start and end positions of valid speech segments in the audio signal. Performing speech endpoint detection on the target audio signal yields the start and end positions of multiple speech segments. The target Chinese text contains multiple text segments; each speech segment obtained through speech endpoint detection corresponds to a different Chinese text segment—that is, one speech segment corresponds to one Chinese text segment.

[0084] In this embodiment of the disclosure, speech endpoint detection is performed based on the energy characteristics of the audio signal. After obtaining the signal amplitude corresponding to several sampling points, the energy of each frame of audio signal can be determined based on the signal amplitude of each sampling point, thereby segmenting the target audio signal to obtain multiple speech segments.

[0085] If the noise energy characteristics of multiple consecutive speech frames meet the speech decision criteria, the multiple consecutive speech frames can be identified as a speech segment.

[0086] To illustrate, based on the above example, after segmenting the target audio signal, we can obtain 13 speech segments: "Emperor Renzong of the Northern Song Dynasty established Bianliang", "The emperor is upright and the ministers are virtuous", "The people are safe and sound", "How hateful", "Western Xia", "Invasion", "Caused", "Central Plains", "Blazing with fire", "The weather in March", "All things revive", "Eight hundred miles of Qin Chuan", "Green willows in rows".

[0087] Step 103: Take each of the multiple speech segments as the target speech segment, and determine the vowel position sequence corresponding to the target speech segment based on the formant energy of the speech signal in the target speech segment. The vowel position sequence contains the vowel position of each vowel in the target speech segment.

[0088] After each speech segment is detected, a rough detection of the syllable position can be performed first. In one possible implementation, vowels in the speech segment can be detected. Among them, the energy of Chinese syllables mainly concentrates on vowels, and the formant frequencies of vowels mainly concentrate between 290 - 4000 Hz. Since the formant energy (FE) of vowels is much greater than the FE parameter of non-vowels and is less affected by noise, the vowel positions in the speech segment are detected based on the FE parameter to obtain a vowel position sequence.

[0089] During the detection process, for each speech segment, the corresponding vowel position sequence is detected. Each speech segment can be used as the target speech segment, and the FE parameter of the speech signal in the target speech segment is determined, so as to determine the vowel position sequence of the target speech segment based on the FE parameter of the target speech segment. The vowel position sequence contains the vowel positions of all vowels in the target speech segment.

[0090] Illustratively, when performing vowel detection on the speech segment corresponding to "民安康", the detected vowel position sequence contains the positions of the three vowels in "民安康".

[0091] Step 104, based on the signal amplitudes of the sampling points between two adjacent vowel positions in the vowel position sequence and the initials and finals of the Chinese text segment corresponding to the target speech segment, determine the syllable segmentation points and the initial-final segmentation points of the target speech segment.

[0092] After the vowel positions are determined, the syllable position and / or the initial-final position are included between two vowels. In one possible implementation, according to the Chinese text segment corresponding to the target speech segment, the pinyin of the Chinese text segment can be determined, and the initials and finals included in the pinyin corresponding to the Chinese text segment can be obtained, so as to determine the phoneme nodes and / or the initial-final segmentation points included in the two vowels. Illustratively, the vowel position sequence of "民安康" contains {P1, P2, P3}. Among them, between the starting sampling point P0 - P1 of the speech segment corresponding to "民安康", there is a segmentation point between the initial "m" and the final "in"; between P1 - P2, there is a syllable segmentation point between "min" and "an"; between P2 - P3, there is a syllable segmentation point between "an" and "kang" and a segmentation point between the initial "k" and the final "ang".

[0093] That is, according to the initials and finals of the Chinese text segment corresponding to the target speech segment, the number of segmentation points and the types of segmentation points included in the speech segment (vowel segment) between two vowel positions can be determined. Then, combined with the amplitudes of the sampling points included in the vowel segment, the vowel segment is searched to obtain the syllable segmentation points and / or the initial-final segmentation points therein, so as to improve the detection accuracy.

[0094] That is, in the Chinese speech signal segmentation method provided by the embodiments of the present disclosure, such as Figure 2As shown, it includes a speech endpoint detection process, a vowel segment detection process, and a vowel segmentation process. In the speech endpoint detection process, multiple speech segments corresponding to Chinese text segments are detected. Then, vowel segment detection is performed on the multiple speech segments to obtain the vowel segments contained in each speech segment. Based on the vowel segments, vowel segmentation and syllable segmentation are performed to obtain syllable segmentation points and vowel segmentation points.

[0095] Step 105: Based on the syllable segmentation points and initial and final segmentation points of the target speech segment, the target speech segment is segmented to obtain the segmented speech primitives. Speech primitives are the basic units for labeling the target audio signal.

[0096] Based on the syllable segmentation points and initial / final segmentation points contained in the target speech segment, the target speech segment can be segmented to obtain the speech primitives contained in the target speech segment. After segmenting all speech segments, all speech primitives contained in the target audio data can be obtained, enabling the segmentation of initials, finals, and syllables in continuous speech to obtain speech primitives in units of initials and finals.

[0097] Speech primitives can be used in the process of building a corpus to annotate the target audio signal in units of speech primitives, obtain sample audio signals for training, and store them in the corpus.

[0098] In this embodiment of the present disclosure, for the target audio signal to be segmented, firstly, speech endpoint detection is performed on the target audio signal based on the audio amplitude values ​​of several sampling points in the target audio signal to obtain the effective speech segments contained in the target audio signal. Speech endpoint detection can eliminate noise or silent segments in the target audio signal, resulting in a more accurate speech signal. Then, taking advantage of the characteristic that the formant energy of vowels is higher than that of other sounds, the formant energy of the speech signal in the speech segment is calculated, thereby determining the position of each vowel contained in the speech segment based on the formant energy, and obtaining a vowel position sequence, which can improve the accuracy of vowel position extraction. Since the vowel position sequence has been determined, combined with the initials and finals corresponding to the Chinese text, the position segments where the initials and finals segmentation points and syllable segmentation points are located in the speech segment can be obtained. Then, combined with the amplitude values ​​of each sampling point in the position segment, the specific initials and finals segmentation points and syllable segmentation points are determined, and the target audio signal is segmented based on the initials and finals segmentation points and syllable segmentation points. In this embodiment, the target audio signal is first inspected for noise and silence to obtain speech segments. These segments are then segmented, which helps improve segmentation accuracy. Furthermore, since formant energy detection is less susceptible to noise interference, using formant energy to detect vowel position sequences improves vowel position detection accuracy. Based on vowel positions and the actual initials and finals of the Chinese text, the location segments of each segmentation point are determined. Determining the required segmentation points from these segments further improves the accuracy of initials, finals, and syllable segmentation, enhancing the accuracy of segmentation for continuous speech signals. This, in turn, improves the accuracy of speech primitive segmentation. Accurate annotation of speech primitives improves the accuracy of speech primitives in the corpus and the accuracy of feature extraction based on speech primitives. Moreover, the automatic initials and finals segmentation method improves the efficiency of segmenting large amounts of audio signals, contributing to the efficiency of building large-scale corpora. By improving the accuracy of speech primitives in the corpus, speech synthesis using these speech primitives can improve the naturalness of the synthesized speech; and speech recognition using these speech primitives can improve the accuracy of the speech recognition.

[0099] In this embodiment of the disclosure, during the speech endpoint detection process, the noise energy of the audio is calculated based on the audio amplitude, and the target audio signal is segmented according to the changes in noise energy and signal amplitude to obtain speech segments. Specifically, the process of performing speech endpoint detection on the target audio signal based on the signal amplitude corresponding to several sampling points to obtain multiple speech segments from the target audio signal includes the following steps:

[0100] Step 1021: Calculate the average value of the signal amplitude corresponding to the sampling point in each frame of audio signal to obtain the frame average amplitude of each frame of audio signal.

[0101] First, the average amplitude of the sampling points of the audio signal in each frame can be calculated to obtain the average amplitude of each audio frame.

[0102] The average amplitude of the m-th frame is calculated as follows:

[0103]

[0104] Where N represents the number of sampling points in a frame, and S[N(m-1)+i] represents the signal amplitude of the i-th sampling point in the m-th frame. For illustration, referring to the example above, N is 441.

[0105] Step 1022: If the average amplitude of K consecutive audio frames is less than the amplitude threshold, traverse the audio frames contained in the target audio signal starting from the end of the K consecutive audio frames. Here, K is a positive integer and K is a preset value. The amplitude threshold is calculated based on the average amplitude of several sampling points.

[0106] The initial noise or silence segment is first determined based on the average amplitude of K consecutive audio frames. When the average amplitude of all K consecutive audio frames is less than the amplitude threshold, the search for a speech segment can begin from the end of the K consecutive audio frames.

[0107] The amplitude threshold can be calculated based on the average signal amplitude of all sampling points (i.e., the average total amplitude of the audio signal).

[0108] The average total amplitude of the target audio signal is calculated as follows:

[0109]

[0110] Where L is the total number of sampling points, and S(i) represents the signal amplitude of the i-th sampling point among all sampling points. For illustration, referring to the example above, L is 992250.

[0111] Optionally, the amplitude threshold can be 1 / 5 of the average total amplitude.

[0112] K is a preset positive integer. For illustration, K is 10. When A... m If all values ​​are less than 1 / 5A, the speech segment can be searched starting from the last sampling point in 10 consecutive frames.

[0113] Step 1023: During the traversal, if the noise energy of each audio signal in the M consecutive frames of audio signals is greater than the energy threshold, and the number retention rate of each audio signal in the M consecutive frames of audio signals is greater than the number retention rate threshold, then the M consecutive frames of audio signals are determined to be a speech segment, where M is a positive integer and M is a preset value. The energy threshold is calculated based on the average noise energy of the K consecutive frames of audio signals, and the number retention rate threshold is calculated based on the average number retention rate of the K consecutive frames.

[0114] When searching for a speech segment, the noise energy and number retention rate of the K consecutive audio frames can be used as a basis for the search. First, the average noise energy of the K consecutive audio frames can be calculated to determine the energy threshold required to judge the speech segment. The noise energy of each frame is calculated as follows:

[0115]

[0116] Where K is the number of sampling points in a frame, S(i) represents the signal amplitude of the i-th sampling point, and E n Let be the noise energy of the nth frame.

[0117] The average noise energy of K consecutive frames is calculated as follows:

[0118]

[0119] Furthermore, the retention rate threshold for judgment can be determined based on the average retention rate of K consecutive audio frames. The retention rate refers to the maximum number of sampling points in a frame where the sampled value continuously retains the same symbol, expressed as follows:

[0120]

[0121] Among them, L m This refers to the number of sampling points contained in the m-th segment of a frame that continuously maintains the same symbol, where each segment consists of consecutive sampling points with the same symbol and amplitude.

[0122] Each audio segment sequence satisfies the following conditions:

[0123]

[0124] Among them, S n,m (i) represents the signal amplitude of the i-th sampling point of the m-th segment of the n-th frame of the target audio signal.

[0125] The retention rate of each frame can be calculated using the above method. Therefore, the average retention rate of K consecutive audio frames is:

[0126]

[0127] In one possible implementation, an energy threshold can be defined as an integer multiple of the average noise energy, and a number retention rate threshold can be defined as an integer multiple of the average number retention rate. Illustratively, the energy threshold could be 5 times the average noise energy (5 * E_arverage), and the number retention rate threshold could be 5 times the average number retention rate (5 * Rks_arverage).

[0128] Starting from the last sampling point in the aforementioned K consecutive audio frames, the search begins. When the noise energy of each frame in the M consecutive audio frames is greater than the energy threshold, and the number retention rate of each frame in the M consecutive audio frames is greater than the number retention rate threshold, the M consecutive audio frames can be identified as a speech segment. That is, the first sampling point in the M consecutive audio frames is the starting point of the speech segment, and the last sampling point is the ending point of the speech segment.

[0129] For illustrative purposes, M can be 5. When the noise energy of 5 consecutive audio frames is greater than 5*E_arverage, and the number retention rate of 5 consecutive audio frames is greater than 5*Rks_arverage, the 5 consecutive audio frames are determined to be a speech segment.

[0130] This traversal method can be used to obtain several speech segments contained in the target audio signal.

[0131] In this embodiment of the disclosure, speech segments are detected by the noise energy of each frame of audio signal and the number retention rate of each frame of audio signal. The noise energy can reflect the intensity of the sound, and the number retention rate can reflect the change of the sound intensity over a period of time. Detecting speech based on these two factors can improve the accuracy of speech detection.

[0132] In the above embodiments, speech endpoint detection is performed on the target audio signal to obtain several speech segments. Then, the vowel positions in the speech segments are detected using FE parameters to roughly determine the phoneme nodes and the segments where the initial and final vowel segmentation points are located, and then segmentation is performed one by one. The process of determining the vowel position sequence corresponding to a speech segment based on the formant energy of the speech signal in the speech segment includes the following steps:

[0133] Step 1031: Window and frame the target speech segment to obtain a short-time signal.

[0134] First, the speech signal corresponding to the target speech segment is windowed and framed to obtain the short-time signal x(n) corresponding to the target speech segment, where x(n) represents the short-time signal corresponding to the nth frame.

[0135] Step 1032: Perform a Fourier transform on the short-time signal to obtain the frequency domain signal.

[0136] The short-time signal is a time-domain signal, which needs to be converted into a frequency-domain signal using Fourier transform. Optionally, Fast Fourier Transform (FFT) can be used to convert the short-time time-domain signal into a frequency-domain signal X(K).

[0137] Step 1033: Calculate the formant energy of each frame of audio signal in the target speech segment based on the frequency domain signal.

[0138] Next, the FE parameters of the frequency domain signal X(K) in the range of 290-4000Hz are calculated. The calculation method is shown below:

[0139]

[0140] Where FE is the formant energy corresponding to the nth frame of the audio signal.

[0141] Based on the above example, each frame of N = 441 time-domain data points can be zero-padded to 512 data points before undergoing an FFT transform to obtain the frequency-domain signal X(k). The highest frequency is 11025Hz, the sampling frequency is 22050Hz, and the frequency resolution is 43Hz. The short-time energy FE in the frequency range of 290–4000Hz is calculated as follows:

[0142]

[0143] Step 1034: Based on the formant energy of each frame of audio signal, perform a traversal search of the target speech segment.

[0144] Step 1035: During the traversal search process, if the formant energy of the consecutive S frames of audio signals is greater than the energy threshold, the sampling point in the consecutive S frames of audio signals is determined as the vowel position.

[0145] The formant energy corresponding to a vowel is greater than that corresponding to other sounds. Based on the formant energy of each frame of audio signal, a traversal search is performed in the speech segment. In one possible implementation, the formant energy of S consecutive frames of audio signal can be determined.

[0146] Furthermore, the energy threshold can be determined based on the maximum value among the FE parameters corresponding to all frames. Optionally, the energy threshold can be 1 / 8f. m f m This is the maximum value of the FE parameters corresponding to all frames in the speech segment.

[0147] If the energy field (FE) of S consecutive audio frames is greater than the energy threshold, the sampling points in the S consecutive audio frames are determined as vowel positions. Specifically, the target sampling point can be the first sampling point or an intermediate sampling point in the S consecutive frames.

[0148] If the condition that the FE of all consecutive S-frame audio signals is greater than the energy threshold is not met, and then the FE of all consecutive S-frame audio signals is found to be greater than the energy threshold again, the target sampling point of the newly found consecutive S-frame audio signals will be determined as the next vowel position.

[0149] In this embodiment of the disclosure, the accuracy of vowel position detection can be improved by using the formant energy of multiple consecutive frames to detect the vowel position in a speech segment.

[0150] After vowel detection, the initials and finals can be segmented according to the detected vowel position sequence. This process includes the following steps:

[0151] 1041. Based on the initials and finals of the Chinese text segment corresponding to the target speech segment, determine the initial type of the P-th syllable where the P-th vowel position point in the vowel position sequence is located.

[0152] The number of vowels can be obtained while detecting vowels, and then search in the corresponding target Chinese text to find the Chinese syllables corresponding to the speech segment. Each vowel position in the target speech segment corresponds to a syllable. In a possible implementation manner, the Chinese text can be converted into Chinese pinyin, and the specific initial type of the latter syllable in the syllable segmentation can be obtained. The initial types include zero initial, voiceless initial, and voiced initial.

[0153] Combined with the above example, for the speech segment corresponding to "min an kang" in the 13 speech segments, vowel segment detection will obtain 3 vowel positions, the number of vowels is 3, corresponding to the 3 syllables "min an kang", and the Chinese pinyin is "min an kang". The initial of the third syllable "kang" is "k".

[0154] 1042. Based on the signal amplitudes of each sampling point between the (P - 1)-th vowel position point and the P-th vowel position point in the vowel position sequence, calculate the second-order difference value of consecutive frames between the (P - 1)-th vowel position point and the P-th vowel position point.

[0155] After vowel segment detection, a vowel position sequence {P1, P2, P3,..., Pn} will be obtained. When performing initial-final segmentation, it is necessary to perform segmentation between position point 1 and position point P1, position point P1 and position point P2, position point P2 and position point P3,..., position point Pn - 1 and position point Pn.

[0156] Step a1. Based on the signal amplitudes of each sampling point between the (P - 1)-th vowel position point and the P-th vowel position point in the vowel position sequence, frame the speech segment between the (P - 1)-th vowel position point and the P-th vowel position point to obtain multiple speech frames.

[0157] During the segmentation process, reframe the speech segment between two position points. When framing the speech segment between the (P - 1)-th vowel position point and the P-th vowel position point, search for the target zero-crossing point starting from the (P - 1)-th vowel position point. The target zero-crossing point refers to the sampling point where the signal amplitude crosses zero and the future signal amplitude trend changes to a positive value.

[0158] During the search process, determine the i-th speech frame starting from the i-th target zero-crossing point, and search for the (i + 1)-th target zero-crossing point starting from the end point of the i-th speech frame, where i is a positive integer.

[0159] That is, starting from the sampling point corresponding to the P-1 vowel position, the search is made to find the target zero-crossing point as the starting point of the new frame. The target zero-crossing point refers to the sampling point where the signal amplitude crosses zero and the future trend of change is towards the positive value. That is, the amplitude of the sampling point before the target zero-crossing point is negative, and the amplitude of its own corresponding point is positive, and the amplitude of the next sampling point is also positive.

[0160] Using the target zero-crossing point as the starting point of the new frame, the corresponding speech frame is divided according to the preset frame length, which can be 20ms. Then, using the end sampling point of the divided speech frame as the starting point, the next target zero-crossing point is searched, and the next speech frame is divided with the next target zero-crossing point as the starting point.

[0161] Step a2: Perform correlation calculations on adjacent frames in multiple speech frames to obtain the correlation coefficients of adjacent frames.

[0162] After dividing the speech into multiple speech frames in the above manner, correlation operations can be performed on adjacent frames in the multiple speech frames. It should be noted that adjacent frames refer to two adjacent speech frames obtained according to the division order, and the sampling points in adjacent frames may not be continuous.

[0163] The correlation coefficient between adjacent frames is calculated as follows:

[0164]

[0165] Among them, S j (n) represents the signal amplitude corresponding to the nth sampling point in the jth frame.

[0166] Step a3: Perform second-order difference on the correlation coefficients corresponding to three consecutive speech frames to obtain multiple second-order difference values ​​for three consecutive frames.

[0167] In this embodiment of the disclosure, after performing correlation operations on consecutive frames, the second-order difference is taken to determine the segmentation points between syllables or initials and finals. The second-order difference is performed as follows:

[0168] G 2 (j)=|R(j+2)-R(j)| / H

[0169] Among them, G 2 (j) represents the second-order difference value between frame j and frame j+2.

[0170] and

[0171] By performing second-order difference on three consecutive speech frames respectively, multiple second-order difference values ​​for three consecutive frames can be obtained.

[0172] 1043. Based on the initial consonant type of the P-th syllable and the second-order difference value, determine the syllable-final segmentation point and the syllable segmentation point between the (P - 1)-th vowel position point and the P-th vowel position point.

[0173] After obtaining the second-order difference value, different discrimination methods will be used to determine the segmentation point according to the category of the initial consonant of the following syllable (zero initial, voiced initial, and voiceless initial).

[0174] Among them, the syllable segmentation point between the (P - 1)-th syllable and the P-th syllable is included between the (P - 1)-th vowel position point and the P-th vowel position point, and the discrimination method can be determined according to the initial consonant type of the following syllable (i.e., the P-th syllable). In the above example, after performing vowel detection on "min an kang", the vowel position sequence {P1, P2, P3} will be obtained. Among them, the syllable segmentation point between the syllables "min" and "an" is included between the P1 and P2 position points, and the method for determining the segmentation point can be determined according to the initial consonant type of the following syllable, that is, the initial consonant type of "an".

[0175] Optionally, when the initial consonant type of the P-th syllable is zero initial, the middle frame of the three consecutive frames corresponding to the maximum second-order difference value is determined as the syllable segmentation point.

[0176] When the following syllable is zero initial, there is a syllable segmentation point between the (P - 1)-th vowel position point and the P-th vowel position point, and there is no syllable-final segmentation point. Therefore, the maximum value among multiple second-order difference values can be obtained, and the syllable segmentation point is determined according to the maximum value. Among them, the middle frame of the three consecutive speech frames corresponding to the maximum second-order difference value can be determined as the syllable segmentation point. That is, when the second-order difference value of the cross-correlation between the k-th speech frame and the (k + 1)-th speech frame and the cross-correlation between the (k + 1)-th speech frame and the (k + 2)-th speech frame is the maximum value, then the (k + 1)-th speech frame is the syllable segmentation point.

[0177] Schematically, in combination with the above example, "an" is zero initial, and the middle frame of the second-order difference values of the three consecutive speech frames with the largest value between P1 and P2 can be determined as the syllable segmentation point.

[0178] Optionally, when the initial consonant type of the P-th syllable is voiceless initial, the middle frame of the three consecutive speech frames corresponding to the maximum second-order difference value is determined as the first segmentation point, and the middle frame of the three consecutive speech frames corresponding to the second-largest second-order difference value is determined as the second segmentation point; the previous segmentation point among the first segmentation point and the second segmentation point is determined as the syllable segmentation point, and the latter segmentation point among the first segmentation point and the second segmentation point is determined as the syllable-final segmentation point.

[0179] When the following syllable is a voiceless initial consonant, there are syllable segmentation points between the P-1 vowel position point and the P vowel position point, as well as initial-final segmentation points. The maximum and second-largest second-order difference values ​​can be obtained. The middle frame of the three consecutive speech frames corresponding to the maximum second-order difference value is one segmentation point, and the middle frame of the three consecutive speech frames corresponding to the maximum second-order difference value is another segmentation point. Since the syllable segmentation point is located before the initial-final segmentation point on the time axis, the segmentation point with the earlier time axis (the first and second segmentation points) can be determined as the syllable segmentation point, and the segmentation point with the later time axis can be determined as the initial-final segmentation point.

[0180] Indicative, in conjunction with the examples above, such as Figure 3 As shown, the syllable segmentation point between P2 and P3 includes the syllables "an" and "kang", as well as the initial and final segmentation point in "kang". The first segmentation point can be determined as the segmentation point between the syllables "an" and "kang", and the second segmentation point can be determined as the segmentation point between the initial consonant "k" and the final vowel "ang".

[0181] Optionally, if the initial consonant of the Pth syllable is a voiced initial consonant, the first T maximum values ​​in the second-order difference value are obtained, and the intermediate frames between the first T maximum values ​​are combined to obtain multiple combined frame segments; based on the mean zero-crossing rate of the multiple combined frame segments, the syllable segmentation point and the initial-final segmentation point are determined.

[0182] When the following syllable is a voiced initial consonant, the mean of the zero-crossing rate is used for judgment. Specifically, the first T maximum values ​​of the second-order difference can be obtained first, and speech frames are combined based on the first T maximum values ​​to obtain the combined frame segment.

[0183] The multiple combined frame segments include the minimum speech segments between each pair of adjacent points among the first T maximum value points, as well as combined segments obtained by combining the minimum speech segments. Among them, when combining the minimum speech segments, two consecutive segments can be combined, three consecutive segments can be combined, and so on, up to all minimum speech segment combinations.

[0184] Indicatively, when T is 5, the first 5 maximum values ​​in the second-order difference are obtained. Each pair of sampling points from these 5 maximum values ​​can be combined to form a minimum speech segment; therefore, the 5 maximum values ​​can be combined to form 4 minimum speech segments. These 4 minimum speech segments are then combined in pairs, in groups of three consecutive segments, or in groups of four. Pairwise combinations yield 3 new combined segments, and grouping in groups of three consecutive segments yields 2 new combined segments. In other words, the combined frame segments obtained from combining the 5 maximum values ​​contain 4 minimum speech segments, 3 combined segments from pairwise combinations, 2 combined segments from grouping in groups of three consecutive segments, and 4 combined segments, for a total of 10 combined segments.

[0185] Next, the combined segments can be filtered to obtain the segmented segments. This process includes the following steps:

[0186] Step b1: Calculate the average zero-crossing rate of the combined segments and the average zero-crossing rate of each minimum speech segment.

[0187] The zero-crossing rate for each frame is calculated as follows:

[0188]

[0189] Where n is the nth sampling point, sgn[] is the sign function, and w(n) is the window function. The window function is as follows:

[0190]

[0191] When calculating the average zero-crossing rate corresponding to the smallest speech segment, the frame zero-crossing rate corresponding to all frames contained in the smallest speech segment can be calculated, and then the frame zero-crossing rates can be averaged to obtain the average zero-crossing rate.

[0192] Schematic representation: When the smallest speech segment contains 3 frames, the average zero-crossing rate of the smallest speech segment is (Z). j1 +Z j2 +Z j3 ) / 3.

[0193] A combined segment is a combined speech segment obtained by continuously combining the smallest speech segments. The average zero-crossing rate of the combined segment can be calculated. The average zero-crossing rate of the combined segment is the ratio of the sum of the zero-crossing rates of all frames contained in the combined segment to the sum of the number of frames of all frames.

[0194] Schematic illustration: When a combined segment is obtained by combining two consecutive minimum speech segments, as in the example above, the combined segment contains 6 frames, and the average zero-crossing rate is (Z). j1 +Z j2 +Z j3 +Z j4 +Z j5 +Z j6 ) / 6.

[0195] Step b2: If the deviation between the average zero-crossing rate of the combined segment and the average zero-crossing rate of the smallest speech segment contained in the combined segment is greater than a preset deviation threshold, the combined segment in the combined frame segment is filtered out.

[0196] The combined segments are filtered based on the deviation between the average zero-crossing rate of the combined segments and the average zero-crossing rate of their corresponding smallest speech segments. The preset deviation thresholds may differ for different combination methods.

[0197] When performing deviation comparison, the average zero-crossing rate of the combined segment is compared with the average zero-crossing rate corresponding to each of the smallest speech segments it contains. When one of the deviations is greater than the deviation threshold, the combined segment can be filtered.

[0198] Illustratively, in combination with the above example, when the smallest speech segments are combined pairwise, the average zero-crossing rate of the combined segment obtained by combining two adjacent segments is compared with the average zero-crossing rate of each of the corresponding two smallest speech segments. When the deviation is greater than 30%, the combined segment is filtered;

[0199] When the smallest speech segments are combined in three consecutive segments, the average zero-crossing rate of the combined segment obtained by combining three adjacent segments is compared with the average zero-crossing rate of each of the corresponding three smallest speech segments. When the deviation is greater than 20%, the combined segment is filtered;

[0200] When the smallest speech segments are combined in four consecutive segments, the average zero-crossing rate of the combined segment obtained by combining four adjacent segments is compared with the average zero-crossing rate of each of the corresponding four smallest speech segments. When the deviation is greater than 10%, the combined segment is filtered.

[0201] Step b3, determine the maximum average zero-crossing rate among the average zero-crossing rates of the remaining combined frame segments, and determine the combined frame segment corresponding to the maximum average zero-crossing rate as the segmentation segment.

[0202] After filtering the combined segments in the combined frame segments, a set of possible segmentation segments is obtained. The maximum value of the average zero-crossing rate in the remaining combined frame segments can be determined, and the combined frame segment corresponding to the maximum average zero-crossing rate is determined as the segmentation segment. The remaining combined frame segments are the combined frame segments remaining after filtering the combined segments in the above manner.

[0203] Step b4, determine the left endpoint of the segmentation segment as the syllable segmentation point, and determine the right endpoint of the segmentation segment as the initial-final segmentation point.

[0204] Illustratively, in the speech segment "green willows in rows", the initial of the syllable "liu" is "l", which is a voiced initial. Finally, the left endpoint of the segmentation segment is determined as the segmentation point between the syllables "lv" and "liu", and the right endpoint is the segmentation point between the initial "l" and the final "iu".

[0205] In the embodiments of the present disclosure, according to the initial type of the syllable after segmentation, the discrimination method of the segmentation point is determined. For different types of initials, different judgment conditions are used to determine the syllable segmentation point and the initial-final segmentation point, improving the segmentation accuracy.

[0206] Figure 4 It is a structural block diagram of a segmentation device for Chinese speech signals provided by an exemplary embodiment of the present disclosure. As Figure 4 shown, the device includes:

[0207] The sampling module 401 is used to sample the target audio signal containing the speech corresponding to the target Chinese text to obtain the signal amplitude corresponding to multiple sampling points;

[0208] Endpoint detection module 402 is used to perform speech endpoint detection on the target audio signal based on the signal amplitude corresponding to the multiple sampling points, to obtain multiple speech segments in the target audio signal, wherein different speech segments correspond to different Chinese text segments, and the Chinese text segments belong to the target Chinese text;

[0209] The vowel detection module 403 is used to determine the vowel position sequence corresponding to the target speech segment based on the formant energy of the speech signal in the target speech segment, with each of the plurality of speech segments as the target speech segment; the vowel position sequence contains the vowel position of each vowel in the target speech segment.

[0210] The segmentation point determination module 404 is used to determine the syllable segmentation point and the initial and final segmentation point of the target speech segment based on the signal amplitude of the sampling point between two adjacent vowel positions in the vowel position sequence and the initial and final sounds of the corresponding Chinese text segment of the target speech segment.

[0211] The segmentation module 405 is used to segment the target speech segment based on the syllable segmentation points and the initial and final segmentation points to obtain segmented speech primitives, wherein the speech primitives are basic units for labeling the target audio signal.

[0212] In an exemplary embodiment, the endpoint detection module 402 is further configured to:

[0213] Calculate the average value of the signal amplitude corresponding to the sampling point in each frame of the audio signal to obtain the frame average amplitude of each frame of the audio signal;

[0214] If the average amplitude of K consecutive audio frames is less than the amplitude threshold, the audio frames contained in the target audio signal are traversed starting from the end of the K consecutive audio frames, where K is a positive integer and K is a preset value. The amplitude threshold is calculated based on the average amplitude of the signals at the multiple sampling points.

[0215] During the traversal, if the noise energy of each audio signal in M ​​consecutive frames of audio signals is greater than the energy threshold, and the number retention rate of each audio signal in M ​​consecutive frames of audio signals is greater than the number retention rate threshold, then the M consecutive frames of audio signals are determined to be the speech segment, where M is a positive integer and M is a preset value. The energy threshold is calculated based on the average noise energy of the K consecutive frames of audio signals, and the number retention rate threshold is calculated based on the average number retention rate of the K consecutive frames of audio signals.

[0216] In an exemplary embodiment, the vowel detection module 403 is further configured to:

[0217] Windowing and framing are applied to the target speech segment to obtain a short-time signal;

[0218] The frequency domain signal is obtained by performing a Fourier transform on the short-time signal;

[0219] Calculate the formant energy of each frame of audio signal in the target speech segment based on the frequency domain signal;

[0220] The target speech segment is searched by traversing through the formant energy of each frame of audio signal.

[0221] During the traversal search process, if the formant energy of the consecutive S frames of audio signals is greater than the energy threshold, the sampling point in the consecutive S frames of audio signals is determined as the vowel position, where S is a preset positive integer.

[0222] In an exemplary embodiment, the segmentation point determination module 404 is further configured to:

[0223] Based on the initials and finals of the corresponding Chinese text segment of the target speech segment, determine the initial consonant type of the Pth syllable of the Pth vowel position point in the vowel position sequence, where P is a positive integer;

[0224] Based on the signal amplitude of each sampling point of the P-1 vowel position point in the vowel position sequence and the P vowel position point, calculate the second-order difference value of consecutive frames between the P-1 vowel position point and the P vowel position point.

[0225] Based on the initial and final vowel type of the Pth syllable and the second-order difference value, determine the initial and final vowel segmentation point and the syllable segmentation point between the P-1th vowel position point and the Pth vowel position point.

[0226] In an exemplary embodiment, the segmentation point determination module 404 is further configured to:

[0227] Based on the signal amplitude of each sampling point at the P-1 vowel position point and the P vowel position point in the vowel position sequence, the speech segments at the P-1 vowel position point and the P vowel position point are framed to obtain multiple speech frames.

[0228] Perform correlation calculations on adjacent frames in the plurality of speech frames to obtain the correlation coefficients of adjacent frames;

[0229] The correlation coefficients of three consecutive speech frames are subjected to second-order difference to obtain the second-order difference values ​​of multiple consecutive three speech frames.

[0230] In an exemplary embodiment, the segmentation point determination module 404 is further configured to:

[0231] The search for the target zero-crossing point begins from the P-1th vowel position point. The target zero-crossing point refers to a sampling point where the signal amplitude crosses zero and the future trend of the signal amplitude changes towards a positive value.

[0232] During the search process, the i-th voice frame is determined with the i-th target zero-crossing point as the starting point, and the (i+1)-th target zero-crossing point is searched with the end point of the i-th voice frame as the starting point, where i is a positive integer.

[0233] In an exemplary embodiment, the segmentation point determination module 404 is further configured to:

[0234] When the initial consonant type of the Pth syllable is zero, the middle frame among the three consecutive speech frames corresponding to the maximum second-order difference value is determined as the syllable segmentation point.

[0235] When the initial consonant type of the P-th syllable is a voiceless initial consonant, the middle frame of the three consecutive speech frames corresponding to the largest second-order difference value is determined as the first segmentation point, and the middle frame of the three consecutive speech frames corresponding to the second largest second-order difference value is determined as the second segmentation point; the first segmentation point and the second segmentation point are determined as the syllable segmentation point, and the last segmentation point of the first segmentation point and the second segmentation point are determined as the initial and final segmentation point;

[0236] When the initial consonant type of the Pth syllable is a voiced initial consonant, the first T maximum values ​​of the second-order difference are obtained, and the intermediate frames between the first T maximum values ​​are combined to obtain multiple combined frame segments; based on the average zero-crossing rate of the multiple combined frame segments, the syllable segmentation point and the initial-final segmentation point are determined, where T is a preset positive integer.

[0237] In one exemplary implementation, the plurality of combined frame segments include the minimum speech segment between each pair of adjacent points among the first T maximum value points and the combined segment obtained by combining the minimum speech segments;

[0238] The segmentation point determination module 404 is further configured to:

[0239] Calculate the average zero-crossing rate of the combined segments and the average zero-crossing rate of each of the minimum speech segments;

[0240] If the deviation between the average zero-crossing rate of the combined segment and the average zero-crossing rate of the smallest speech segment contained in the combined segment is greater than a preset deviation threshold, the combined segment in the combined frame segment will be filtered out.

[0241] Determine the maximum average zero-crossing rate among the remaining combined frame segments, and determine the combined frame segment corresponding to the maximum average zero-crossing rate as the segmentation segment;

[0242] The left endpoint of the segment is determined as the syllable segmentation point, and the right endpoint of the segment is determined as the initial and final segmentation point.

[0243] The Chinese speech signal segmentation device of this disclosure corresponds to the Chinese speech signal segmentation method of this disclosure, and the relevant content can be referred to each other, which will not be repeated here. The beneficial technical effects of the Chinese speech signal segmentation device of this disclosure can be referred to the corresponding beneficial technical effects in the above-mentioned exemplary method section, which will not be repeated here.

[0244] In addition, this disclosure also provides an electronic device, including:

[0245] Memory, used to store computer programs;

[0246] A processor is configured to execute a computer program stored in the memory, wherein when the computer program is executed, it implements the Chinese speech signal segmentation method described in any of the above embodiments of the present disclosure.

[0247] Figure 5 This is a schematic diagram illustrating the structure of an application embodiment of the electronic device disclosed herein. Below, reference is made to… Figure 5 This describes an electronic device according to embodiments of the present disclosure. The electronic device may be either or both of a first device and a second device, or a standalone device independent of them, which may communicate with the first device and the second device to receive acquired input signals from them.

[0248] like Figure 5 As shown, the electronic device includes one or more processors and memory.

[0249] A processor can be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and can control other components in an electronic device to perform desired functions.

[0250] The memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor may execute the program instructions to implement the Chinese speech signal segmentation methods of the various embodiments of this disclosure described above and / or other desired functions.

[0251] In one example, the electronic device may also include input devices and output devices, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0252] In addition, the input device may include, for example, a keyboard, a mouse, etc.

[0253] This output device can output various information to the outside, including determined distance information, direction information, etc. The output device may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0254] Of course, for the sake of simplicity, Figure 5 Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0255] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the methods for segmenting Chinese speech signals according to various embodiments of this disclosure as described in the foregoing portions of this specification.

[0256] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0257] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the Chinese speech signal segmentation method according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0258] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0259] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as ROM, RAM, magnetic disk, or optical disk.

[0260] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0261] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0262] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0263] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0264] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0265] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0266] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A method for segmenting Chinese speech signals, characterized in that, include: The target audio signal containing the speech corresponding to the target Chinese text is sampled to obtain the signal amplitude corresponding to multiple sampling points; Based on the signal amplitude corresponding to the multiple sampling points, speech endpoint detection is performed on the target audio signal to obtain multiple speech segments in the target audio signal, wherein different speech segments correspond to different Chinese text segments, and the Chinese text segments belong to the target Chinese text; Each of the multiple speech segments is taken as the target speech segment. Based on the formant energy of the speech signal in the target speech segment, the vowel position sequence corresponding to the target speech segment is determined. The vowel position sequence contains the vowel position of each vowel in the target speech segment. Based on the signal amplitude of the sampling points between two adjacent vowel positions in the vowel position sequence and the initials and finals of the corresponding Chinese text segment of the target speech segment, the syllable segmentation points and initials and finals segmentation points of the target speech segment are determined. Based on the syllable segmentation points and initial and final segmentation points of the target speech segment, the target speech segment is segmented to obtain segmented speech primitives, which are basic units for labeling the target audio signal.

2. The method according to claim 1, characterized in that, The step of performing speech endpoint detection on the target audio signal based on the signal amplitude corresponding to the multiple sampling points to obtain multiple speech segments in the target audio signal includes: Calculate the average value of the signal amplitude corresponding to the sampling point in each frame of the audio signal to obtain the frame average amplitude of each frame of the audio signal; If the average amplitude of K consecutive audio frames is less than the amplitude threshold, the audio frames contained in the target audio signal are traversed starting from the end of the K consecutive audio frames, where K is a positive integer and K is a preset value. The amplitude threshold is calculated based on the average amplitude of the signals at the multiple sampling points. During the traversal, if the noise energy of each audio signal in M ​​consecutive frames of audio signals is greater than the energy threshold, and the number retention rate of each audio signal in M ​​consecutive frames of audio signals is greater than the number retention rate threshold, then the M consecutive frames of audio signals are determined to be the speech segment, where M is a positive integer and M is a preset value. The energy threshold is calculated based on the average noise energy of the K consecutive frames of audio signals, and the number retention rate threshold is calculated based on the average number retention rate of the K consecutive frames of audio signals. The number retention rate refers to the maximum number of sampling points in an audio signal whose corresponding signal amplitude continuously maintains the same symbol.

3. The method according to claim 1, characterized in that, Determining the vowel position sequence corresponding to the target speech segment based on the formant energy of the speech signal in the target speech segment includes: Windowing and framing are applied to the target speech segment to obtain a short-time signal; The frequency domain signal is obtained by performing a Fourier transform on the short-time signal; Calculate the formant energy of each frame of audio signal in the target speech segment based on the frequency domain signal; The target speech segment is searched by traversing through the formant energy of each frame of audio signal. During the traversal search process, if the formant energy of the consecutive S frames of audio signals is greater than the energy threshold, the sampling point in the consecutive S frames of audio signals is determined as the vowel position, where S is a preset positive integer.

4. The method according to any one of claims 1 to 3, characterized in that, The step of determining the syllable segmentation points and initial-final segmentation points of the target speech segment based on the signal amplitude of sampling points between two adjacent vowel positions in the vowel position sequence and the initials and finals of the corresponding Chinese text segment of the target speech segment includes: Based on the initials and finals of the corresponding Chinese text segment of the target speech segment, determine the initial consonant type of the Pth syllable of the Pth vowel position point in the vowel position sequence, where P is a positive integer; Based on the signal amplitude of each sampling point of the P-1 vowel position point in the vowel position sequence and the P vowel position point, calculate the second-order difference value of consecutive frames between the P-1 vowel position point and the P vowel position point. Based on the initial consonant type of the Pth syllable and the second-order difference value, determine the initial-final segmentation point and syllable segmentation point between the P-1th vowel position point and the Pth vowel position point; The step of calculating the second-order difference value between the (P-1)th vowel position point and the Pth vowel position point based on the signal amplitude of each sampling point in the vowel position sequence includes: Based on the signal amplitude of the (P-1)th vowel position point and each sampling point in the Pth vowel position point in the vowel position sequence, the speech segments in the (P-1)th vowel position point and the Pth vowel position point are framed to obtain multiple speech frames. Perform correlation calculations on adjacent frames in the plurality of speech frames to obtain the correlation coefficients of adjacent frames; The correlation coefficients of three consecutive speech frames are subjected to second-order difference to obtain the second-order difference values ​​of multiple consecutive three speech frames.

5. The method according to claim 4, characterized in that, The process involves segmenting the speech segment between the (P-1)th vowel position point and the Pth vowel position point based on the amplitude values ​​of each sampling point in the vowel position sequence, resulting in multiple speech frames, including: The search for the target zero-crossing point begins from the P-1th vowel position point. The target zero-crossing point refers to a sampling point where the signal amplitude crosses zero and the future trend of the signal amplitude changes towards a positive value. During the search process, the i-th voice frame is determined with the i-th target zero-crossing point as the starting point, and the (i+1)-th target zero-crossing point is searched with the end point of the i-th voice frame as the starting point, where i is a positive integer.

6. The method according to claim 4, characterized in that, The determination of the initial-final segmentation point and syllable segmentation point between the (P-1)th vowel position point and the Pth vowel position point based on the initial consonant type of the Pth syllable and the second-order difference value includes: When the initial consonant type of the Pth syllable is zero, the middle frame among the three consecutive speech frames corresponding to the maximum second-order difference value is determined as the syllable segmentation point. When the initial consonant type of the P-th syllable is a voiceless initial consonant, the middle frame of the three consecutive speech frames corresponding to the largest second-order difference value is determined as the first segmentation point, and the middle frame of the three consecutive speech frames corresponding to the second largest second-order difference value is determined as the second segmentation point; the first segmentation point and the second segmentation point are determined as the syllable segmentation point, and the last segmentation point of the first segmentation point and the second segmentation point are determined as the initial and final segmentation point; When the initial consonant type of the P-th syllable is a voiced initial consonant, the first T maximum values ​​of the second-order difference are obtained, and the intermediate frames between the first T maximum values ​​are combined to obtain multiple combined frame segments, where T is a preset positive integer. The multiple combined frame segments include the minimum speech segments between each pair of adjacent points of the first T maximum values ​​and the combined segments obtained by combining the minimum speech segments. The average zero-crossing rate of the combined segments and the average zero-crossing rate of each minimum speech segment are calculated. If the deviation between the average zero-crossing rate of the combined segments and the average zero-crossing rate of the minimum speech segments contained in the combined segments is greater than a preset deviation threshold, the combined segments in the combined frame segments are filtered. The maximum average zero-crossing rate of the remaining combined frame segments is determined, and the combined frame segment corresponding to the maximum average zero-crossing rate is determined as a segmentation segment. The left endpoint of the segmentation segment is determined as the syllable segmentation point, and the right endpoint of the segmentation segment is determined as the initial-final segmentation point.

7. A device for segmenting Chinese speech signals, characterized in that, include: The sampling module is used to sample the target audio signal containing the speech corresponding to the target Chinese text, and obtain the signal amplitude corresponding to multiple sampling points; The endpoint detection module is used to perform speech endpoint detection on the target audio signal based on the signal amplitude corresponding to the multiple sampling points, to obtain multiple speech segments in the target audio signal, wherein different speech segments correspond to different Chinese text segments, and the Chinese text segments belong to the target Chinese text; The vowel detection module is used to determine the vowel position sequence corresponding to the target speech segment based on the formant energy of the speech signal in the target speech segment, with each of the plurality of speech segments as the target speech segment; the vowel position sequence contains the vowel position of each vowel in the target speech segment. The segmentation point determination module is used to determine the syllable segmentation point and the initial and final segmentation point of the target speech segment based on the signal amplitude of the sampling point between two adjacent vowel positions in the vowel position sequence and the initial and final sounds of the corresponding Chinese text segment of the target speech segment. The segmentation module is used to segment the target speech segment based on the syllable segmentation points and initial and final segmentation points to obtain segmented speech primitives, wherein the speech primitives are basic units for labeling the target audio signal.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor is configured to execute a computer program stored in the memory, wherein when the computer program is executed, it implements the method for segmenting Chinese speech signals as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method for segmenting Chinese speech signals as described in any one of claims 1-6.

Images