Phoneme classification-based speech alignment methods, devices, equipment, and media

Abstract

Embodiments of the present application disclose a phoneme classification-based speech alignment method and device, an electronic device, and a storage medium. In the speech alignment method, the to-be-processed audio data is input into a pre-trained phoneme classification model to obtain a first phoneme segmentation result; speech recognition is performed on the to-be-processed audio data to obtain a speech recognition text, and phoneme segmentation is performed on the speech recognition text to obtain a second phoneme segmentation result; and the first phoneme segmentation result and the second phoneme segmentation result are solved by path to obtain an alignment result. By accurately classifying the pronunciation state and pronunciation details of the to-be-processed audio data, and by forcibly aligning the two phoneme segmentation results by path solving, and by taking the pause symbol and the silence symbol as constraints for the speaking interval and the continuous same phonemes during alignment, accurate alignment of the two phoneme segmentation results can be achieved, and the operation accuracy can be significantly improved when the audio and video are edited according to the speech content.

Description

Technical Field

[0001] This application relates to the field of multimedia data processing technology, and in particular to a speech alignment method, apparatus, device, and medium based on phoneme classification. Background Technology

[0002] Traditional audio and video editing software modifies the audio and video by manipulating the timeline. This process often requires repeated viewing and / or listening to confirm the time segments to be edited, which is time-consuming and labor-intensive. To improve the efficiency of audio and video editing, a method for rapid editing using forced speech alignment technology has been developed. Specifically, the speech content in the audio and video is transcribed into text using ASR (Automatic Speech Recognition), and the timestamp information (start and end times in the audio and video) of each text unit (Chinese character) is obtained through a forced alignment model, thereby enabling rapid editing of audio and video by manipulating the text.

[0003] When the inventors used existing voice forced alignment technology for audio and video editing, they found that the alignment accuracy at the character level was poor in the voice forced alignment stage, resulting in an inaccurate overall alignment result. Consequently, the operational precision was not high when editing audio and video based on voice content. Summary of the Invention

[0004] This invention provides a speech alignment method, apparatus, device, and medium based on phoneme classification to solve the technical problem that existing speech alignment results are not accurate enough, resulting in low operational precision when editing audio and video based on speech content.

[0005] In a first aspect, embodiments of this application provide a speech alignment method based on phoneme classification, the speech alignment method based on phoneme classification including:

[0006] The audio data to be processed is input into a pre-trained phoneme classification model to obtain the first phoneme segmentation result;

[0007] The audio data to be processed is subjected to speech recognition to obtain speech recognition text, and the speech recognition text is subjected to phoneme segmentation to obtain the second phoneme segmentation result;

[0008] The path is solved by combining the first phoneme segmentation result and the second phoneme segmentation result to obtain the alignment result;

[0009] The first and second phoneme splitting results include a separator and a silence symbol. The separator is located after the last phoneme of each word in the audio data to be processed, and the silence symbol is located in the silent pauses between words in the audio data to be processed, as well as the silent parts at the beginning and end.

[0010] Secondly, embodiments of this application also provide a speech alignment device based on phoneme classification, the speech alignment device based on phoneme classification comprising:

[0011] The first splitting unit is used to input the audio data to be processed into a pre-trained phoneme classification model to obtain the first phoneme splitting result;

[0012] The second splitting unit is used to perform speech recognition on the audio data to be processed to obtain speech recognition text, and to perform phoneme splitting on the speech recognition text to obtain the second phoneme splitting result.

[0013] The phoneme alignment unit is used to perform path solving on the first phoneme splitting result and the second phoneme splitting result to obtain the alignment result.

[0014] The first and second phoneme splitting results include a separator and a silence symbol. The separator is located after the last phoneme of each word in the audio data to be processed, and the silence symbol is located in the silent pauses between words in the audio data to be processed, as well as the silent parts at the beginning and end.

[0015] Thirdly, embodiments of this application also provide an electronic device, which includes:

[0016] One or more processors;

[0017] Memory, used to store one or more computer programs;

[0018] When one or more computer programs are executed by one or more processors, electronic devices implement a speech alignment method based on phoneme classification, as described in the first aspect.

[0019] Fourthly, embodiments of this application also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the phoneme-based speech alignment method as described in the first aspect.

[0020] The aforementioned speech alignment method, apparatus, device, and medium based on phoneme classification, wherein the method involves inputting the audio data to be processed into a pre-trained phoneme classification model to obtain a first phoneme segmentation result; performing speech recognition on the audio data to be processed to obtain speech recognition text, performing phoneme segmentation on the speech recognition text to obtain a second phoneme segmentation result; and performing path solving on the first and second phoneme segmentation results to obtain an alignment result. The first and second phoneme segmentation results include a separator symbol and a silence symbol. The separator symbol is located after the last phoneme corresponding to each word in the audio data to be processed, and the silence symbol is located in the silent pauses between words in the audio data to be processed, as well as the silent parts at the beginning and end. By accurately classifying the pronunciation state and pronunciation details of the audio data to be processed, two phoneme decomposition results are obtained. The two phoneme decomposition results are then forced to align through path solving. Separators and silence symbols are used as constraints on speech intervals and consecutive identical phonemes during alignment. This avoids disorder of multiple consecutive identical phonemes and silence periods when aligning the two phoneme decomposition results, thus achieving accurate alignment of the two phoneme decomposition results. This can significantly improve the operational accuracy when editing audio and video through voice content. Attached Figure Description

[0021] Figure 1 A flowchart illustrating a speech alignment method based on phoneme classification, provided for an embodiment of this application.

[0022] Figure 2 This is a schematic diagram illustrating the adjustment of sample phoneme labels for valid sample audio data, as provided in an embodiment of this application.

[0023] Figure 3 This is a schematic diagram illustrating the discarding of invalid sample audio data and its sample phoneme labels, provided in an embodiment of this application.

[0024] Figure 4 This is a schematic diagram of the phoneme segmentation results for generating speech recognition text provided in an embodiment of this application.

[0025] Figure 5 This is a schematic diagram illustrating the process of correcting the alignment results provided in an embodiment of this application.

[0026] Figure 6 This is a schematic diagram of the structure of a speech alignment device based on phoneme classification provided in an embodiment of this application.

[0027] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0028] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and not for limiting the invention. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention and not the entire structure.

[0029] It should be noted that, due to space limitations, this application specification does not exhaustively list all possible implementation methods. Those skilled in the art should be able to conceive after reading this application specification that, as long as the technical features do not contradict each other, any combination of technical features can constitute an optional implementation method.

[0030] The embodiments of the present invention will be described in detail below.

[0031] Forced alignment is a technique that aligns audio data segments in a media file with their corresponding text phoneme sequences, thereby establishing a one-to-one mapping between media file segments and phonemes, and ultimately finding the timing information of each phoneme's pronunciation. Existing forced alignment techniques mainly fall into two categories: forced alignment methods based on traditional methods such as GMM (Gaussian Mixed Model) and forced alignment methods based on CTC (Connectionist Temporal Classification) speech recognition.

[0032] The basic process of the forced alignment method based on GMM is as follows: First, obtain the training data of the model, including speech data and its corresponding text data; second, extract the MFCC (Mel Frequency Cepstrum Coefficient) features of the original speech samples; then, train the monophone GMM to obtain the preliminary phoneme alignment sequence; finally, train the triphone GMM model through the contextual relationship between phonemes to obtain a more accurate alignment model.

[0033] The principle of the forced alignment method based on CTC speech recognition is as follows: First, obtain model training data; second, train the CTC speech recognition model; finally, given an audio segment and its corresponding ASR text, the CTC model outputs the alignment information of the audio and text.

[0034] For the GMM phoneme classification model, the traditional machine learning methods used result in poor alignment (low alignment accuracy), making it difficult to achieve the same performance as larger (neural network) models. As for the forced alignment method in CTC speech recognition, due to the characteristics of the CTC algorithm, the recognized text occurs at the peak of the model output (the peak can occur anywhere between the beginning and end of the word), resulting in unsatisfactory word-level alignment accuracy. Furthermore, it cannot align silent parts of the speech. In summary, existing forced speech alignment schemes have poor word-level alignment accuracy, leading to inaccurate overall alignment results and consequently, low precision in audio / video editing based on speech content.

[0035] To address the above technical problems, this application proposes a speech alignment method based on phoneme classification. By accurately classifying the pronunciation state and details of the audio data to be processed, and through forced alignment achieved by path solving, it can achieve accurate alignment of the phoneme splitting results of two paths, which can significantly improve the operational accuracy when editing audio and video based on speech content. The various embodiments of the present invention will be described in detail below.

[0036] Figure 1 This application provides a flowchart of a speech alignment method based on phoneme classification, which is implemented by an electronic device, such as a personal computer or cloud server used for editing multimedia files with sound. Figure 1 As shown, the speech alignment method based on phoneme classification includes steps S110-S130:

[0037] Step S110: Input the audio data to be processed into the pre-trained phoneme classification model to obtain the first phoneme splitting result.

[0038] The audio data to be processed refers to the data portion of a multimedia file that contains speech content, specifically the portion used to record that speech content. The multimedia file referred to here mainly refers to a media file containing speech content obtained by data collection in a speaking scenario, such as an audio file, a video file containing speech content, or more specifically, a teaching video obtained by recording a scenario of teaching in Chinese.

[0039] Input the audio data to be processed into a pre-trained phoneme classification model. The phoneme classification model splits the audio data to be processed into phonemes, obtaining a first phoneme splitting result. The first phoneme splitting result includes phonemes obtained by splitting the words in the spoken content according to pronunciation. In linguistics, a phoneme is the smallest speech unit divided according to the natural attributes of speech. Analyzing based on the pronunciation actions in a syllable, one action constitutes one phoneme. For example, the Chinese syllable "啊(ā)" has only one phoneme, "爱(ài)" has two phonemes, "代(dài)" has three phonemes, etc. Of course, in the specific implementation process, phonemes can also be customized. For example, a set of initials and finals constitutes a phoneme set. In this implementation, "代(dài)" has two phonemes. The first phoneme splitting result also includes separator symbols and silence symbols. The separator symbol is added after the last phoneme corresponding to each word in the audio data to be processed, and the silence symbol is added to the silent pause parts between words in the audio data to be processed, as well as the silent parts at the beginning and end. For example, the silence symbol (exemplarily represented by "SIL" in the embodiments of this application) is used to represent the silent pause part where no one is speaking, and the separator symbol (exemplarily represented by "|" in the embodiments of this application). The separator symbol is used to separate adjacent words.

[0040] Before training the phoneme classification model, first align multiple sample audio data and the corresponding sample text data through a pre-trained Gaussian mixture model to obtain the phoneme-level labels of each sample audio data. Humans have relatively poor overall resolution of sounds and it is difficult to efficiently perform phoneme-level annotation in a manual annotation manner. Existing Gaussian mixture models have relatively mature phoneme recognition capabilities. By aligning the sample audio data and the corresponding sample text data, phoneme-level labels at the frame level can be obtained.

[0041] For the obtained phoneme-level labels, the phoneme-level labels can be adjusted according to a preset adjustment rule to obtain sample phoneme labels. For example, find the last frame phoneme at the end of each Chinese character in the corresponding phoneme sequence according to the text sequence. If the phoneme is continuous (more than or equal to 2 frames), then change the last frame to the separator symbol "|"; if the phoneme is discontinuous (only 1 frame), then discard this data, thereby obtaining the actually available sample audio data. Take each finally available sample audio data and its corresponding sample phoneme label as a sample data to obtain a sample dataset. The specific adjustment process can refer to Figure 2 and Figure 3 In Figure 2 In a sample audio data shown, the last frame phonemes 'i' and 'ao' corresponding to the two words "你好" are continuous, so change the last 'i' and 'ao' to separator symbols; in Figure 3In another sample audio data shown, for the last frame phonemes 'i' and 'ao' corresponding to the two characters in "Hello", if 'i' is discontinuous, then this sample audio data is invalid sample audio data and is discarded.

[0042] Based on the sample data set, a pre-set phoneme classification model is trained using the sample data set to obtain a phoneme classification model that takes audio data as input and phoneme labels as output. In the embodiments of this application, the pre-set phoneme classification model consists of a wav2vec2.0 pre-trained model and a FC (Fully Connected Layers) classification layer. The number of output units of the FC classification layer is the total number of phonemes in the phoneme set, and the training target for each frame of audio signal is its corresponding phoneme label. The training of the pre-set phoneme classification model can use cross-entropy loss, that is , where x is the model output and class represents a certain phoneme. In the specific implementation process, in order to reduce the number of output layer units of the network model, the tone information of the pronunciation of all phonemes is not considered (for example, the phonemes corresponding to the actual pronunciations āo, áo, ǎo, ào and the light tone ao are all 'ao').

[0043] Based on the phoneme classification model described above, inference is performed on the audio data to be processed, that is, the audio data to be processed is input into the phoneme classification model, and a classification probability matrix T*N can be obtained, where T is the number of frames and N is the number of phoneme classifications.

[0044] Step S120: Perform speech recognition on the audio data to be processed to obtain a speech recognition text, and perform phoneme splitting on the speech recognition text to obtain a second phoneme splitting result.

[0045] Convert the speech recognition text into corresponding phonemes according to the pronunciation to obtain an initial phoneme sequence. For example Figure 4 as shown, the initial phoneme sequence corresponding to "Hello" is 'n', 'i', 'h', 'ao'. Continuing to process the initial phoneme sequence, a separator symbol '|' is added after the last phoneme corresponding to each character to separate adjacent characters; a silence symbol 'SIL' is added after the separator symbol to predict the silent pause part between words; silence symbols 'SIL' are added at the beginning and end of the initial phoneme sequence to predict the silent parts at the beginning and end of the audio data. After the above processing, the second phoneme splitting result is obtained. In terms of the overall structure, the second phoneme splitting result includes not only the phonemes directly confirmed by the pronunciation of the text, but also separator symbols and silence symbols. The separator symbols are added after the last phoneme corresponding to each character in the audio data to be processed, and the silence symbols are added to the silent pause parts between words and the silent parts at the beginning and end of the audio data to be processed.

[0046] Combining the processing steps S110 and S120, it can be seen that the speech alignment in this embodiment of the application, while focusing on the phonemes corresponding to the text or speech, also focuses on the extraction of information from the speaking process, that is, on the impact of speaking pauses on forced alignment. By analyzing the speech itself and the transitions between texts during speech, further feature analysis is performed from the temporal structure of pronunciation to obtain the most accurate alignment result possible. Based on the specific details of rhythm and pronunciation, pauses (i.e., the silence state in the speaking interval) are accurately divided using silence symbols; and phonemes are accurately divided into words by using separator symbols for consecutively repeated phonemes. Continuously repeated monophonemes include consecutively identical monophoneme words, as well as cases where the last phoneme of the previous word and the first phoneme of the next word are the same.

[0047] Step S130: Perform path solving on the first phoneme splitting result and the second phoneme splitting result to obtain the alignment result.

[0048] Based on the first and second phoneme segmentation results, i.e. the corresponding classification probability matrix and the processed phoneme sequence, path solving is performed. For example, the alignment result can be obtained by solving the optimal path using the Viterbi algorithm. During the alignment process, separators and silence symbols are used as constraints on speech intervals and consecutive identical phonemes during alignment. This avoids disorder when aligning multiple consecutive identical phonemes and silence periods in the two phoneme decomposition results. For example, if the speech content consists of five consecutive "ā" sounds, the first phoneme decomposition result can obtain multiple consecutive frames of identical phonemes. In the phoneme classification model, the interval between two "ā" sounds can be identified. However, if these multiple consecutive identical phonemes are only recorded as identical pronunciation phoneme symbols in the first phoneme decomposition result, then when aligning with the second phoneme decomposition result, the actual phoneme alignment may not be possible due to changes in speech rate and the short duration of each frame. Separators and silence symbols are equivalent to marking the phonemes corresponding to each character in the first and second phoneme decomposition results. Path solving based on single-character marking can obtain accurate alignment results.

[0049] In actual processing, after obtaining the alignment result, considering that the rhythm and speed of speech may affect the alignment result, the alignment result can be corrected to further improve the final alignment accuracy and obtain the target alignment result. In the specific correction process, silence symbols that appear consecutively more than a preset number in the alignment result are retained, and the remaining silence symbols are corrected to separator symbols, resulting in the first correction result. Based on the silence symbols and separator symbols, silence frame segments and single-word frame segments are extracted from the first correction result to obtain the second correction result. Based on the duration of each frame, the timestamp information corresponding to the silence frame segments and single-word frame segments in the second correction result is confirmed to obtain the target alignment result.

[0050] The correction process mainly addresses the issues of the separator '|' and the silence symbol 'SIL'. '|' is actually silent and is used to separate adjacent characters, so it needs to be merged into the phoneme. 'SIL' is fixedly inserted between every two words to predict the pause between them, but in reality, there may be no pause between words, or the pause may be very short; therefore, the 'SIL' may also need to be merged.

[0051] In the specific correction process, for the initial alignment result, a threshold s is defined (for example, a threshold s=10, that is, silent segments with a duration of less than 10 frames will be merged). If the number of consecutive 'SIL' frames is greater than s, then 'SIL' is retained (if there are s or more consecutive 'SIL' segments, then they are retained); otherwise, 'SIL' is changed to '|', that is, merged into the segment of the previous word, to obtain the first correction result.

[0052] For the first correction result, find the first and last frames of consecutive 'SIL' segments, as well as the first and last frames of non-'SIL' segments (the frame after the last frame of a consecutive 'SIL' segment is the first frame, and the last frame of a subsequent consecutive '|' segment is the last frame). Since the above preprocessing steps use a fixed insertion '|' rule, the number of non-'SIL' segments here is equal to and corresponds one-to-one with the given number of text characters. Based on the above segment extraction, the second correction result can be obtained.

[0053] For the third correction result, the unit of the segment is a frame. The frame is converted to the time unit ms (if the wav2vec2.0 model described above is used, one frame equals 20ms) to obtain the target alignment result.

[0054] For the detailed correction process described above, please refer to... Figure 5 The example illustrates the data change process from the initial alignment result to the target alignment result, where the second correction result involves details of the segment and the corresponding frame range. Figure 5 The results are further distinguished and displayed as a second correction result a and a second correction result b. The second correction result a shows four segments, with adjacent segments distinguished by different fonts; the second correction result b shows the frame ranges corresponding to the four segments. The specific change process can be found in the correction processing described above.

[0055] Based on the timestamp information corresponding to each character confirmed in the alignment results, a timeline association between the speech recognition text and the media file corresponding to the audio data to be processed can be established according to the timestamp information. When an editing operation corresponding to the target speech recognition text is received, the editing operation is performed on the media file according to the timestamp information of the target speech recognition text.

[0056] The data organization of document files allows for quick location of elements such as creating directories and searching for text. However, for multimedia files, especially audio and video files captured by data acquisition devices, such as microphones or microphones combined with cameras, without data processing of the raw audio and video files, the target for editing is usually only located through timeline manipulation. Based on the alignment results obtained in the embodiments of this application, a timeline association between the speech-recognized text and the media file corresponding to the audio data to be processed is established. For multimedia files, the speech itself can already present the basic content of the file to the user, while text has a strong information-carrying capacity and convenient information viewing method. Users can quickly identify the part to be edited through speech-recognized text, and the timeline association between speech-recognized text and the media file allows for quick manipulation of the speech-recognized text, selecting a portion or all of it as the target speech-recognized text, and then editing the corresponding playback period of the target speech-recognized text in the media file. For example, after selecting a sentence in the speech-recognition text, this sentence becomes the target speech-recognition file for the current editing operation. The specific operation is deletion. Therefore, responding to this deletion operation involves editing the media file, specifically deleting the content corresponding to the target speech-recognition file within the media file. In this embodiment, based on a highly accurate alignment method, media files can be precisely edited through speech-recognition text, significantly improving operational precision when editing audio and video using speech content.

[0057] When editing media files based on the speech alignment method in the embodiments of this application, especially when editing classroom videos, conference videos, etc. with a lot of speaking content, the rich speaking content can provide a large amount of speech recognition text. The average granularity of each character in the speech recognition text in the media file is more delicate, which can provide a more refined video editing effect.

[0058] The aforementioned speech alignment method based on phoneme classification inputs the audio data to be processed into a pre-trained phoneme classification model to obtain the first phoneme segmentation result; performs speech recognition on the audio data to be processed to obtain speech recognition text, performs phoneme segmentation on the speech recognition text to obtain the second phoneme segmentation result; and performs path solving on the first and second phoneme segmentation results to obtain the alignment result. The first and second phoneme segmentation results include separators and silence symbols. The separators are located after the last phoneme of each word in the audio data to be processed, and the silence symbols are located in the silent pauses between words in the audio data to be processed, as well as the silent parts at the beginning and end. By accurately classifying the pronunciation state and pronunciation details of the audio data to be processed, two phoneme decomposition results are obtained. The two phoneme decomposition results are then forced to align through path solving. Separators and silence symbols are used as constraints on speech intervals and consecutive identical phonemes during alignment. This avoids disorder of multiple consecutive identical phonemes and silence periods when aligning the two phoneme decomposition results, thus achieving accurate alignment of the two phoneme decomposition results. This can significantly improve the operational accuracy when editing audio and video through voice content.

[0059] Figure 6 This is a schematic diagram of a speech alignment device based on phoneme classification, provided as an embodiment of this application. Figure 6 As shown, the speech alignment device based on phoneme classification includes a first splitting unit 210, a second splitting unit 220, and a phoneme alignment unit 230.

[0060] The first splitting unit 210 is used to input the audio data to be processed into a pre-trained phoneme classification model to obtain the first phoneme splitting result; the second splitting unit 220 is used to perform speech recognition on the audio data to be processed to obtain speech recognition text, and to perform phoneme splitting on the speech recognition text to obtain the second phoneme splitting result; the phoneme alignment unit 230 is used to perform path solving on the first phoneme splitting result and the second phoneme splitting result to obtain the alignment result; wherein the first phoneme splitting result and the second phoneme splitting result include a separator symbol and a silence symbol. The separator symbol is located after the last phoneme corresponding to each word in the audio data to be processed, and the silence symbol is located in the silent pause part between words in the audio data to be processed, as well as the silent part at the beginning and end.

[0061] Based on the above embodiments, the speech alignment device based on phoneme classification further includes:

[0062] The alignment correction unit is used to correct the alignment result to obtain the target alignment result.

[0063] Based on the above embodiments, the alignment correction unit includes:

[0064] The first correction module is used to retain the silence symbols that appear consecutively more than a preset number in the alignment result, and correct the remaining silence symbols to separator symbols to obtain the first correction result.

[0065] The second correction module is used to extract the silence frame segment and single-word frame segment from the first correction result based on the silence symbol and the separator symbol, and obtain the second correction result;

[0066] The third correction module is used to confirm the timestamp information corresponding to the silent frame segment and the single-word frame segment in the second correction result according to the duration of each frame, so as to obtain the target alignment result.

[0067] Based on the above embodiments, the phoneme classification model is trained in the following manner:

[0068] By aligning multiple sample audio data with corresponding sample text data using a pre-trained Gaussian mixture model, phoneme-level labels for each sample audio data are obtained.

[0069] The phoneme-level labels are adjusted according to the preset adjustment rules to obtain sample phoneme labels. Each sample audio data and its corresponding sample phoneme label are used as a sample data to obtain a sample dataset.

[0070] The pre-set phoneme classification model is trained based on the sample dataset to obtain a phoneme classification model that takes audio data as input and phoneme labels as output.

[0071] Based on the above embodiments, the pre-set phoneme classification model consists of a wav2vec2.0 pre-trained model and an FC classification layer.

[0072] Based on the above embodiments, the pre-defined phoneme classification model uses cross-entropy loss.

[0073] Based on the above embodiments, the alignment result includes the timestamp information corresponding to each character;

[0074] Correspondingly, the speech alignment device based on phoneme classification also includes:

[0075] The time association unit is used to establish a timeline association between the speech recognition text and the media file corresponding to the audio data to be processed, based on timestamp information.

[0076] The editing response unit performs editing operations on the media file based on the timestamp information of the target speech recognition text when the user receives an editing operation corresponding to the target speech recognition text.

[0077] The phoneme-based speech alignment device provided in this application embodiment is included in an electronic device and can be used to execute the corresponding phoneme-based speech alignment method provided in the above embodiment, and has corresponding functions and beneficial effects.

[0078] It is worth noting that in the above embodiments of the speech alignment device based on phoneme classification, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of the present invention.

[0079] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 7 As shown, the electronic device includes a processor 310 and a memory 320, and may also include an input device 330, an output device 340, and a communication device 350; the number of processors 310 in the electronic device may be one or more. Figure 7 Taking a processor 310 as an example; the processor 310, memory 320, input device 330, output device 340, and communication device 350 in the electronic device can be connected via a bus or other means. Figure 7 Taking the example of a connection between China and Israel via a bus.

[0080] The memory 320, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the phoneme-based speech alignment method in this embodiment. The processor 310 executes various functional applications and data processing of the electronic device by running the software programs, instructions, and modules stored in the memory 320, thereby implementing the aforementioned phoneme-based speech alignment method.

[0081] The memory 320 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on the use of the electronic device. Furthermore, the memory 320 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 320 may further include memory remotely located relative to the processor 310, which can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0082] Input device 330 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the electronic device. Output device 340 may include display devices such as a display screen.

[0083] The aforementioned electronic device includes a phoneme-based speech alignment device, which can be used to execute any phoneme-based speech alignment method and has corresponding functions and beneficial effects.

[0084] This application also provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program performs relevant operations in the phoneme classification-based speech alignment method provided in any embodiment of this application, and has corresponding functions and beneficial effects.

[0085] Those skilled in the art will understand that embodiments of this application may be provided as methods, systems, or computer program products.

[0086] Therefore, this application may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce implementations of the flowchart... Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0087] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory. Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0088] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0089] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0090] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A phoneme classification based speech alignment method, characterized in that, The method comprises the following steps: inputting the to-be-processed audio data into a pre-trained phoneme classification model to obtain a first phoneme segmentation result; performing speech recognition on the to-be-processed audio data to obtain a speech recognition text, and performing phoneme segmentation on the speech recognition text to obtain a second phoneme segmentation result; solving paths of the first phoneme segmentation result and the second phoneme segmentation result to obtain an alignment result; retaining the silence symbols that continuously appear more than a preset number in the alignment result, and correcting the remaining silence symbols into delimiter symbols to obtain a first correction result; extracting silence frame segments and single-word frame segments in the first correction result according to the silence symbols and the delimiter symbols to obtain a second correction result; confirming timestamp information corresponding to the silence frame segments and the single-word frame segments in the second correction result according to the time length of each frame to obtain a target alignment result. The first phoneme segmentation result and the second phoneme segmentation result comprise delimiter symbols and silence symbols, the delimiter symbols are located after the last phoneme corresponding to each word in the to-be-processed audio data, and the silence symbols are located in the silence pause part between words in the to-be-processed audio data and the silence part at the beginning and end.

2. The speech alignment method of claim 1, wherein, The phoneme classification model is trained in the following manner: aligning a plurality of sample audio data and corresponding sample text data through a pre-trained Gaussian mixture model to obtain phoneme-level labels of each sample audio data; adjusting the phoneme-level labels according to a preset adjustment rule to obtain sample phoneme labels, taking each sample audio data and the corresponding sample phoneme label as a sample data to obtain a sample data set; training a preset phoneme classification model based on the sample data set to obtain a phoneme classification model taking audio data as input and phoneme labels as output.

3. The speech alignment method of claim 2, wherein, The preset phoneme classification model is composed of a wav2vec2.0 pre-training model and an FC classification layer.

4. The speech alignment method of claim 3, wherein, The preset phoneme classification model uses a cross-entropy loss.

5. The speech alignment method of claim 1, wherein, The alignment result comprises timestamp information corresponding to each word. Correspondingly, after the step of solving paths of the first phoneme segmentation result and the second phoneme segmentation result to obtain an alignment result, the method further comprises the following steps: establishing a time axis association relationship between the speech recognition text and a media file corresponding to the to-be-processed audio data according to the timestamp information; when an editing operation corresponding to a target speech recognition text is received, performing the editing operation on the media file according to the timestamp information of the target speech recognition text.

6. Apparatus for speech alignment based on phoneme classification, characterized in that The method comprises the following steps: a first splitting unit is configured to input to-be-processed audio data into a pre-trained phoneme classification model to obtain a first phoneme segmentation result; a second splitting unit is configured to perform speech recognition on the to-be-processed audio data to obtain a speech recognition text, and perform phoneme segmentation on the speech recognition text to obtain a second phoneme segmentation result; a phoneme alignment unit is configured to solve paths of the first phoneme segmentation result and the second phoneme segmentation result to obtain an alignment result; a first correction module is configured to retain the silence symbols that continuously appear more than a preset number in the alignment result, and correct the remaining silence symbols into delimiter symbols to obtain a first correction result; The second correction module is configured to extract the silence frame segment and the single-word frame segment in the first correction result according to the silence symbol and the separator symbol, and obtain a second correction result. The third correction module is configured to confirm the timestamp information corresponding to the silence frame segment and the single-word frame segment in the second correction result according to the time length corresponding to each frame, and obtain a target alignment result. The first phoneme split result and the second phoneme split result include the separator symbol and the silence symbol, the separator symbol is located after the last phoneme corresponding to each word in the to-be-processed audio data, the silence symbol is located in the silence pause part between words in the to-be-processed audio data, and the silence parts at the beginning and the end.

7. An electronic device, comprising: The computer program is executed by the processor to implement the phoneme classification-based speech alignment method according to any one of claims 1-5. The computer program is executed by the processor to implement the phoneme classification-based speech alignment method according to any one of claims 1-5. ​ ​ 8. A computer-readable storage medium having stored thereon a computer program, characterized in that, ​

Images