Voice alignment method and device based on phoneme classification, equipment and medium

Abstract

The embodiment of the invention discloses a voice alignment method and device based on phoneme classification, electronic equipment and a storage medium. The voice alignment method comprises the following steps: inputting to-be-processed audio data into a pre-trained phoneme classification model to obtain a first phoneme splitting result; performing voice recognition on the to-be-processed audio data to obtain a voice recognition text, and performing phoneme splitting on the voice recognition text to obtain a second phoneme splitting result; and performing path solving on the first phoneme splitting result and the second phoneme splitting result to obtain an alignment result. Pronunciation states and pronunciation details of to-be-processed audio data are precisely classified, two paths of phoneme splitting results are forcibly aligned through path solving, and a separation symbol and a mute symbol number are used as constraints on a speaking interval and continuous same phonemes during alignment, so that precise alignment of the two paths of phoneme splitting results can be realized, and the accuracy of alignment of the two paths of phoneme splitting results is improved. The operation precision can be obviously improved when audio and video are edited through the voice content.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of multimedia data processing technology, and in particular to a speech alignment method, apparatus, device and medium based on phoneme classification. Background Art

[0002] The editing method of traditional audio and video editing software operates on the audio and video timeline to achieve audio and video editing. This process often requires repeated viewing and / or listening to confirm the time segment of the editing, which is time-consuming and labor-intensive. In order to improve the efficiency of audio and video editing, a method of fast editing through voice forced alignment technology has been developed. The specific operation process is to transcribe the voice content in the audio and video into text through ASR (Automatic Speech Recognition), and obtain the timestamp information (start and end time in the audio and video) of each text unit (Chinese character) through the forced alignment model, so as to achieve fast audio and video editing by operating text.

[0003] When the inventor used the existing speech forced alignment technology for audio and video editing, he found that the alignment accuracy at the word level in the speech forced alignment link was poor, resulting in the overall alignment result being inaccurate, and thus the operation accuracy was not high when editing audio and video through speech content. Summary of the invention

[0004] The present invention provides a speech alignment method, device, equipment and medium based on phoneme classification to solve the technical problem that the existing speech alignment result is not accurate enough and the operation accuracy is not high when editing audio and video according to speech content.

[0005] In a first aspect, an embodiment of the present application provides a speech alignment method based on phoneme classification, the speech alignment method based on phoneme classification comprising:

[0006] Inputting the audio data to be processed into a pre-trained phoneme classification model to obtain a first phoneme segmentation result;

[0007] Performing speech recognition on the audio data to be processed to obtain a speech recognition text, and performing phoneme splitting on the speech recognition text to obtain a second phoneme splitting result;

[0008] Perform path solving on the first phoneme splitting result and the second phoneme splitting result to obtain an alignment result;

[0009] Among them, the first phoneme splitting result and the second phoneme splitting result include a separator symbol and a silent symbol. The separator symbol is located after the last phoneme corresponding to each word in the audio data to be processed, and the silent symbol is not located in the silent pause part between words in the audio data to be processed, as well as the silent part at the beginning and the end.

[0010] In a second aspect, an embodiment of the present application further provides a speech alignment device based on phoneme classification, the speech alignment device based on phoneme classification comprising:

[0011] A first splitting unit, used for inputting the audio data to be processed into a pre-trained phoneme classification model to obtain a first phoneme splitting result;

[0012] A second splitting unit is used to perform speech recognition on the audio data to be processed to obtain a speech recognition text, and perform phoneme splitting on the speech recognition text to obtain a second phoneme splitting result;

[0013] A phoneme alignment unit, used for performing path solving on the first phoneme splitting result and the second phoneme splitting result to obtain an alignment result;

[0014] Among them, the first phoneme splitting result and the second phoneme splitting result include a separator symbol and a silent symbol. The separator symbol is located after the last phoneme corresponding to each word in the audio data to be processed, and the silent symbol is not located in the silent pause part between words in the audio data to be processed, as well as the silent part at the beginning and the end.

[0015] In a third aspect, an embodiment of the present application further provides an electronic device, the electronic device comprising:

[0016] one or more processors;

[0017] a memory for storing one or more computer programs;

[0018] When one or more computer programs are executed by one or more processors, the electronic device implements the speech alignment method based on phoneme classification as in the first aspect.

[0019] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the speech alignment method based on phoneme classification as in the first aspect.

[0020] The above-mentioned speech alignment method, device, equipment and medium based on phoneme classification, in which the audio data to be processed is input into a pre-trained phoneme classification model to obtain a first phoneme splitting result; speech recognition is performed on the audio data to be processed to obtain a speech recognition text, and the speech recognition text is phoneme split to obtain a second phoneme splitting result; the first phoneme splitting result and the second phoneme splitting result are path-solved to obtain an alignment result, the first phoneme splitting result and the second phoneme splitting result include a separator and a silent symbol, the separator is located after the last phoneme corresponding to each word in the audio data to be processed, and the silent symbol is not located in the silent pause part between words in the audio data to be processed, as well as the silent part at the beginning and the end. By accurately classifying the pronunciation status and pronunciation details of the audio data to be processed, two-way phoneme splitting results are obtained, and the two-way phoneme splitting results are forced to be aligned through path solving. Separation symbols and silence symbols are used as constraints on speaking intervals and consecutive identical phonemes during alignment to avoid confusion of multiple consecutive identical phonemes and silent periods when aligning the two-way phoneme splitting results. This can achieve accurate alignment of the two-way phoneme splitting results and significantly improve the operation accuracy when editing audio and video through voice content. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A method flow chart of a speech alignment method based on phoneme classification provided in an embodiment of the present application.

[0022] Figure 2 A schematic diagram of adjusting sample phoneme labels of valid sample audio data provided in an embodiment of the present application.

[0023] Figure 3 A schematic diagram of discarding invalid sample audio data and its sample phoneme labels provided in an embodiment of the present application.

[0024] Figure 4 A schematic diagram of the phoneme splitting results of generating speech recognition text provided in an embodiment of the present application.

[0025] Figure 5 A schematic diagram of the process of correcting the alignment result provided in an embodiment of the present application.

[0026] Figure 6 A schematic diagram of the structure of a speech alignment device based on phoneme classification provided in an embodiment of the present application.

[0027] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only the parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0029] It should be noted that due to space limitations, this application description does not list all optional implementation methods. After reading this application description, those skilled in the art should be able to understand that as long as the technical features do not contradict each other, any combination of technical features can constitute an optional implementation method.

[0030] The following is a detailed description of the various embodiments of the present invention.

[0031] Forced alignment technology is a method of aligning the audio data part in the media file with its corresponding text phoneme sequence, thereby establishing a one-to-one mapping relationship between the media file segment and the phoneme, and finding the time point information of each phoneme pronunciation. There are two main existing speech forced alignment technologies: the traditional GMM (Gaussian Mixed Model) based on the forced alignment method of speech recognition based on CTC (Connectionist Temporal Classification).

[0032] The basic process of the GMM-based forced alignment method is as follows: first, obtain the training data of the model, including speech data and its corresponding text data; second, extract the MFCC (Mel Frequency Cepstrum Coefficient) features of the original speech sample; then, train the monophone GMM to obtain a preliminary phoneme alignment sequence; finally, train the triphone GMM model through the contextual relationship between phonemes to further obtain a more accurate alignment model.

[0033] The principle of the forced alignment method based on CTC speech recognition is: first, obtain model training data; second, train the CTC speech recognition model; finally, given a piece of audio and its corresponding ASR text, the CTC model outputs the alignment information of the audio and text.

[0034] For the GMM phoneme classification model, due to the traditional machine learning method used, the alignment effect achieved is poor (low alignment accuracy), and it is difficult to achieve the same effect as the large (neural network) model; for the forced alignment method of CTC speech recognition, due to the characteristics of the CTC algorithm, the recognized text is at the peak moment of the model output (the peak moment may appear at any position between the start and end time of the word), so the alignment accuracy at the word level is not ideal, and on the other hand, it is impossible to align the silent part of the speech. In general, the existing speech forced alignment scheme has poor alignment accuracy at the word level, resulting in the overall alignment result being not accurate enough, and then the operation accuracy is not high when editing audio and video through speech content.

[0035] In view of the above technical problems, the embodiments of the present application propose a speech alignment method based on phoneme classification, which can achieve accurate alignment of the two-way phoneme splitting results by accurately classifying the pronunciation state and pronunciation details of the audio data to be processed and implementing forced alignment through path solving, and can significantly improve the operation accuracy when editing audio and video by voice content. The embodiments of the present invention are described in detail below.

[0036] Figure 1 A method flow chart of a speech alignment method based on phoneme classification provided in an embodiment of the present application, the speech alignment method based on phoneme classification is implemented by an electronic device, such as a personal computer, a cloud server, etc. for editing multimedia files with sound. Figure 1 As shown, the speech alignment method based on phoneme classification includes steps S110 to S130:

[0037] Step S110: input the audio data to be processed into a pre-trained phoneme classification model to obtain a first phoneme segmentation result.

[0038] The audio data to be processed refers to the data part used to record the voice content in the multimedia file carrying the voice content. The multimedia file mentioned here mainly refers to the media file recording the speech content obtained by data collection in the speaking scene, such as audio files, video files with speaking content, etc., and more specifically, a teaching video obtained by recording the scene of teaching Chinese.

[0039] Input the audio data to be processed into a pre-trained phoneme classification model. The phoneme classification model splits the audio data to be processed into phonemes to obtain a first phoneme splitting result. The first phoneme splitting result includes phonemes obtained by splitting the words in the speech content according to pronunciation. In linguistics, a phoneme is the smallest speech unit divided according to the natural attributes of speech. Analyzing according to the pronunciation actions in a syllable, one action constitutes one phoneme. For example, the Chinese syllable "啊(ā)" has only one phoneme, "爱(ài)" has two phonemes, "代(dài)" has three phonemes, etc. Of course, in the specific implementation process, phonemes can also be customized. For example, a set of initials and finals constitutes a phoneme set. In this implementation, "代(dài)" has two phonemes. The first phoneme splitting result also includes separator symbols and silence symbols. The separator symbol is added after the last phoneme corresponding to each word in the audio data to be processed, and the silence symbol is added to the silent pause parts between words in the audio data to be processed, as well as the silent parts at the beginning and end. For example, the silence symbol (exemplarily represented by "SIL" in the embodiments of the present application) is used to represent the silent pause part where no one is speaking, and the separator symbol (exemplarily represented by "|" in the embodiments of the present application). The separator symbol is used to separate adjacent words.

[0040] Before training the phoneme classification model, first align multiple sample audio data and the corresponding sample text data through a pre-trained Gaussian mixture model to obtain the phoneme-level labels of each sample audio data; humans have relatively poor overall resolution of sounds and it is difficult to efficiently perform phoneme-level annotation in a manual annotation manner. Existing Gaussian mixture models have relatively mature phoneme recognition capabilities. Aligning the sample audio data and the corresponding sample text data can obtain frame-level phoneme labels.

[0041] For the obtained phoneme-level labels, the phoneme-level labels can be adjusted according to a preset adjustment rule to obtain sample phoneme labels. For example, find the last frame phoneme at the end of each Chinese character in the corresponding phoneme sequence according to the text sequence. If the phoneme is continuous (more than or equal to 2 frames), then change the last frame to the separator symbol "|"; if the phoneme is not continuous (only 1 frame), then discard the data, thereby obtaining the actually available sample audio data; use each finally available sample audio data and the corresponding sample phoneme label as a sample data to obtain a sample data set. The specific adjustment process can refer to Figure 2 and Figure 3 , in Figure 2 In a sample audio data shown, the last frame phonemes 'i' and 'ao' corresponding to the two words in "你好" are continuous, so change the last 'i' and 'ao' to separator symbols; in Figure 3In another sample audio data shown, the last frame phonemes 'i' and 'ao' corresponding to the two characters in "hello" are discontinuous, so the sample audio data is invalid and is discarded.

[0042] Based on the sample data set, the preset phoneme classification model is trained based on the sample data set to obtain a phoneme classification model with audio data as input and phoneme labels as output. In the embodiment of the present application, the preset phoneme classification model is composed of a wav2vec2.0 pre-trained model and a FC (Fully Connected Layers) classification layer. The number of output units of the FC classification layer is the total number of phonemes in the phoneme set, and the training target of each frame of audio signal is its corresponding phoneme label. The preset phoneme classification model training can use cross entropy loss, that is, Where x is the model output, and class represents a phoneme. In the specific implementation process, in order to reduce the number of output layer units of the network model, all phonemes do not consider the tone information of pronunciation (for example, the actual pronunciations of āo, áo, ǎo, ào and the phonemes corresponding to the light tone ao are all 'ao').

[0043] Based on the phoneme classification model described above, the audio data to be processed is inferred, that is, the audio data to be processed is input into the phoneme classification model, and a classification probability matrix T*N can be obtained, where T is the number of frames and N is the number of phoneme classifications.

[0044] Step S120: performing speech recognition on the audio data to be processed to obtain speech recognition text, and performing phoneme splitting on the speech recognition text to obtain a second phoneme splitting result.

[0045] The speech recognition text is converted into corresponding phonemes according to the pronunciation to obtain the initial phoneme sequence, such as Figure 4 As shown, the initial phoneme sequence corresponding to "hello" is 'n', 'i', 'h', 'ao'. The initial phoneme sequence is further processed, and a separator symbol '|' is added after the last phoneme corresponding to each word to separate adjacent words; a silent symbol "SIL" is added after the separator symbol to predict the silent pause part between words; a silent symbol "SIL" is added to the beginning and end of the initial phoneme sequence to predict the silent part at the beginning and end of the audio data. After the above processing, the second phoneme splitting result is obtained. From the overall structure, the second phoneme splitting result includes not only the phonemes directly confirmed by the pronunciation of the text, but also separator symbols and silent symbols. The separator symbol is added after the last phoneme corresponding to each word in the audio data to be processed, and the silent symbol is added to the silent pause part between words in the audio data to be processed, as well as the silent part at the beginning and end.

[0046] Combining the processing in step S110 and step S120, it can be found that the speech alignment in the embodiment of the present application not only pays attention to the phonemes corresponding to the text or speech, but also pays attention to the information extraction of the speaking process, that is, pays attention to the impact of speaking pauses on forced alignment, and further performs feature analysis on the time structure of pronunciation through the speech itself and the transition connection between text and text during speaking, so as to obtain the most accurate alignment result. According to the specific details of the rhythm and pronunciation, the pauses (i.e. the silent state in the speaking interval) are accurately divided by the mute symbol; the phonemes corresponding to the single words are accurately divided by the separator symbol for the continuously repeated phonemes, wherein the continuously repeated monophones include consecutive identical monophone words, and the last phoneme of the previous word is the same as the first phoneme of the next word.

[0047] Step S130: performing path solving on the first phoneme splitting result and the second phoneme splitting result to obtain an alignment result.

[0048] Based on the first phoneme splitting result and the second phoneme splitting result, that is, the corresponding classification probability matrix and the processed phoneme sequence, a path solution is performed, for example, the alignment result can be obtained by performing optimal path solution through the Viterbi algorithm. During the alignment process, separators and silence symbols are used as constraints on speaking intervals and consecutive identical phonemes during alignment to avoid confusion of multiple consecutive identical phonemes and silent periods when aligning the two phoneme splitting results. For example, if the speaking content is a word with five consecutive "ā" pronunciations, multiple consecutive frames of identical phonemes can be obtained in the first phoneme splitting result. In the phoneme classification model, the interval between two "ā"s can be identified. However, if these consecutive frames of identical phonemes are only recorded as the same pronunciation phoneme symbols in the first phoneme splitting result, then when aligning with the second phoneme splitting result, the corresponding time length of a single frame may be very short due to changes in the speaking speed, resulting in the inability to achieve accurate alignment of the phonemes. The separators and silence symbols are equivalent to marking the phonemes corresponding to each word in the first phoneme splitting result and the second phoneme splitting result. Path solving is performed based on the single word marking to obtain accurate alignment results.

[0049] In the actual processing process, after obtaining the alignment result, considering that the rhythm and speed of speaking may affect the alignment result, in order to further improve the final alignment accuracy, the alignment result can be corrected to obtain the target alignment result. In the specific correction process, the silent symbols that appear continuously for more than a preset number in the alignment result are retained, and the remaining silent symbols are corrected to separators to obtain the first correction result; based on the silent symbols and separators, the silent frame segments and single-word frame segments in the first correction result are extracted to obtain the second correction result; based on the corresponding duration of each frame, the timestamp information corresponding to the silent frame segments and single-word frame segments in the second correction result is confirmed to obtain the target alignment result.

[0050] The correction process mainly solves the problem of the separator symbol '|' and the silent symbol 'SIL'. '|' is actually silent and is used to separate adjacent characters, so it needs to be merged into the pronunciation phoneme. 'SIL' is fixed and inserted into every two words to predict the pause between words. In actual situation, there may be no pause between words, or the pause is very short, and the 'SIL' may also need to be merged.

[0051] In the specific correction process, for the initial alignment result, a threshold s is defined (for example, a threshold s=10 is defined, that is, silent segments with a duration of less than 10 frames will be merged). If the number of consecutive 'SIL' frames is greater than s, 'SIL' is retained (if there are s or more consecutive 'SIL' segments, it is retained); otherwise, 'SIL' is changed to '|', that is, merged into the segment of the previous word to obtain the first correction result.

[0052] For the first correction result, find the first and last frames of the continuous 'SIL' segment, as well as the first and last frames of the non-'SIL' segment (the next frame of the last frame of the continuous 'SIL' segment is the first frame, and the last frame '|' of the subsequent continuous '|' segment is the last frame). Since the above pre-processing step adopts a fixed insertion '|' rule, the number of non-'SIL' segments here is equal to the given number of text words and corresponds one to one. Based on the above segment extraction, the second correction result can be obtained.

[0053] For the third correction result, the unit of the segment is frame, and the frame is converted into the time unit ms (if the wav2vec2.0 model described above is used, one frame is equal to 20ms) to obtain the target alignment result.

[0054] For the specific calibration process above, please refer to Figure 5 , which exemplarily presents the data change process of gradually obtaining the target alignment result from the alignment result, wherein the second correction result involves the details of the fragment and the corresponding frame range of the fragment, Figure 5 The second correction result a and the second correction result b are further distinguished in the figure, wherein the second correction result a shows four segments, and adjacent segments are distinguished by different fonts; the second correction result b shows the frame ranges corresponding to the four segments. The specific change process can refer to the correction process described above.

[0055] According to the timestamp information corresponding to each word confirmed in the alignment result, on the basis of the alignment result, a time axis association relationship between the speech recognition text and the media file corresponding to the audio data to be processed can also be established according to the timestamp information; when an editing operation corresponding to the target speech recognition text is received, the editing operation is performed on the media file according to the timestamp information of the target speech recognition text.

[0056] The data organization method of document files allows them to be quickly located by setting directories and searching for texts. For multimedia files, especially those acquired through data acquisition devices, such as audio and video files acquired through microphones or microphones in combination with cameras, if the acquired original audio and video files are not processed, they can usually only be located by operating the timeline to view the target and then edited. Based on the alignment results obtained in the scheme in the embodiment of the present application, a timeline association relationship between the speech recognition text and the media file corresponding to the audio data to be processed is established. For multimedia files, the voice therein can basically present the corresponding file content to the user, and the text has a strong information carrying capacity and a convenient way to view information. The user can quickly confirm the part that needs to be edited through the speech recognition text, and the timeline association relationship between the speech recognition text and the media file can quickly operate on the speech recognition text, select a part or all as the target speech recognition text, and then edit the corresponding playback period of the target speech recognition text in the media file. For example, after a sentence is selected in the speech recognition text, this sentence is the target speech recognition file of the current editing operation, and the specific operation content is a deletion operation. Then, when responding to the deletion operation, the media file is specifically edited, and the specific editing content is to delete the content corresponding to the target speech recognition file in the media file. In the embodiment of the present application, based on a highly accurate alignment method, the media file can be accurately edited through the speech recognition text, and the operation accuracy is significantly improved when the audio and video are edited through the speech content.

[0057] When editing media files based on the speech alignment method in the embodiment of the present application, especially when editing classroom videos, conference videos, etc. with a lot of speech content, because the rich speech content can provide a large amount of speech recognition text, the average granularity of each character in the speech recognition text corresponding to the media file is finer, which can provide a more refined video editing effect.

[0058] The above-mentioned speech alignment method based on phoneme classification inputs the audio data to be processed into a pre-trained phoneme classification model to obtain a first phoneme splitting result; performs speech recognition on the audio data to be processed to obtain a speech recognition text, and performs phoneme splitting on the speech recognition text to obtain a second phoneme splitting result; performs path solving on the first phoneme splitting result and the second phoneme splitting result to obtain an alignment result; wherein the first phoneme splitting result and the second phoneme splitting result include a separator symbol and a silent symbol, the separator symbol is located after the last phoneme corresponding to each word in the audio data to be processed, and the silent symbol is not located in the silent pause part between words in the audio data to be processed, as well as the silent part at the beginning and the end. By accurately classifying the pronunciation status and pronunciation details of the audio data to be processed, two-way phoneme splitting results are obtained, and the two-way phoneme splitting results are forced to be aligned through path solving. Separation symbols and silence symbols are used as constraints on speaking intervals and consecutive identical phonemes during alignment to avoid confusion of multiple consecutive identical phonemes and silent periods when aligning the two-way phoneme splitting results. This can achieve accurate alignment of the two-way phoneme splitting results and significantly improve the operation accuracy when editing audio and video through voice content.

[0059] Figure 6 A schematic diagram of the structure of a speech alignment device based on phoneme classification provided in an embodiment of the present application. Figure 6 As shown, the speech alignment device based on phoneme classification includes a first splitting unit 210, a second splitting unit 220 and a phoneme alignment unit 230.

[0060] Among them, the first splitting unit 210 is used to input the audio data to be processed into a pre-trained phoneme classification model to obtain a first phoneme splitting result; the second splitting unit 220 is used to perform speech recognition on the audio data to be processed to obtain a speech recognition text, and perform phoneme splitting on the speech recognition text to obtain a second phoneme splitting result; the phoneme alignment unit 230 is used to perform path solving on the first phoneme splitting result and the second phoneme splitting result to obtain an alignment result; wherein the first phoneme splitting result and the second phoneme splitting result include a separator and a silent symbol, the separator is located after the last phoneme corresponding to each word in the audio data to be processed, and the silent symbol is not located in the silent pause part between words in the audio data to be processed, as well as the silent part at the beginning and the end.

[0061] On the basis of the above embodiment, the speech alignment device based on phoneme classification further includes:

[0062] The alignment correction unit is used to correct the alignment result to obtain a target alignment result.

[0063] Based on the above embodiment, the alignment correction unit includes:

[0064] A first correction module is used to retain the silent symbols that appear continuously for more than a preset number in the alignment result, and correct the remaining silent symbols into separation symbols to obtain a first correction result;

[0065] A second correction module is used to extract the silence frame segment and the single-word frame segment in the first correction result according to the silence symbol and the separation symbol to obtain a second correction result;

[0066] The third correction module is used to confirm the timestamp information corresponding to the silent frame segments and the single-word frame segments in the second correction result according to the duration corresponding to each frame, and obtain the target alignment result.

[0067] Based on the above embodiment, the phoneme classification model is trained in the following way:

[0068] Align multiple sample audio data with corresponding sample text data through a pre-trained Gaussian mixture model to obtain a phoneme-level label for each sample audio data;

[0069] The phoneme-level labels are adjusted according to preset adjustment rules to obtain sample phoneme labels, and each sample audio data and the corresponding sample phoneme label are taken as a sample data to obtain a sample data set;

[0070] The preset phoneme classification model is trained based on the sample data set to obtain a phoneme classification model that takes audio data as input and phoneme labels as output.

[0071] Based on the above embodiment, the preset phoneme classification model is composed of a wav2vec2.0 pre-trained model and a FC classification layer.

[0072] Based on the above embodiment, the preset phoneme classification model uses cross entropy loss.

[0073] Based on the above embodiment, the alignment result includes timestamp information corresponding to each word;

[0074] Correspondingly, the speech alignment device based on phoneme classification also includes:

[0075] A time association unit, used to establish a time axis association relationship between the speech recognition text and the media file corresponding to the audio data to be processed according to the timestamp information;

[0076] The editing response unit performs the editing operation on the media file according to the timestamp information of the target speech recognition text when the user receives the editing operation corresponding to the target speech recognition text.

[0077] The speech alignment device based on phoneme classification provided in the embodiment of the present application is included in an electronic device and can be used to execute the corresponding speech alignment method based on phoneme classification provided in the above embodiment, and has corresponding functions and beneficial effects.

[0078] It is worth noting that in the above-mentioned embodiment of the speech alignment device based on phoneme classification, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0079] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 7 As shown, the electronic device includes a processor 310 and a memory 320, and may also include an input device 330, an output device 340 and a communication device 350; the number of the processor 310 in the electronic device may be one or more, Figure 7 A processor 310 is taken as an example; the processor 310, the memory 320, the input device 330, the output device 340 and the communication device 350 in the electronic device can be connected via a bus or other means. Figure 7 The example of connecting through bus is taken in the following.

[0080] The memory 320, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the speech alignment method based on phoneme classification in the embodiment of the present application. The processor 310 executes various functional applications and data processing of the electronic device by running the software programs, instructions and modules stored in the memory 320, that is, realizing the above-mentioned speech alignment method based on phoneme classification.

[0081] The memory 320 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 320 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 320 may further include a memory remotely arranged relative to the processor 310, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0082] The input device 330 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the electronic device. The output device 340 may include a display device such as a display screen.

[0083] The electronic device mentioned above includes a speech alignment device based on phoneme classification, which can be used to execute any speech alignment method based on phoneme classification and has corresponding functions and beneficial effects.

[0084] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it is used to perform relevant operations in the speech alignment method based on phoneme classification provided in any embodiment of the present application, and has corresponding functions and beneficial effects.

[0085] Those skilled in the art should understand that the embodiments of the present application may be provided as methods, systems, or computer program products.

[0086] Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the functions in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0087] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory. The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0088] Computer readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0089] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0090] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A speech alignment method based on phoneme classification, characterized in that: include: Inputting the audio data to be processed into a pre-trained phoneme classification model to obtain a first phoneme segmentation result; Performing speech recognition on the audio data to be processed to obtain a speech recognition text, and performing phoneme splitting on the speech recognition text to obtain a second phoneme splitting result; Perform path solving on the first phoneme splitting result and the second phoneme splitting result to obtain an alignment result; Among them, the first phoneme splitting result and the second phoneme splitting result include a separator symbol and a silent symbol, the separator symbol is located after the last phoneme corresponding to each word in the audio data to be processed, and the silent symbol is not located in the silent pause part between words in the audio data to be processed, as well as the silent part at the beginning and the end.

2. The speech alignment method according to claim 1, characterized in that: After performing path solving on the first phoneme splitting result and the second phoneme splitting result to obtain an alignment result, the method further includes: The alignment result is corrected to obtain a target alignment result.

3. The speech alignment method according to claim 2, characterized in that: The correcting the alignment result to obtain a target alignment result includes: Retaining the silent symbols that appear continuously for more than a preset number in the alignment result, and correcting the remaining silent symbols into separation symbols to obtain a first correction result; Extracting the silence frame segments and the single-word frame segments in the first correction result according to the silence symbol and the separation symbol to obtain a second correction result; According to the duration corresponding to each frame, the timestamp information corresponding to the silent frame segments and the single-word frame segments in the second correction result is confirmed to obtain the target alignment result.

4. The speech alignment method according to any one of claims 1 to 3, characterized in that: The phoneme classification model is trained in the following way: Aligning multiple sample audio data with corresponding sample text data through a pre-trained Gaussian mixture model to obtain a phoneme-level label for each of the sample audio data; The phoneme-level labels are adjusted according to a preset adjustment rule to obtain sample phoneme labels, and each of the sample audio data and the corresponding sample phoneme label is taken as a sample data to obtain a sample data set; The preset phoneme classification model is trained based on the sample data set to obtain a phoneme classification model that takes the audio data as input and the phoneme label as output.

5. The speech alignment method according to claim 4, characterized in that: The preset phoneme classification model is composed of a wav2vec2.0 pre-trained model and an FC classification layer.

6. The speech alignment method according to claim 5, characterized in that: The preset phoneme classification model uses cross entropy loss.

7. The speech alignment method according to any one of claims 1 to 3, characterized in that: The alignment result includes timestamp information corresponding to each word; Correspondingly, after performing path solving on the first phoneme splitting result and the second phoneme splitting result to obtain an alignment result, the method further includes: According to the timestamp information, establishing a time axis association relationship between the speech recognition text and the media file corresponding to the audio data to be processed; When an editing operation corresponding to the target speech recognition text is received, the editing operation is performed on the media file according to the timestamp information of the target speech recognition text.

8. A speech alignment device based on phoneme classification, characterized in that: include: A first splitting unit, used for inputting the audio data to be processed into a pre-trained phoneme classification model to obtain a first phoneme splitting result; A second splitting unit is used to perform speech recognition on the audio data to be processed to obtain a speech recognition text, and perform phoneme splitting on the speech recognition text to obtain a second phoneme splitting result; A phoneme alignment unit, configured to perform path solving on the first phoneme splitting result and the second phoneme splitting result to obtain an alignment result; Among them, the first phoneme splitting result and the second phoneme splitting result include a separator symbol and a silent symbol, the separator symbol is located after the last phoneme corresponding to each word in the audio data to be processed, and the silent symbol is not located in the silent pause part between words in the audio data to be processed, as well as the silent part at the beginning and the end.

9. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more computer programs; When the one or more computer programs are executed by the one or more processors, the electronic device implements the speech alignment method based on phoneme classification as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the speech alignment method based on phoneme classification as described in any one of claims 1 to 7 is implemented.

Images