Audio text alignment method, computer device and computer readable storage medium

Abstract

This application relates to an audio-text alignment method, a computer device, and a computer-readable storage medium. The method includes: acquiring audio and text to be aligned; extracting at least one inter-word transition point from the audio; the inter-word transition point is located between two adjacent characters in the text content expressed by the audio; based on the inter-word transition point, adjusting the predicted transition probabilities of the text belonging to various phoneme states to obtain new transition probabilities; the probability adjustment is used to adjust the predicted transition probability between two adjacent characters in the text to a preset transition probability, the preset transition probability representing the transition probability between two adjacent characters belonging to different phoneme states; based on the new transition probabilities, aligning the audio and text to obtain aligned audio-text. This method can accelerate the audio-text alignment process and improve the naturalness and expressiveness of the aligned audio-text.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an audio text alignment method, a computer device, and a computer-readable storage medium. Background Technology

[0002] When a song is played in a music player application, the playback interface usually displays lyrics corresponding to the current playback progress. This effect requires time-aligned song text. Specifically, time-aligned song text aligns the vocals for each frame with the lyrics for each corresponding phoneme.

[0003] In related technologies, time-aligned song text is usually aligned manually. However, music playback applications add a large number of songs to their music libraries every day. If the time-aligned song text is obtained solely through manual alignment, the efficiency and quality of the generated time-aligned song text will be low. Summary of the Invention

[0004] Therefore, it is necessary to provide an audio / text alignment method, computer device, and computer-readable storage medium that can improve the efficiency and quality of audio / text alignment, addressing the aforementioned technical problems.

[0005] According to a first aspect of the present disclosure, an audio text alignment method is provided, comprising:

[0006] Get the audio and text to be aligned;

[0007] Extract at least one word transition point from the audio; the word transition point is located between two adjacent characters in the text content expressed by the audio;

[0008] Based on the inter-character transition points, the predicted transition probabilities of the text belonging to various phoneme states are adjusted to obtain new transition probabilities; the probability adjustment is used to adjust the predicted transition probability between two adjacent characters in the text to a preset transition probability, the preset transition probability representing the transition probability between two adjacent characters having different phoneme states.

[0009] Based on the new transition probability, the audio and the text are aligned to obtain the aligned audio and text.

[0010] In an exemplary embodiment, before adjusting the predicted transition probabilities of the text belonging to various phoneme states based on the inter-word transition points, the method further includes:

[0011] The audio is segmented into frames to obtain an audio frame sequence;

[0012] The text is processed by phoneme conversion based on the audio frame sequence to obtain a phoneme sequence; the number of phonemes in the phoneme sequence corresponds to the number of audio frames in the audio frame sequence.

[0013] Obtain the predicted transition probability of each phoneme in the phoneme sequence belonging to various phoneme states.

[0014] In an exemplary embodiment, obtaining the predicted transition probabilities of each phoneme in the phoneme sequence belonging to various phoneme states includes:

[0015] Obtain the predicted transition probabilities of each phoneme in the phoneme sequence belonging to various hidden Markov states; the predicted transition probabilities include inter-state transition probabilities, which represent the probability that in the phoneme sequence, the first hidden Markov state of the previous phoneme transitions to the second hidden Markov state of the next phoneme, and the first hidden Markov state and the second hidden Markov state are different.

[0016] The step of adjusting the predicted transition probabilities of the text belonging to various phoneme states based on the inter-word transition points includes:

[0017] Based on the inter-word transition points, the state transition probabilities of some phonemes in the phoneme sequence are adjusted.

[0018] In an exemplary embodiment, adjusting the state transition probabilities of some phonemes in the phoneme sequence based on the inter-word transition points includes:

[0019] In the audio, two adjacent target audio frames corresponding to the positions of the word transition points are identified;

[0020] Two target phonemes corresponding to the two target audio frames are determined in the phoneme sequence respectively;

[0021] The predicted state transition probability of the latter of the two target phonemes is adjusted to a preset transition probability.

[0022] In one exemplary embodiment, extracting at least one inter-word transition point from the audio includes:

[0023] The audio is subjected to fundamental frequency detection to determine the first type of transition point in the audio; the first type of transition point represents the fundamental frequency value transition between two adjacent audio frames in the audio.

[0024] Energy detection is performed on the audio to determine the second type of transition point in the audio; the second type of transition point represents the energy value transition between two adjacent audio frames in the audio.

[0025] Extract at least one transition point from the first type of transition point and the second type of transition point as an inter-word transition point.

[0026] In an exemplary embodiment, determining the first type of transition point in the audio includes:

[0027] Perform Fourier transform on the fundamental frequency value of each audio frame in the audio to determine the amplitude spectrum data for the audio.

[0028] When the amplitude change between two adjacent audio frames in the amplitude spectrum data exceeds a preset level, it is determined that there is a first type of jump point between the two adjacent audio frames;

[0029] The first type of transition points in the audio are collected to obtain a sequence of first type of transition points for the audio.

[0030] In an exemplary embodiment, determining the second type of transition point in the audio includes:

[0031] In each audio frame of the audio, when the degree of change between the energy values ​​of two adjacent audio frames exceeds a preset level, it is determined that there is a second type of jump point between the two adjacent audio frames;

[0032] By combining the second type of transition points in the audio, a sequence of second type transition points for the audio is obtained.

[0033] In an exemplary embodiment, extracting at least one transition point from the first type of transition points and the second type of transition points as an inter-word transition point includes:

[0034] Intersecting jump points are determined from each of the first type of jump points in the first type of jump point sequence and each of the second type of jump points in the second type of jump point sequence, and the intersecting jump points are used as inter-word jump points in the audio; the number of intersecting jump points is at least one.

[0035] In one exemplary embodiment, aligning the audio and the text based on the new transition probability to obtain aligned audio and text includes:

[0036] Obtain the emission probability of the audio belonging to various phoneme states;

[0037] The transmission probability and the new transition probability are decoded to determine the optimal phoneme state corresponding to each audio frame in the audio.

[0038] Based on the optimal phoneme state of each audio frame, the time correspondence between each audio frame and each phoneme in the text is determined;

[0039] Based on the time correspondence, the audio and the text are aligned to obtain the aligned audio and text.

[0040] According to a second aspect of the present disclosure, an audio text alignment device is provided, comprising:

[0041] The data acquisition unit is configured to acquire the audio and text to be aligned;

[0042] The word transition point extraction unit is configured to extract at least one word transition point from the audio; the word transition point is located between two adjacent characters in the text content expressed by the audio.

[0043] The probability adjustment unit is configured to perform probability adjustment on the predicted transition probabilities of the text belonging to various phoneme states based on the inter-character transition points, so as to obtain new transition probabilities; the probability adjustment is used to adjust the predicted transition probability between two adjacent characters in the text to a preset transition probability, the preset transition probability representing the transition probability between two adjacent characters having different phoneme states.

[0044] The alignment unit is configured to perform alignment of the audio and the text based on the new transition probability to obtain aligned audio and text.

[0045] According to a third aspect of the present disclosure, a computer device is provided, comprising:

[0046] processor;

[0047] Memory for storing the executable instructions of the processor;

[0048] The processor is configured to execute the executable instructions to implement the audio text alignment method as described in any of the preceding claims.

[0049] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, the computer-readable storage medium including a computer program that, when executed by a processor of a computer device, enables the computer device to perform the audio text alignment method as described in any of the preceding claims.

[0050] According to a fifth aspect of the present disclosure, a computer program product is provided, the computer program product including program instructions that, when executed by a processor of a computer device, enable the computer device to perform the audio text alignment method as described in any of the preceding claims.

[0051] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:

[0052] The method first acquires the audio and text to be aligned; then, it extracts at least one inter-word transition point from the audio; the inter-word transition point is located between two adjacent characters in the text content expressed by the audio; then, based on the inter-word transition point, it adjusts the predicted transition probabilities of the text belonging to various phoneme states to obtain new transition probabilities; wherein, the probability adjustment is used to adjust the predicted transition probability between two adjacent characters in the text to a preset transition probability, the preset transition probability representing the transition probability between two adjacent characters having different phoneme states; finally, based on the new transition probabilities, the audio and text are aligned to obtain the aligned audio-text. In this way, on the one hand, unlike existing technologies, audio and text are aligned by using the transition probabilities of the phoneme states corresponding to the audio and text to be aligned, thus optimizing the audio-text alignment process, accelerating the audio-text alignment speed, and reducing the consumption of manpower and time costs; on the other hand, by using the inter-word transition points in the audio to adjust the predicted transition probabilities of the phoneme states corresponding to the text, and aligning the audio and text based on the transition probabilities, the naturalness and expressiveness of the aligned audio-text can be improved, resulting in better quality and display effect of the aligned audio-text.

[0053] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0054] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0055] Figure 1 This is an application environment diagram illustrating an audio text alignment method according to an exemplary embodiment.

[0056] Figure 2 This is a flowchart illustrating an audio text alignment method according to an exemplary embodiment.

[0057] Figure 3 This is a flowchart illustrating a step for extracting word transition points according to an exemplary embodiment.

[0058] Figure 4 This is a flowchart illustrating a step for obtaining a predicted transition probability according to an exemplary embodiment.

[0059] Figure 5 This is a flowchart illustrating a step of adjusting the predicted transition probability according to an exemplary embodiment.

[0060] Figure 6This is a flowchart illustrating a step of aligning audio and text according to another exemplary embodiment.

[0061] Figure 7 This is a flowchart illustrating an audio text alignment method according to another exemplary embodiment.

[0062] Figure 8 This is a module diagram illustrating an audio text alignment method according to another exemplary embodiment.

[0063] Figure 9 This is a schematic diagram of a state transition probability diagram according to an exemplary embodiment.

[0064] Figure 10 This is a block diagram illustrating an audio text alignment device according to an exemplary embodiment.

[0065] Figure 11 This is a block diagram illustrating a computer device for audio text alignment according to an exemplary embodiment.

[0066] Figure 12 This is a block diagram illustrating a computer-readable storage medium for audio-text alignment according to an exemplary embodiment.

[0067] Figure 13 This is a block diagram illustrating a computer program product for audio text alignment according to an exemplary embodiment. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0069] The term "and / or" in the embodiments of this application refers to any and all possible combinations including one or more of the associated listed items. It should also be noted that, when used in this specification, "including / comprising" specifies the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, and / or components and / or groups thereof.

[0070] The terms "first," "second," etc., used in this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0071] Furthermore, although the terms "first," "second," etc., are used repeatedly in this application to describe various operations (or various elements, or various applications, or various instructions, or various data), these operations (or elements, or applications, or instructions, or data) should not be limited by these terms. These terms are only used to distinguish one operation (or element, or application, or instruction, or data) from another operation (or element, or application, or instruction, or data). For example, a first hidden Markov state can be called a second hidden Markov state, and a second hidden Markov state can be called a first hidden Markov state; the only difference is the scope they encompass, but it does not depart from the scope of this application. Both the first and second hidden Markov states are sets of various hidden Markov states, but they are not the same set of hidden Markov states.

[0072] The audio text alignment method provided in this application embodiment can be applied to, for example, Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a communication network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on other network servers.

[0073] In some embodiments, reference Figure 1 First, server 104 acquires the audio and text to be aligned. Then, server 104 extracts at least one inter-word transition point from the audio. The inter-word transition point is located between two adjacent characters in the text content expressed by the audio. Then, based on the inter-word transition point, server 104 adjusts the predicted transition probabilities of the text belonging to various phoneme states to obtain new transition probabilities. The probability adjustment is used to adjust the predicted transition probability between two adjacent characters in the text to a preset transition probability, which represents the transition probability between two adjacent characters with different phoneme states. Finally, server 104 aligns the audio and text based on the new transition probabilities to obtain the aligned audio and text.

[0074] In some embodiments, terminal 102 (such as a mobile terminal or a fixed terminal) can be implemented in various forms. Terminal 102 can be a mobile terminal, including mobile phones, smartphones, laptops, portable handheld devices, personal digital assistants (PDAs), tablet computers (PADs), etc., capable of aligning audio and text based on predicted transition probabilities of the inter-word transition points of the audio to be aligned and the text belonging to various phoneme states. Terminal 102 can also be a fixed terminal, including automated teller machines (ATMs), automated kiosks, digital TVs, desktop computers, fixed-line computers, etc., capable of aligning audio and text based on predicted transition probabilities of the inter-word transition points of the audio to be aligned and the text belonging to various phoneme states.

[0075] Hereinafter, it is assumed that terminal 102 is a fixed terminal. However, those skilled in the art will understand that, if there are operations or elements specifically designed for mobile purposes, the construction according to the embodiments disclosed in this application can also be applied to mobile type terminal 102.

[0076] In some embodiments, the data processing component running on server 104 may load any of the various additional server applications and / or middleware applications being executed, such as HTTP (Hypertext Transfer Protocol), FTP (File Transfer Protocol), CGI (Common Gateway Interface), RDBMS (Relational Database Management System), etc.

[0077] In some embodiments, server 104 may be implemented using a standalone server or a server cluster consisting of multiple servers. Server 104 may be adapted to run one or more application services or software components that provide the terminal 102 described in the foregoing disclosure.

[0078] In some embodiments, the application service may include a service interface that provides users with a selection of audio and text to be aligned, as well as corresponding program services, etc. The software component may include, for example, an application (SDK) or client (APP) that has the function of aligning audio and text based on the inter-word transition points of the audio to be aligned and the predicted transition probabilities of the text belonging to various phoneme states.

[0079] In some embodiments, the application or client provided by server 104 with audio and text alignment capabilities includes a portal port that provides one-to-one application services to users in the foreground and multiple business systems that perform data processing in the background, so as to extend the relevant functional applications in the process of generating music animation videos to the APP or client, so that users can use and access the functions associated with audio and text alignment anytime and anywhere.

[0080] In some embodiments, the audio and text alignment function of an app or client can be a computer program running in user mode to perform one or more specific tasks. It can interact with the user and has a visual user interface. The app or client can include two parts: a graphical user interface (GUI) and an engine, which together provide users with various application services in the form of a user interface in a digital client system.

[0081] In some embodiments, users can input corresponding code data or control parameters into the APP or client through a preset input device or automatic control program to execute application services of the computer program in the server 104 and display application services in the user interface.

[0082] In some embodiments, the operating system running the app or client may include various versions of Microsoft... and / or Linux operating system, various commercial or similar Operating systems (including but not limited to various GNU / Linux operating systems, Google) (etc.) and / or mobile operating systems, such as Operating systems, as well as other online or offline operating systems, are not specifically limited here.

[0083] In some embodiments, such as Figure 2 As shown, an audio text alignment method is provided, which is applied to... Figure 1 Taking server 104 as an example, the method includes the following steps:

[0084] Step S11: Obtain the audio and text to be aligned.

[0085] In some embodiments, the server obtains the audio to be aligned and the corresponding text transmitted by the user account from the terminal application (such as a mobile phone, tablet, etc.).

[0086] The audio to be aligned can be a released official version of a music song, or a local song recorded by the terminal application (e.g., a live song recorded offline by the terminal application and a web song recorded online).

[0087] In some embodiments, the audio to be aligned includes dry audio for representing a sequence of lyrics. The dry audio consists of multiple consecutive audio frames.

[0088] In some embodiments, the server may first acquire the input audio transmitted by the user account (e.g., a live song recorded offline by a terminal application), and then pass the input audio into a preset audio-accompaniment separation model (e.g., the Spleeter algorithm) to separate the dry audio from the accompaniment, thereby extracting the dry audio of the input audio consisting of multiple consecutive audio frames, and using the dry audio as the audio to be aligned. Then, a preset lyrics parsing model is used to extract lyrics from the dry audio of each audio frame to obtain the lyrics sequence of the input audio, and the lyrics sequence is used as the text to be aligned.

[0089] As an example, the Spleeter algorithm first re-segments the song segment and projects the segmented song into a low-dimensional space to reduce dimensionality and compress the information data, thereby extracting audio depth features related to the song. Then, a multilayer perceptron based on an MLP is used to classify the audio depth features, obtaining audio depth features for the dry audio and the accompaniment. Finally, the compressed low-dimensional features are restored to the original dimensions of the dry audio and accompaniment audio to extract the dry audio, i.e., the audio to be aligned, from the input audio.

[0090] As an example, the lyrics parsing model first extracts the text information of the dry audio from each audio frame using a pre-defined speech recognition algorithm. This speech recognition algorithm can be based on Dynamic Time Warping, Hidden Markov Models (HMMs) based on parametric models, artificial neural networks, or a hybrid algorithm, etc. Then, the lyrics parsing model performs word segmentation on the text information of the dry audio to obtain the words to be recognized. Finally, it matches the words to be recognized in the dry audio with the lyrics text information corresponding to multiple songs to obtain the dry audio lyrics sequence, i.e., the text to be aligned.

[0091] Step S12: Extract at least one word transition point from the audio.

[0092] In one embodiment, the server performs Voice Activity Detection (VAD) on the audio to be aligned in order to extract at least one word transition point from the audio.

[0093] The word transition point is located between two adjacent characters in the text content expressed by the audio; that is, there is a word transition point between every two characters expressed in the audio to be aligned.

[0094] In some embodiments, sound activity event detection includes fundamental frequency detection or energy detection of the audio to be aligned, so as to determine the inter-word transition points between characters expressed in the audio to be aligned based on the detected fundamental frequency data or energy data of the audio to be aligned.

[0095] Step S13: Based on the inter-word transition points, adjust the predicted transition probabilities of the text belonging to various phoneme states to obtain new transition probabilities.

[0096] In one embodiment, probability adjustment is used to adjust the predicted transition probability between two adjacent characters in the text to a preset transition probability.

[0097] In one embodiment, the preset transition probability represents the transition probability between two adjacent characters that have different phoneme states.

[0098] The preset transition probability can be an absolute probability, meaning the transition probability between the predicted states of the next character and the next character in two adjacent characters is 100%. For example, if the two adjacent characters are character 1 and character 2, and character 2 is the next character, before probability adjustment, the transition probability of character 2 is 60%; after probability adjustment, the transition probability of character 2 is 0% or 100%, so that when character 1 is in phoneme state 1, the phoneme state of character 2 is a different phoneme state from phoneme state 1. In other embodiments, the preset transition probability can also be configured to other probabilities as needed, which is not specifically limited here.

[0099] In some embodiments, before adjusting the predicted transition probabilities of the text belonging to various phoneme states, the server needs to obtain the predicted transition probabilities of each phoneme in the phoneme sequence corresponding to the text belonging to various phoneme states. This method specifically includes: obtaining the predicted transition probabilities of each phoneme in the phoneme sequence belonging to various hidden Markov states.

[0100] In some embodiments, a phoneme state refers to the Hidden Markov State of each phoneme in a phoneme sequence. Each phoneme may contain at least three Hidden Markov States: a pre-phoneme state, a mid-phoneme state, and a post-phoneme state. The predicted transition probability of a phoneme state is the transition probability of a phoneme belonging to each of the various phoneme states.

[0101] The predicted transition probabilities include inter-state transition probabilities, which represent the probability in a phoneme sequence of transitioning from a first Hidden Markov state of a preceding phoneme to a second Hidden Markov state of a following phoneme, wherein the first Hidden Markov state and the second Hidden Markov state are different. For example, in the above embodiment, when character 1 is in phoneme state 1, the phoneme state of character 2 is a different phoneme state from phoneme state 1.

[0102] In some embodiments, the server adjusts the predicted transition probabilities of the text belonging to various phoneme states based on inter-word transition points. Specifically, this includes adjusting the state transition probabilities of some phonemes in the phoneme sequence corresponding to the text based on inter-word transition points.

[0103] Specifically, based on the inter-word transition points, the server determines which part of the phoneme pairs (i.e., the preceding phoneme and the following phoneme) in the phoneme sequence corresponding to the text has an inter-word transition point, and then adjusts the state transition probability of the following phoneme in that part of the phoneme pair to a preset transition probability.

[0104] Step S14: Based on the new transition probability, align the audio and text to obtain the aligned audio and text.

[0105] In some embodiments, the server first extracts audio features (such as Mel-spectral signatures and perceptual linear prediction features) from each audio frame in the audio, and then calculates the emission probabilities of each audio frame's audio features belonging to various Hidden Markov states based on a preset Hidden Markov model. Further, the server inputs the emission probabilities of the audio and the new transition probabilities into a Hidden Markov-Gaussian mixture model to perform forced alignment of the audio text, obtaining the aligned audio text.

[0106] In the above-mentioned audio-text alignment process, the server first obtains the audio and text to be aligned; then, it extracts at least one inter-word transition point from the audio; the inter-word transition point is located between two adjacent characters in the text content expressed by the audio; then, based on the inter-word transition point, the predicted transition probabilities of the text belonging to various phoneme states are adjusted to obtain new transition probabilities; wherein, the probability adjustment is used to adjust the predicted transition probability between two adjacent characters in the text to a preset transition probability, the preset transition probability representing the transition probability between two adjacent characters having different phoneme states; finally, based on the new transition probabilities, the audio and text are aligned to obtain the aligned audio-text. In this way, on the one hand, unlike existing technologies, audio and text are aligned by using the transition probabilities of the phoneme states corresponding to the audio and text to be aligned, thus optimizing the audio-text alignment process, accelerating the audio-text alignment speed, and reducing the consumption of manpower and time costs; on the other hand, by using the inter-word transition points in the audio to adjust the predicted transition probabilities of the phoneme states corresponding to the text, and aligning the audio and text based on the transition probabilities, the naturalness and expressiveness of the aligned audio-text can be improved, resulting in better quality and display effect of the aligned audio-text.

[0107] Those skilled in the art will understand that the methods disclosed in the above-described specific embodiments can be implemented in more concrete ways. For example, the implementation described above, in which the server aligns audio and text based on a new transition probability to obtain aligned audio and text, is merely illustrative.

[0108] For example, the server may extract at least one inter-word transition point from the audio; the server may adjust the predicted transition probabilities of the text belonging to various phoneme states based on the inter-word transition points, etc. This is just one set of methods. In actual implementation, there may be other ways of dividing it. For example, the inter-word transition points of the audio and the predicted transition probabilities of the text may be combined or integrated into another system, or some features may be ignored or not executed.

[0109] In one exemplary embodiment, see Figure 3 , Figure 3 This is a flowchart illustrating an embodiment of extracting inter-word transition points in this application. In step S12, the process by which the server extracts at least one inter-word transition point from the audio can be implemented in the following way:

[0110] Step S121: Perform fundamental frequency detection on the audio to determine the first type of transition point in the audio.

[0111] In one embodiment, a first type of transition point represents a transition in the fundamental frequency value between two adjacent audio frames in the audio.

[0112] The first type of transition point is the inter-frame transition point between adjacent audio frames.

[0113] As an example, the fundamental frequency of the first audio frame A1 is A2, and the fundamental frequency of the second audio frame B1 is B2. The first audio frame A1 and the second audio frame B1 are a pair of adjacent audio frames. Specifically, if the distance between the fundamental frequency values ​​A2 and B2 is less than a preset distance threshold, then there is no first-type transition point between the first audio frame A1 and the second audio frame B1; if the distance between the fundamental frequency values ​​A2 and B2 is greater than or equal to the preset distance threshold, then there is a first-type transition point between the first audio frame A1 and the second audio frame B1.

[0114] In other implementations, the server may determine the first type of transition point in the audio in the following manner:

[0115] Step 1: Perform Fourier transform on the fundamental frequency value of each audio frame in the audio to determine the amplitude spectrum data for the audio.

[0116] In some embodiments, the server first performs a Fourier transform on the fundamental frequency value of each audio frame to determine the complex spectrum of the corresponding transformed audio. The amplitude spectrum data includes phase frequency spectrum data and amplitude frequency spectrum data.

[0117] In other embodiments, the server may perform Fourier transform processing on the fundamental frequency values ​​of each audio frame, or it may perform processing on the audio frames including Fast Fourier Transform, (improved) Discrete Cosine Transform, Wavelet Transform, etc., without specific limitations.

[0118] As an example, after the m-th audio frame undergoes Fourier transform processing, the complex spectrum of the corresponding m-th frame is represented by the formula:

[0119] Where K represents the frame length (where the frame length is exactly equal to the number of points in the Fourier transform).

[0120] The complex spectral features corresponding to the audio frame are represented by X(k), where i represents the index of the signal in the nth frame, w(i) represents the windowing function, and j is a preset coefficient.

[0121] Furthermore, the server performs feature extraction processing on the complex spectrum of the audio frame to obtain phase frequency spectrum data and amplitude frequency spectrum data.

[0122] As an example, amplitude-frequency spectrum data is represented based on the formula: A X (k) = ||X(k)||, where ||·|| represents the modulo operation of a complex number.

[0123] Among them, the phase frequency spectrum data is represented by the formula: Φ X (k)=atan2(X r (k),X i (k)), where X r (k),X i (k) represent the real and imaginary parts of the complex spectrum, respectively, i.e.: X(k) = X r (k)+jX i (k), where j represents an imaginary number.

[0124] Step 2: When the amplitude change between two adjacent audio frames in the amplitude spectrum data exceeds a preset level, a first type of transition point is determined to exist between the two adjacent audio frames.

[0125] As an example, the amplitude spectrum data of the first audio frame A1 is A3, and the amplitude spectrum data of the second audio frame B1 is B3. The first audio frame A1 and the second audio frame B1 are a pair of adjacent audio frames. Specifically, if the distance between amplitude spectrum data A3 and amplitude spectrum data B3 is less than a preset distance threshold, then there is no first-type transition point between the first audio frame A1 and the second audio frame B1; if the distance between amplitude spectrum data A3 and amplitude spectrum data B3 is greater than or equal to the preset distance threshold, then there is a first-type transition point between the first audio frame A1 and the second audio frame B1.

[0126] In other embodiments, step two can also be: when the phase frequency change between two adjacent audio frames in the phase frequency spectrum data exceeds a preset level, it is determined that there is a first type of jump point between the two adjacent audio frames.

[0127] As an example, the phase frequency spectrum data of the first audio frame A1 is A4, and the phase frequency spectrum data of the second audio frame B1 is B4. The first audio frame A1 and the second audio frame B1 are a pair of adjacent audio frames. Specifically, if the distance between phase frequency spectrum data A4 and phase frequency spectrum data B4 is less than a preset distance threshold, then there is no first-type transition point between the first audio frame A1 and the second audio frame B1; if the distance between phase frequency spectrum data A4 and phase frequency spectrum data B4 is greater than or equal to the preset distance threshold, then there is a first-type transition point between the first audio frame A1 and the second audio frame B1.

[0128] Step 3: Collect the first type of transition points in the audio to obtain the sequence of first type of transition points for the audio.

[0129] In one embodiment, the server can collect any one of the following: jump points that jump between the fundamental frequency values ​​of the audio frame, jump points that jump between the phase frequency spectrum data of the audio frame, and jump points that jump between the amplitude spectrum data of the audio frame. The server can collect the jump points of each of the three to obtain a first type of jump point sequence. No specific limitation is made here.

[0130] Step S122: Perform energy detection on the audio to determine the second type of transition point in the audio.

[0131] In one embodiment, the second type of transition point represents a jump in energy value between two adjacent audio frames in the audio. Specifically, the second type of transition point is an inter-frame transition point between adjacent audio frames.

[0132] In one implementation, the server can determine the second type of transition point in the audio in the following manner:

[0133] Step 1: In each audio frame, when the degree of change between the energy values ​​of two adjacent audio frames exceeds a preset level, a second type of transition point is determined to exist between the two adjacent audio frames.

[0134] As an example, the energy value of the first audio frame C1 is C2, and the energy value of the second audio frame D1 is D2. The first audio frame C1 and the second audio frame D1 are a pair of adjacent audio frames. Specifically, if the distance between energy values ​​C2 and D2 is less than a preset distance threshold, then there is no second-type transition point between the first audio frame C1 and the second audio frame D1; if the distance between energy values ​​C2 and D2 is greater than or equal to the preset distance threshold, then there is a second-type transition point between the first audio frame C1 and the second audio frame D1.

[0135] Step 2: Collect the second type of transition points in the audio to obtain the second type of transition point sequence for the audio.

[0136] In one embodiment, the server can collect transition points where there are jumps between the energy values ​​of audio frames to obtain a first type of transition point sequence.

[0137] Step S123: Extract at least one transition point from the first type of transition point and the second type of transition point as an inter-word transition point.

[0138] In one embodiment, the server determines the intersection point from each of the first type of jump points in the first type of jump point sequence and each of the second type of jump points in the second type of jump point sequence, and uses the intersection point as the inter-word jump point of the audio.

[0139] In one embodiment, an intersection transition point represents an audio transition between two adjacent characters in the character content expressed by the audio.

[0140] As an example, the first type of transition point sequence includes five transition points a1, a2, a3, a4, and a5; the second type of transition point sequence includes five transition points b1, b2, b3, b4, and b5. Here, a3 and b3 both represent transition points between audio frames Z1 and Z2, and a4 and b5 both represent transition points between audio frames Z3 and Z4. The server uses P1 to represent the set of a3 and b3, and P2 to represent the set of a4 and b5. Both P1 and P2 are considered intersection transition points, meaning that P1 and P2 are both inter-word transition points in the audio.

[0141] The number of intersection transition points is at least one. When the number of intersection transition points is 0, the server reselects the first type of transition point and the second type of transition point of the audio to redetermine the intersection transition point.

[0142] In one exemplary embodiment, see Figure 4 , Figure 4 This is a flowchart illustrating an embodiment of obtaining predicted transition probabilities in this application. Before step S13, before the server adjusts the predicted transition probabilities of the text belonging to various phoneme states based on inter-word transition points, it can also be implemented in the following way:

[0143] Step a1: Perform a frame segmentation operation on the audio to obtain an audio frame sequence.

[0144] In one embodiment, the server first performs a frame segmentation operation on the audio, and then performs a windowing operation on the audio to obtain an audio frame sequence.

[0145] The framing operation can be represented as: x n (i) = x(n·M+i), where n represents the signal of the nth frame, M represents the frame shift, and i represents the index of the signal within the nth frame, with values ​​ranging from 0, 1, 2, ..., L-1, where L represents the frame length. This application uses a frame length t. frmhop =0.02s (seconds) frame shift t frmhop = 0.01s (seconds).

[0146] The windowing operation is represented by: xw n (i)=x n (i)·w(i), where w(i) represents the window function and i represents the i-th sample point. In one example, a Hanning window can be used.

[0147] Step a2: Perform phoneme conversion on the text based on the audio frame sequence to obtain a phoneme sequence.

[0148] In one embodiment, the server inputs the text to be aligned into a preset timbre recognition network for phoneme segmentation, or a music theorist manually segments the text to be aligned to obtain corresponding multiple phoneme segments.

[0149] In one embodiment, the number of phonemes in the phoneme sequence corresponds to the number of audio frames in the audio frame sequence. That is, the timbre recognition network segments the phonemes in the phoneme sequence according to the number of audio frames in the audio frame sequence, or the music theorist segments the phonemes in the phoneme sequence according to the number of audio frames in the audio frame sequence, so as to obtain multiple phoneme segments with the same number of audio frames as the audio frame sequence.

[0150] Step a3: Obtain the predicted transition probabilities of each phoneme in the phoneme sequence belonging to various phoneme states.

[0151] In some embodiments, the server inputs each phoneme in the phoneme sequence into a probability prediction model for probability prediction, obtaining the predicted transition probabilities of each phoneme belonging to various phoneme states. This probability prediction model can be, for example, a Hidden Markov-Gaussian Mixture Model (HMM-GMM), which, through a series of algorithmic optimizations and iterations, outputs the predicted transition probabilities of each phoneme belonging to various phoneme states.

[0152] In some embodiments, a phoneme state refers to the Hidden Markov State of each phoneme in a phoneme sequence. Each phoneme may contain at least three Hidden Markov States: a pre-phoneme state, a mid-phoneme state, and a post-phoneme state. The predicted transition probability of a phoneme state is the transition probability of a phoneme belonging to each of the various phoneme states.

[0153] As an example, during actual speech, people do not pronounce each word separately. Many pronunciations are connected together, which is particularly common in English. Therefore, even for the same phoneme, if the preceding and following phonemes are different, the pronunciation of the final phoneme will also be different. Since the pronunciation of a phoneme is affected by its preceding and following phonemes, the phoneme can be combined with its preceding and following phonemes, and three states are used to represent a phoneme. For example, the phoneme "en" can be represented by "zh-en", "en", and "en-h". Therefore, in some embodiments, when the language types of the text to be aligned and the audio to be aligned are Chinese, the above-mentioned three-phoneme splitting can be to split the characters in the text to be aligned into three-phoneme expressions. In some embodiments, the three-phoneme splitting can be based on the result of single-phoneme splitting. For example, after splitting the character "今" in the text to be aligned into "jin", based on this result, the "j" can be further split into three-phoneme expressions of "sli-j j j-in". After obtaining the three-phoneme expressions, the three phonemes can be clustered through unsupervised learning, so that the same or similar three phonemes are grouped into one category. Similar three phonemes refer to those with similar pronunciations. For example, for "ning" and "ming", the pronunciation of the vowel "ing" is not the same in these two pronunciations. However, since the initial consonants "n" and "m" have similar pronunciations, the pronunciations of the three phonemes "n-ing" and "m-ing" are similar and can be grouped into one category. After the three-phoneme clustering, grouping the same or similar three phonemes into one category can reduce the number of features to be processed during the audio-text alignment process, thereby accelerating the alignment speed.

[0154] In some embodiments, predicting the transition probability includes the self-transition probability and the inter-state transition probability. Among them, the self-transition probability represents the probability that a state transitions to its own state; the inter-state transition probability represents the probability that a state transitions to other states.

[0155] As an example, audio frame A1 and audio frame A2 are a pair of adjacent audio frames, phoneme B1 and phoneme B2 are a pair of adjacent phonemes, and audio frame A1 corresponds to phoneme B1 in their respective sequences (i.e., the audio frame sequence and the phoneme sequence), and audio frame A2 corresponds to phoneme B2 in their respective sequences. Among them, the probability that phoneme B1 and phoneme B2 are in the same phoneme state is the self-transition probability (i.e., phoneme B1 is in state 1 at audio frame A1 and phoneme B2 is also in state 1 at audio frame A2), and the probability that phoneme B1 and phoneme B2 are in different phoneme states is the inter-state transition probability (i.e., phoneme B1 is in state 1 at audio frame A1 and phoneme B2 is in state 2 at audio frame A and states 1 and 2 are different).

[0156] In an exemplary embodiment, refer to Figure 5 , Figure 5This is a flowchart illustrating an embodiment of adjusting predicted transition probabilities in this application. In step S13, the server adjusts the state transition probabilities of some phonemes in the phoneme sequence based on inter-word transition points. This can be implemented in the following way:

[0157] Step S131: In the audio, identify two adjacent target audio frames that correspond to the positions of the word transition points.

[0158] Step S132: Identify the two target phonemes corresponding to the two target audio frames in the phoneme sequence.

[0159] Step S133: Adjust the predicted state transition probability of the latter target phoneme among the two target phonemes to the preset transition probability.

[0160] As an example, if there are word transition points X1, X2, and X3 in the audio, the server first determines the audio frames a1 and a2 corresponding to X1, a3 and a4 corresponding to X2, and a5 and a6 corresponding to X3 from the audio frame sequence. Then, the server determines the phonemes b1 and b2 corresponding to a1 and a2, b3 and b4 corresponding to a3 and a4, and b5 and b6 corresponding to a5 and a6 from the phoneme sequence. Finally, the server adjusts the predicted state transition probability of phoneme b2, the predicted state transition probability of phoneme b4, and the predicted state transition probability of phoneme b6 to the preset transition probability.

[0161] The preset transition probability can be an absolute probability, meaning the predicted state transition probability of the latter of two target phonemes is 100%. For example, before probability adjustment, the self-transition probability of phoneme a6 is 60%, and the state transition probability is 40%; after probability adjustment, the self-transition probability of phoneme a6 is 0%, and the state transition probability is 100%.

[0162] In one exemplary embodiment, see Figure 6 , Figure 6 This is a schematic flowchart illustrating an embodiment of the audio and text alignment in this application. In step S14, the server aligns the audio and text based on the new transition probability to obtain the aligned audio and text. This process can be implemented in the following way:

[0163] Step S141: Obtain the emission probability of the audio belonging to various phoneme states.

[0164] In some embodiments, the server extracts the MFCC (Melch Cepstral Parameter) features of each audio frame, and then integrates the emission probabilities of the MFCC features of each audio frame belonging to various hidden Markov states to obtain the emission probability matrix P corresponding to each audio frame.

[0165] Wherein, the emission probability matrix P is based on the formula Characterization.

[0166] Where pij represents the probability that the i-th frame belongs to the j-th state (i = 1, ..., n; j = 1, ..., m), and k is a number belonging to j, i.e., j = 1, ..., k, ..., m.

[0167] In other embodiments, the server may also extract the PLP (Perceptual Linear Predictive) features of each audio frame, and then integrate the emission probabilities of each audio frame belonging to various hidden Markov states to obtain the emission probability matrix P corresponding to each audio frame.

[0168] Step S142: Decode the emission probability and the new transition probability to determine the optimal phoneme state corresponding to each audio frame in the audio.

[0169] In some embodiments, the server performs Viterbi decoding on the emission probability matrix and the new transition probability matrix using a GMM-HMM framework to obtain the optimal Hidden Markov state for each audio frame. The optimal Hidden Markov state is the optimal solution for the audio frame searched in the text space.

[0170] Step S143: Based on the optimal phoneme state of each audio frame, determine the time correspondence between each audio frame and each phoneme in the text.

[0171] Step S144: Based on the time correspondence, align the audio and text to obtain the aligned audio and text.

[0172] In some embodiments, the server converts the optimal phoneme state corresponding to each audio frame in the audio into a phoneme sequence, and then into a word sequence to determine the time correspondence between each audio frame and each phoneme in the text. Finally, the server aligns each phoneme with each audio frame according to the time correspondence between each phoneme in the text and each audio frame to obtain the aligned audio text. The time correspondence between audio frames and phonemes is used to represent the timestamp of the text.

[0173] To more clearly illustrate the audio text alignment method provided in this disclosure, a specific embodiment will be used to describe the audio text alignment method in detail below. In an exemplary embodiment, reference is made to... Figure 7 and Figure 8 , Figure 7 This is a flowchart illustrating an audio text alignment method according to another exemplary embodiment. Figure 8 This is a block diagram illustrating an audio text alignment method according to another exemplary embodiment. The audio / video synthesis method is used in server 104 and specifically includes the following:

[0174] Step S21: Obtain the vocal audio and the corresponding lyrics text from the song audio.

[0175] Step S22: Perform frame segmentation on the human voice audio to obtain an audio frame sequence, and extract audio features from each audio frame in the audio frame sequence.

[0176] Among them, the audio features are the MFCC features corresponding to the non-audio features.

[0177] Step S23: Input the MFCC features of each audio frame into the acoustic model DNN for probability prediction to obtain the emission probabilities of each audio frame's MFCC features belonging to various HMM states.

[0178] Step S24: Integrate the emission probabilities of the MFCC features of each audio frame belonging to various HMM states to obtain the emission probability matrix P corresponding to each audio frame.

[0179] Wherein, the emission probability matrix P is based on the formula Characterization.

[0180] Where pij represents the probability that the i-th frame belongs to the j-th state (i = 1, ..., n; j = 1, ..., m), and k is a number belonging to j, i.e., j = 1, ..., k, ..., m.

[0181] Step S25: First, convert the lyrics text into a phoneme sequence, and then convert the phoneme sequence into a state sequence belonging to various HMM states.

[0182] The phoneme sequence is represented by ph1, ph2, ..., phn (where n is the length of the phoneme sequence); the state sequence is represented by S1, S2, ..., Sn (where n is the length of the state sequence).

[0183] Each state in the state sequence includes a self-transition probability and an inter-state transition probability.

[0184] Among them, the self-transition probability represents the state transitioning to its own state with a certain probability; the inter-state transition probability represents the state transitioning to other states with a certain probability.

[0185] The HMM states have a preset number of types (e.g., 26 types), each phoneme corresponds to multiple audio frames (each audio frame corresponds to one state), and the state of each frame is determined according to its probability relative to all states.

[0186] Step S26: Generate a state transition matrix S corresponding to each audio frame based on the state sequence.

[0187] The state transition matrix S is represented based on the state transition probability graph. For example... Figure 9 As shown, Figure 9 This is a schematic diagram of a state transition probability graph as an exemplary embodiment. The horizontal axis of the state transition probability graph represents the number of audio frames or phoneme positions (i.e., sequence T(1,2,3,4,5,6,7) represents the number of audio frames in the audio frame sequence, and sequence X(X1,X2,X3,X4,X5,X6,X7) represents the number of phonemes in the phoneme sequence); the vertical axis represents the probability of all types of states (for example, there are 26 states, but the diagram illustrates 5 states: S(S0, S1, S2, S3, S4)). Sequence a... 1 (a 11 a 22 a 33 The sequence a represents the self-transition probability of each phoneme. 2 (a 01 a 12 a 23 a 34 The state transition probability of each phoneme is represented by the thick line in the figure, which is the optimal probability path determined based on the optimal probability path, that is, the optimal state corresponding to each frame.

[0188] Step S27: Perform fundamental frequency detection on the human voice audio to determine the first type of transition point sequence in the audio frame sequence.

[0189] Among them, fundamental frequency detection is based on the autocorrelation function (or the Y-in algorithm). conduct.

[0190] Where Ri(k) is the transition point between two adjacent audio frames in the human voice audio; xi(m) is the fundamental frequency value of the i-th frame; k is the period of the function xi(m). When k is a multiple of xi(m), xi(m) reaches its maximum value, which corresponds to the transition point value between two adjacent audio frames; the first type of transition point sequence is characterized based on (V1, V2, ..., Vi, ..., Vn).

[0191] In this context, a transition point represents a jump in the HMM state of the corresponding audio frame relative to its previous audio frame in the state transition probability graph. That is, the corresponding audio frame has a higher vertical axis value than its previous audio frame in the state transition probability graph. For example... Figure 9x1 and x2 are the same state, while x3 and x4 are different states, meaning there is a state transition between x3 and x4.

[0192] Step S28: Perform short-time energy detection on the audio frame sequence to determine the second type of transition point sequence in the audio frame sequence.

[0193] Short-time energy detection is used to calculate the energy value of each audio frame.

[0194] Among them, when the change between the energy values ​​of two adjacent audio frames exceeds the threshold, the human voice audio is considered to be at an energy jump point; the second type of jump point sequence is characterized based on (D1, D2, ..., Di, ..., Dn).

[0195] Among them, fundamental frequency detection and energy detection are both types of VAD detection.

[0196] Step S29: Take the intersection of the first type of transition point sequence and the second type of transition point sequence to determine the third type of first type of transition point sequence and the second type of transition point sequence in the audio frame sequence.

[0197] Among them, the third type of transition point sequence is represented by (K1,K2,…,Ki,…,Kn).

[0198] Among them, the first type of jump point sequence and the second type of jump point sequence are the sets of jump points between two audio frames that have corresponding jump points, and the third type of jump point sequence is the set of jump points at the intersection of the first type of jump point sequence and the second type of jump point sequence, and the set of jump points at the intersection serves as the set of jump points between every two words.

[0199] Step S30: Modify the state transition probability of the audio frame corresponding to each transition point in the third type of transition point sequence to 1 to obtain a new state transition matrix S'.

[0200] In this context, a state transition probability of 1 between audio frames indicates that there must be a state jump between two audio frames between two words. The new state transition matrix S' is represented based on the new state transition probability graph.

[0201] Within each word, there may be state transitions between different audio frames, or there may be no state transitions.

[0202] Step S31: Viterbi decoding is performed on the emission probability matrix P and the transition probability matrix S' using the GMM-HMM framework to obtain the optimal HMM state corresponding to each audio frame.

[0203] The optimal HMM state is the optimal solution for searching the audio frame in the lyrics text space.

[0204] Step S32: Determine the decoded state sequence based on the optimal HMM state.

[0205] The decoded state sequence represents the optimal state path on the state transition probability graph, that is, the recognition result of the state transition probability graph with respect to the state path.

[0206] Step S33: Convert the state sequence into a phoneme sequence, and then into a word sequence, in order to determine the time correspondence between each audio frame in the human voice audio and each lyric in the lyrics sequence.

[0207] The time correspondence between audio frames and lyrics is used to represent the timestamps of the lyrics.

[0208] Step S34: Based on the time correspondence between each audio frame and each lyric in the lyrics sequence, align each audio frame in the human voice audio with each lyric in the lyrics sequence to obtain the aligned audio text.

[0209] In this way, on the one hand, unlike existing technologies, audio and text are aligned by using the transition probabilities of the phoneme states corresponding to the audio and text to be aligned, thus optimizing the audio-text alignment process, accelerating the audio-text alignment speed, and reducing the consumption of manpower and time costs; on the other hand, by using the inter-word transition points in the audio to adjust the predicted transition probabilities of the phoneme states corresponding to the text, and aligning the audio and text based on the transition probabilities, the naturalness and expressiveness of the aligned audio-text can be improved, resulting in better quality and display effect of the aligned audio-text.

[0210] It should be understood that, although Figures 2-9 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figures 2-9 At least some of the steps in the process may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the steps or stages in other steps.

[0211] It is understood that the same / similar parts between the various embodiments of the methods described above in this specification can be referred to each other. Each embodiment focuses on the differences from other embodiments, and relevant parts can be referred to the description of other method embodiments.

[0212] Figure 10This is a block diagram of an audio text alignment device provided in an embodiment of this application. (Refer to...) Figure 10 The audio text alignment device 10 includes: a data acquisition unit 11, a jump point extraction unit 12, a probability adjustment unit 13, and an alignment unit 14.

[0213] Among them, the data acquisition unit 11 is configured to acquire the audio and text to be aligned;

[0214] The jump point extraction unit 12 is configured to extract at least one inter-character jump point from the audio; the inter-character jump point is located between two adjacent characters in the text content expressed by the audio.

[0215] The probability adjustment unit 13 is configured to perform probability adjustment on the predicted transition probabilities of the text belonging to various phoneme states based on the inter-character transition points, so as to obtain new transition probabilities; the probability adjustment is used to adjust the predicted transition probability between two adjacent characters in the text to a preset transition probability, the preset transition probability representing the transition probability between two adjacent characters having different phoneme states.

[0216] The alignment unit 14 is configured to perform alignment of the audio and the text based on the new transition probability to obtain aligned audio and text.

[0217] In some embodiments, before adjusting the predicted transition probabilities of the text belonging to various phoneme states based on the inter-word transition points, the probability adjustment unit 13 is further configured to:

[0218] The audio is segmented into frames to obtain an audio frame sequence;

[0219] The text is processed by phoneme conversion based on the audio frame sequence to obtain a phoneme sequence; the number of phonemes in the phoneme sequence corresponds to the number of audio frames in the audio frame sequence.

[0220] Obtain the predicted transition probability of each phoneme in the phoneme sequence belonging to various phoneme states.

[0221] In some embodiments, in the aspect of obtaining the predicted transition probabilities of each phoneme in the phoneme sequence belonging to various phoneme states, the probability adjustment unit 13 is further configured to:

[0222] Obtain the predicted transition probabilities of each phoneme in the phoneme sequence belonging to various hidden Markov states; the predicted transition probabilities include inter-state transition probabilities, which represent the probability that in the phoneme sequence, the first hidden Markov state of the previous phoneme transitions to the second hidden Markov state of the next phoneme, and the first hidden Markov state and the second hidden Markov state are different.

[0223] The step of adjusting the predicted transition probabilities of the text belonging to various phoneme states based on the inter-word transition points includes:

[0224] Based on the inter-word transition points, the state transition probabilities of some phonemes in the phoneme sequence are adjusted.

[0225] In some embodiments, in the aspect of probabilistically adjusting the state transition probabilities of some phonemes in the phoneme sequence based on the inter-word transition points, the probability adjustment unit 13 is further configured to:

[0226] In the audio, two adjacent target audio frames corresponding to the positions of the word transition points are identified;

[0227] Two target phonemes corresponding to the two target audio frames are determined in the phoneme sequence respectively;

[0228] The predicted state transition probability of the latter of the two target phonemes is adjusted to a preset transition probability.

[0229] In some embodiments, in the aspect of extracting at least one inter-word transition point from the audio, the transition point extraction unit 12 is further configured to:

[0230] The audio is subjected to fundamental frequency detection to determine the first type of transition point in the audio; the first type of transition point represents the fundamental frequency value transition between two adjacent audio frames in the audio.

[0231] Energy detection is performed on the audio to determine the second type of transition point in the audio; the second type of transition point represents the energy value transition between two adjacent audio frames in the audio.

[0232] Extract at least one transition point from the first type of transition point and the second type of transition point as an inter-word transition point.

[0233] In some embodiments, in determining the first type of transition point in the audio, the transition point extraction unit 12 is further configured to:

[0234] Perform Fourier transform on the fundamental frequency value of each audio frame in the audio to determine the amplitude spectrum data for the audio.

[0235] When the amplitude change between two adjacent audio frames in the amplitude spectrum data exceeds a preset level, it is determined that there is a first type of jump point between the two adjacent audio frames;

[0236] The first type of transition points in the audio are collected to obtain a sequence of first type of transition points for the audio.

[0237] In some embodiments, in determining the second type of transition point in the audio, the transition point extraction unit 12 is further configured to:

[0238] In each audio frame of the audio, when the degree of change between the energy values ​​of two adjacent audio frames exceeds a preset level, it is determined that there is a second type of jump point between the two adjacent audio frames;

[0239] By combining the second type of transition points in the audio, a sequence of second type transition points for the audio is obtained.

[0240] In some embodiments, in the aspect of extracting at least one transition point as an inter-word transition point from the first type of transition points and the second type of transition points, the transition point extraction unit 12 is further configured to:

[0241] Intersecting jump points are determined from each of the first type of jump points in the first type of jump point sequence and each of the second type of jump points in the second type of jump point sequence, and the intersecting jump points are used as inter-word jump points in the audio; the number of intersecting jump points is at least one.

[0242] In some embodiments, in the aspect of aligning the audio and the text based on the new transition probability to obtain aligned audio and text, the alignment unit 14 is further configured to:

[0243] Obtain the emission probability of the audio belonging to various phoneme states;

[0244] The transmission probability and the new transition probability are decoded to determine the optimal phoneme state corresponding to each audio frame in the audio.

[0245] Based on the optimal phoneme state of each audio frame, the time correspondence between each audio frame and each phoneme in the text is determined;

[0246] Based on the time correspondence, the audio and the text are aligned to obtain the aligned audio and text.

[0247] Figure 11This is a block diagram of a computer device 20 provided in an embodiment of this application. For example, the computer device 20 can be an electronic device, an electronic component, or a server array, etc. (Refer to...) Figure 11 The computer device 20 includes a processor 21, which may be a collection of processors, including one or more processors. The computer device 20 also includes memory resources represented by memory 22, on which computer programs, such as application programs, are stored. The computer programs stored in memory 22 may include one or more modules, each corresponding to a set of executable instructions. Furthermore, the processor 21 is configured to implement the audio text alignment method described above when executing the executable instructions.

[0248] In some embodiments, the computer device 20 is an electronic device whose computing system can run one or more operating systems, including any operating systems discussed above and any commercially available server operating system. The computer device 20 can also run any of a variety of additional server applications and / or middleware applications, including HTTP (Hypertext Transfer Protocol) servers, FTP (File Transfer Protocol) servers, CGI (Common Gateway Interface) servers, super servers, database servers, etc. Exemplary database servers include, but are not limited to, commercially available database servers from companies such as IBM.

[0249] In some embodiments, processor 21 typically controls the overall operation of computer device 20, such as operations associated with display, data processing, data communication, and recording operations. Processor 21 may include one or more processor components to execute computer programs to perform all or part of the steps of the methods described above. Furthermore, processor components may include one or more modules to facilitate interaction between processor components and other components. For example, processor components may include a multimedia module to facilitate control of the interaction between user computer device 20 and processor 21 using multimedia components.

[0250] In some embodiments, the processor component in processor 21 may also be referred to as a CPU (Central Processing Unit). The processor component may be an electronic chip with signal processing capabilities. The processor may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor component. Furthermore, the processor component may be implemented using integrated circuit chips.

[0251] In some embodiments, memory 22 is configured to store various types of data to support operation on computer device 20. Examples of such data include instructions for any application or method operating on computer device 20, acquired data, messages, images, videos, etc. Memory 22 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, optical disk, or graphene storage.

[0252] In some embodiments, the memory 22 can be a memory module, TF card, etc., and can store all information in the computer device 20, including the input raw data, computer programs, intermediate running results, and final running results. In some embodiments, it stores and retrieves information according to the location specified by the processor. In some embodiments, the computer device 20 has a memory function and can ensure normal operation because of the memory 22. In some embodiments, the memory 22 of the computer device 20 can be classified into main memory (RAM) and auxiliary memory (external memory) according to its purpose, or it can be classified into external memory and internal memory. External memory is usually magnetic media or optical discs, which can store information for a long time. RAM refers to the storage component on the motherboard, which is used to store the currently executing data and programs, but it is only used to temporarily store programs and data. The data will be lost when the power is turned off or disconnected.

[0253] In some embodiments, the computer device 20 may further include: a power supply component 23 configured to perform power management of the computer device 20, a wired or wireless network interface 24 configured to connect the computer device 20 to a network, and an input / output (I / O) interface 25. The computer device 20 may operate on an operating system stored in memory 22, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, or similar.

[0254] In some embodiments, power supply component 23 provides power to various components of computer device 20. Power supply component 23 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to computer device 20.

[0255] In some embodiments, the wired or wireless network interface 24 is configured to facilitate wired or wireless communication between the computer device 20 and other devices. The computer device 20 may access wireless networks based on communication standards, such as WiFi, carrier networks (such as 2G, 3G, 4G, or 5G), or combinations thereof.

[0256] In some embodiments, the wired or wireless network interface 24 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the wired or wireless network interface 24 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0257] In some embodiments, the input / output (I / O) interface 25 provides an interface between the processor 21 and peripheral interface modules, such as a keyboard, click wheel, buttons, etc. These buttons may include, but are not limited to, a home button, volume buttons, a power button, and a lock button.

[0258] Figure 12 This is a block diagram of a computer-readable storage medium 30 provided in an embodiment of this application. The computer-readable storage medium 30 stores a computer program 31, which, when executed by a processor, implements the audio-text alignment method described above.

[0259] If the integrated units of the various functional units in the various embodiments of this application are implemented as software functional units and sold or used as independent products, they can be stored in the computer-readable storage medium 30. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer-readable storage medium 30 includes a computer program 31, which includes several instructions to cause a computer device (which may be a personal computer, system server, or network device, etc.), an electronic device (e.g., MP3, MP4, etc., or a mobile phone, tablet computer, wearable device, etc., or a desktop computer, etc.) or a processor to execute all or part of the steps of the methods of the various embodiments of this application.

[0260] Figure 13 This is a block diagram of a computer program product 40 provided in an embodiment of this application. The computer program product 40 includes program instructions 41, which can be executed by the processor of server 20 to implement the audio text alignment method as described above.

[0261] Those skilled in the art will understand that embodiments of this application may provide an audio text alignment method, an audio text alignment device 10, a computer device 20, a computer-readable storage medium 30, or a computer program product 40. Therefore, this application may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application may take the form of a computer program product 40 embodied on one or more computer program instructions 41 (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0262] This application is described with reference to flowchart illustrations and / or block diagrams of audio text alignment methods, audio text alignment devices 10, computer devices 20, computer-readable storage media 30, or computer program products 40 according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by the computer program product 40. These computer program products 40 can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that program instructions 41, executable by the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the alignment of audio text in the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0263] These computer program products 40 may also be stored in a computer-readable storage medium capable of directing a computer or other programmable data processing device to function in a particular manner, such that program instructions 41 stored in the computer program product 40 produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0264] These program instructions 41 may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing the program instructions 41 that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0265] It should be noted that the various methods, apparatuses, electronic devices, computer-readable storage media, computer program products, etc. described above may also include other implementation methods according to the description of the method embodiments. For specific implementation methods, please refer to the description of the relevant method embodiments, which will not be elaborated here.

[0266] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0267] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method of aligning an audio text, characterized by, The method includes: Get the audio and text to be aligned; Extract at least one word transition point from the audio; the word transition point is located between two adjacent characters in the text content expressed by the audio; The audio is segmented into frames to obtain an audio frame sequence; The text is processed by phoneme conversion based on the audio frame sequence to obtain a phoneme sequence; the number of phonemes in the phoneme sequence corresponds to the number of audio frames in the audio frame sequence. Obtain the predicted transition probabilities of each phoneme in the phoneme sequence belonging to various Hidden Markov States; the predicted transition probabilities include inter-state transition probabilities, which represent the probability that in the phoneme sequence, a phoneme transitions from a first Hidden Markov State of a preceding phoneme to a second Hidden Markov State of a following phoneme, and the first Hidden Markov State and the second Hidden Markov State are different; in the audio, determine two adjacent target audio frames corresponding to the position of the inter-word transition point; Two target phonemes corresponding to the two target audio frames are determined in the phoneme sequence respectively; The state transition probability of the latter of the two target phonemes is adjusted to a preset transition probability to obtain a new transition probability; the preset transition probability represents the transition probability between two adjacent characters with different phoneme states. Based on the new transition probability, the audio and the text are aligned to obtain aligned audio and text.

2. The method of claim 1, wherein, Extracting at least one word transition point from the audio includes: The audio is subjected to fundamental frequency detection to determine the first type of transition point in the audio; the first type of transition point represents the fundamental frequency value transition between two adjacent audio frames in the audio. Energy detection is performed on the audio to determine the second type of transition point in the audio; the second type of transition point represents the energy value transition between two adjacent audio frames in the audio. Extract at least one transition point from the first type of transition point and the second type of transition point as an inter-word transition point.

3. The method of claim 2, wherein, Determining the first type of transition point in the audio includes: Perform Fourier transform on the fundamental frequency value of each audio frame in the audio to determine the amplitude spectrum data for the audio. When the amplitude change between two adjacent audio frames in the amplitude spectrum data exceeds a preset level, it is determined that there is a first type of jump point between the two adjacent audio frames; The first type of transition points in the audio are collected to obtain a sequence of first type of transition points for the audio.

4. The method of claim 3, wherein, Determining the second type of transition point in the audio includes: In each audio frame of the audio, when the degree of change between the energy values ​​of two adjacent audio frames exceeds a preset level, it is determined that there is a second type of jump point between the two adjacent audio frames; By combining the second type of transition points in the audio, a sequence of second type transition points for the audio is obtained.

5. The method of claim 4, wherein, Extracting at least one transition point from the first type of transition points and the second type of transition points as an inter-word transition point includes: Intersecting jump points are determined from each of the first type of jump points in the first type of jump point sequence and each of the second type of jump points in the second type of jump point sequence, and the intersecting jump points are used as inter-word jump points in the audio; the number of intersecting jump points is at least one.

6. The method of claim 1, wherein, The step of aligning the audio and text based on the new transition probability to obtain aligned audio and text includes: Obtain the emission probability of the audio belonging to various phoneme states; The transmission probability and the new transition probability are decoded to determine the optimal phoneme state corresponding to each audio frame in the audio. Based on the optimal phoneme state of each audio frame, the time correspondence between each audio frame and each phoneme in the text is determined; Based on the time correspondence, the audio and the text are aligned to obtain the aligned audio and text.

7. A computer device, comprising: include: processor; Memory for storing the executable instructions of the processor; The processor is configured to execute the executable instructions to implement the audio text alignment method as described in any one of claims 1 to 6.

8. A computer-readable storage medium comprising program data, characterized in that, When the program data is executed by the processor of a computer device, the computer device is enabled to perform the audio text alignment method as described in any one of claims 1 to 6.

Images