Method and device for determining timestamp

Abstract

The invention provides a timestamp determination method and device, and the method comprises the steps: inputting an obtained first to-be-synthesized text into a pre-trained speech synthesis model, obtaining a plurality of attention probability distribution diagrams based on the speech synthesis model, and determining a plurality of prediction corresponding relationships; obtaining a preset reference corresponding relation, and determining a deviation value between each prediction corresponding relation and the reference corresponding relation; and determining a target prediction corresponding relationship from the plurality of prediction corresponding relationships based on a deviation value between each prediction corresponding relationship and the reference corresponding relationship, so as to determine timestamp information corresponding to each character in the first to-be-synthesized text based on the target prediction corresponding relationship. Through the method and the device, the complexity of timestamp marking is greatly simplified, the cost is reduced, the marking efficiency is improved, and meanwhile, the precision of timestamp marking can be improved to the level of each frame.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method and device for determining a timestamp. Background Art

[0002] With the rapid development of artificial intelligence (AI), digital human technology, as a key area of human-computer interaction, has been widely applied in fields such as virtual live broadcasting, intelligent customer service, and educational tutoring. In digital human technology, speech synthesis and lip synchronization are key components in achieving natural interaction. Traditional speech synthesis technology primarily focuses on the naturalness and fluency of speech. However, with the increasing application scenarios, higher requirements are being placed on the accuracy of speech and lip synchronization.

[0003] Currently, achieving speech and lip synchronization mainly relies on the accurate acquisition of timestamps. Timestamp information not only needs to determine the start time of pronunciation of each character or word, but also needs to accurately calculate its duration. In the existing technology, commonly used timestamp acquisition methods are mainly divided into two categories: the first category is the estimation method based on a preset speaking rate. This method assumes that the pronunciation duration of each character is fixed and obtains timestamp information through simple mathematical calculations. Although this method is simple to implement, it ignores the natural variation of speech, resulting in mechanical and unnatural lip movements. The second method is to use additional models for timestamp marking, such as using a speech recognition model or a phoneme timestamp prediction model to analyze the synthesized speech. Although this type of method improves the accuracy of timestamps to a certain extent, it requires additional model training and deployment, increasing system complexity and computational cost. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a timestamp determination method and device. This method simply obtains multiple sets of corresponding prediction relationships based on the attention probability distribution maps output by multiple levels in the speech synthesis model, determines the target prediction relationship for accurately predicting the timestamp based on manually labeled data, and then accurately labels the timestamp based on the target prediction relationship. This method greatly simplifies the complexity of timestamp labeling, reduces costs, improves labeling efficiency, and also improves the accuracy of timestamp labeling to the per-frame level.

[0005] In a first aspect, an embodiment of the present application provides a method for determining a character timestamp, the method comprising:

[0006] Inputting the obtained first text to be synthesized into a pre-trained speech synthesis model, obtaining multiple attention probability distribution maps based on the speech synthesis model, and determining multiple prediction correspondences based on the multiple attention probability distribution maps; wherein the prediction correspondences represent the predicted timestamps of each word in the first text to be synthesized;

[0007] Obtaining a preset reference corresponding relationship and determining a deviation value between each predicted corresponding relationship and the reference corresponding relationship;

[0008] Based on the deviation value between each predicted correspondence and the reference correspondence, a target predicted correspondence is determined from multiple predicted correspondences, so as to determine the timestamp information corresponding to each word in the first text to be synthesized based on the target predicted correspondence; wherein the timestamp information corresponding to the word represents the pronunciation time of the word in the speech when the first text to be synthesized generates speech.

[0009] Furthermore, the speech synthesis model includes multiple levels, each level generates an attention probability distribution map corresponding to the first text to be synthesized through an attention mechanism, and each level focuses on different dimensions and / or granularity.

[0010] Furthermore, the first text to be synthesized is split into word units to determine a plurality of word units carried in the first text to be synthesized; for each level, an attention probability distribution map corresponding to the level is generated through the following steps:

[0011] Inputting a plurality of the word units into the hierarchy, and determining a probability value of the input word unit corresponding to each output unit;

[0012] The probability values of each output unit are arranged in the order of generation to generate an attention probability distribution matrix on the time dimension to obtain the attention probability distribution graph; wherein the horizontal axis of the attention probability distribution graph represents each output unit, the vertical axis of the attention probability distribution graph represents each input word, and each intersection point of the horizontal and vertical axes in the attention probability distribution graph represents the probability value of the output unit corresponding to the input word.

[0013] Furthermore, the determining of multiple prediction correspondences based on multiple attention probability distribution maps includes:

[0014] For each output unit in the attention probability distribution map of each level, determine whether the probability value corresponding to each input word is greater than or equal to a threshold value. If so, use the input word as the target point in the predicted correspondence relationship; or

[0015] For each output unit in the attention probability distribution map of each level, the input word with the largest corresponding probability value in each input word is used as the target point in the predicted correspondence relationship;

[0016] By connecting multiple target points in the attention probability distribution map in sequence, a predicted correspondence corresponding to the attention probability distribution map is obtained.

[0017] Furthermore, the determining of multiple prediction correspondences based on multiple attention probability distribution maps includes:

[0018] For each input word in the attention probability distribution map of each level, determine whether the probability value corresponding to each output unit is greater than or equal to a threshold value. If so, use the output unit as the target point in the predicted correspondence relationship; or

[0019] For each input word in the attention probability distribution map of each level, the output unit with the largest corresponding probability value in each output unit is used as the target point in the predicted correspondence relationship;

[0020] By connecting multiple target points in the attention probability distribution map in sequence, a predicted correspondence corresponding to the attention probability distribution map is obtained.

[0021] Furthermore, determining the deviation value between each predicted correspondence and the reference correspondence includes:

[0022] For each predicted correspondence, the predicted timestamp of the output unit corresponding to each input word in the predicted correspondence is subtracted from the reference timestamp of the output unit corresponding to each input word in the reference correspondence to determine the deviation value of the output unit corresponding to each input word.

[0023] Furthermore, the determining of a target predicted corresponding relationship from a plurality of predicted corresponding relationships based on a deviation value between each predicted corresponding relationship and the reference corresponding relationship includes:

[0024] For each predicted correspondence, a target value is determined based on a deviation between the predicted correspondence and the reference correspondence; wherein the target value is at least one of a maximum deviation, a minimum deviation, an average deviation, a median deviation, a standard deviation, and a variance of the deviation;

[0025] Determining the prediction corresponding relationship corresponding to the extreme value of the target value among the multiple prediction corresponding relationships as the target prediction corresponding relationship;

[0026] or;

[0027] Add the deviation values corresponding to any number of prediction correspondences to determine the total number of deviations;

[0028] The multiple prediction correspondences corresponding to the smallest total deviation among the multiple total deviations are determined as the target prediction correspondences.

[0029] Furthermore, determining the timestamp information corresponding to each word in the first to-be-synthesized text based on the target prediction correspondence includes:

[0030] Inputting the acquired first text to be synthesized into a pre-trained speech synthesis model to generate a first synthesized speech;

[0031] The target prediction correspondence in the speech synthesis model is obtained, and based on the target prediction correspondence and the interval time of the output unit, a timestamp corresponding to each word of the first to-be-synthesized text in the first synthesized speech is obtained.

[0032] Furthermore, the determination method further includes:

[0033] Inputting the acquired second text to be synthesized into the speech synthesis model to generate a second synthesized speech;

[0034] Obtain an attention probability distribution map of the target level in the speech synthesis model, and obtain a timestamp corresponding to each word element of the second text to be synthesized in the second synthesized speech based on the attention probability distribution map of the target level and the output interval time; wherein, the target level is one or more levels of the target prediction correspondence output in the speech synthesis model.

[0035] In a second aspect, an embodiment of the present application further provides a device for determining a character timestamp, the device comprising:

[0036] a predicted correspondence determination module, configured to input the acquired first text to be synthesized into a pre-trained speech synthesis model, acquire a plurality of attention probability distribution maps based on the speech synthesis model, and determine a plurality of predicted correspondences based on the plurality of attention probability distribution maps; wherein the predicted correspondences represent predicted timestamps for each word in the first text to be synthesized;

[0037] a deviation value determination module, configured to obtain a preset reference corresponding relationship and determine a deviation value between each predicted corresponding relationship and the reference corresponding relationship;

[0038] A timestamp information determination module is used to determine a target prediction correspondence from multiple prediction correspondences based on the deviation value between each prediction correspondence and the reference correspondence, so as to determine the timestamp information corresponding to each word in the first text to be synthesized based on the target prediction correspondence; wherein the timestamp information corresponding to the word represents the pronunciation time of the word in the speech when the first text to be synthesized is generated.

[0039] An embodiment of the present application provides a timestamp determination method and device. First, the acquired first text to be synthesized is input into a pre-trained speech synthesis model, multiple attention probability distribution maps are acquired based on the speech synthesis model, and multiple prediction correspondences are determined based on the multiple attention probability distribution maps; wherein the prediction correspondence represents the predicted timestamp of each word in the first text to be synthesized; then, a pre-set reference correspondence is acquired, and a deviation value between each prediction correspondence and the reference correspondence is determined; finally, based on the deviation value between each prediction correspondence and the reference correspondence, a target prediction correspondence is determined from the multiple prediction correspondences, so as to determine the timestamp information corresponding to each word in the first text to be synthesized based on the target prediction correspondence; wherein the timestamp information corresponding to the word represents the pronunciation time of the word in the speech when the first text to be synthesized generates speech.

[0040] This application only needs to obtain multiple groups of corresponding prediction correspondences based on the attention probability distribution maps output by multiple levels in the speech synthesis model, determine the target prediction correspondences for accurately predicting timestamps based on the manually labeled data, and accurately mark the timestamps based on the target prediction correspondences. In this way, in the subsequent speech synthesis process, predictions can be made directly based on the target level to which the target prediction correspondences belong. The complexity of timestamp marking is greatly simplified, costs are reduced, and marking efficiency is improved. At the same time, the accuracy of timestamp marking can also be improved to the level of each frame. In this way, through the method for determining character timestamps provided in the embodiment of the present application, when performing subsequent speech synthesis, there is no need to deploy additional speech recognition models or phoneme timestamp prediction models to recognize the synthesized speech and mark the timestamps one by one, which reduces the system complexity and computational costs.

[0041] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 A flowchart of a method for determining a character timestamp provided in an embodiment of the present application;

[0044] Figure 2This is an example diagram of an attention probability distribution diagram provided in an embodiment of the present application;

[0045] Figure 3 An example diagram comparing a reference correspondence relationship and a predicted correspondence relationship provided in an embodiment of the present application;

[0046] Figure 4 A schematic diagram of the structure of a device for determining a character timestamp provided in an embodiment of the present application;

[0047] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for which protection is claimed, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, each other embodiment obtained by those skilled in the art without making creative work falls within the scope of protection of the present application.

[0049] First, the application scenarios to which this application is applicable are introduced. This application can be applied in the field of computer technology.

[0050] With the rapid development of artificial intelligence (AI), digital human technology, as a key area of human-computer interaction, has been widely applied in fields such as virtual live broadcasting, intelligent customer service, and educational tutoring. In digital human technology, speech synthesis and lip synchronization are key components in achieving natural interaction. Traditional speech synthesis technology primarily focuses on the naturalness and fluency of speech. However, with the increasing application scenarios, higher requirements are being placed on the accuracy of speech and lip synchronization.

[0051] Research has found that currently, achieving speech and lip synchronization mainly relies on the accurate acquisition of timestamps. Timestamp information not only needs to determine the start time of pronunciation of each character or word, but also needs to accurately calculate its duration. In the existing technology, commonly used timestamp acquisition methods are mainly divided into two categories: the first category is an estimation method based on a preset speaking rate. This method assumes that the pronunciation duration of each character is fixed and obtains timestamp information through simple mathematical calculations. Although this method is simple to implement, it ignores the natural variation of speech, resulting in mechanical and unnatural lip movements. The second method is to use additional models for timestamp marking, such as using a speech recognition model or a phoneme timestamp prediction model to analyze the synthesized speech. Although this type of method improves the accuracy of timestamps to a certain extent, it requires additional model training and deployment, increasing system complexity and computational cost.

[0052] Based on this, an embodiment of the present application provides a method for determining a character timestamp, which greatly simplifies the complexity of timestamp marking, reduces costs, improves marking efficiency, and can also increase the accuracy of timestamp marking to the level of each frame.

[0053] See also Figure 1 , Figure 1 This is a flow chart of a method for determining a character timestamp provided in an embodiment of the present application. Figure 1 As shown in , the determination method provided in the embodiment of the present application includes:

[0054] S101, inputting the obtained first text to be synthesized into a pre-trained speech synthesis model, obtaining multiple attention probability distribution maps based on the speech synthesis model, and determining multiple prediction correspondences based on the multiple attention probability distribution maps.

[0055] Here, the text to be synthesized can be input into the speech synthesis model in the form of text, or in the form of speech, and this application does not make specific limitations on this. The speech synthesis model can generate corresponding speech based on the received text to be synthesized. The attention probability distribution map (Attention Map) is a visualization tool for the attention mechanism (Attention Mechanism) in the deep learning model, which is used to show the degree of attention of the model to different positions when processing input data. The predicted correspondence relationship represents the predicted timestamp of the input character corresponding to each word element in the first text to be synthesized. Here, in the field of artificial intelligence, especially in natural language processing (NLP), "word element" refers to the smallest unit or basic element for processing text. It can be a word, a phrase, a punctuation mark, a subword or a character.

[0056] Regarding the above step S101, in the specific implementation, the obtained first text to be synthesized is input into a pre-trained speech synthesis model, and based on multiple attention probability distribution maps of the speech synthesis model, multiple prediction correspondences are determined based on the multiple attention probability distribution maps.

[0057] Here, according to the embodiment provided by the present application, the speech synthesis model includes multiple levels, and each level generates an attention probability distribution map corresponding to the first text to be synthesized through the attention mechanism, and the dimensions and / or granularity paid attention to by each level are different. Specifically, each level in the speech synthesis model generates the probability that each output character pays attention to the input character through the attention mechanism. The different levels in the speech synthesis model are essentially different in the dimensions and / or granularity of attention. For example, some layers may pay attention to whether the character is a vowel or a consonant, whether the character is a single tone or a polyphonic tone, or whether the character is an open-mouthed sound or a closed-mouthed sound, etc. Since the dimensions and / or granularity paid attention to by different levels are different, the attention probability distribution maps output by each level are different, and the predicted correspondences determined are definitely different.

[0058] Here, according to the embodiment provided by the present application, after obtaining the first text to be synthesized, the first text to be synthesized is split into word units to determine multiple word units carried in the first text to be synthesized.

[0059] According to the determination method provided in this application, for each level, the attention probability distribution map corresponding to the level is generated through the following steps:

[0060] A: Input a plurality of the word units into the layer, and determine the probability value of the input word unit corresponding to each output unit.

[0061] Regarding step A above, during the specific implementation, the first text to be synthesized is split into characters, and after determining the multiple word units carried in the first text to be synthesized, each word unit is encoded, and the encoded multiple word units are simultaneously input into a certain level in the speech synthesis model to determine the probability value of the input word unit corresponding to each output unit. Here, the probability value of the input word unit corresponding to the output unit is generated, which is an inherent feature of the large language model. In this way, the probability of each output unit is generated sequentially, and the probability distribution of the next unit depends on the probability distribution of the previous unit as input, that is, autoregressive generation. The sum of the probabilities of all input word units corresponding to the same output unit is equal to 1.

[0062] B: Arrange the probability values of each output unit in the order of generation to generate an attention probability distribution matrix in the time dimension to obtain the attention probability distribution graph.

[0063] Here, the horizontal axis of the attention probability distribution graph represents each output unit, the vertical axis of the attention probability distribution graph represents each input word, and each intersection point of the horizontal and vertical axes in the attention probability distribution graph represents the probability value of the output unit corresponding to the input word.

[0064] Regarding step B above, in the specific implementation, the probability values of each output unit are arranged in the generation order to generate the attention probability distribution matrix in the time dimension to obtain the attention probability distribution map of the output of this level. Here, as an optional embodiment, the attention probability distribution matrix can be converted into a heat map, and the heat map can be used as the attention probability distribution map. Figure 2 , Figure 2 This is an example diagram of an attention probability distribution diagram provided by an embodiment of the present application. Figure 2 As shown in , the horizontal axis in the attention probability distribution diagram is the output unit. Since the time interval of the output unit is a fixed value, the horizontal axis can also represent the time value, and the vertical axis is the input word unit, as shown in Figure 2 As shown in the figure, the vertical axis represents input characters numbered 0 to 10 from top to bottom. Each intersection of the horizontal and vertical axes represents the probability value of the input word corresponding to each output unit. The brighter the color, the higher the probability that the output unit has noticed the input word, and the darker the color, the lower the probability. Different layers in the speech synthesis model generate different attention probability distribution maps, that is, the probability distribution of the input word corresponding to each output unit is different.

[0065] As an optional embodiment, with respect to the above step S103, determining multiple predicted correspondences based on multiple attention probability distribution maps includes:

[0066] Step 1031: for each output unit in the attention probability distribution map of each level, determine whether the probability value corresponding to each input word is greater than or equal to a threshold value. If so, use the input word as the target point in the predicted correspondence relationship; or, for each output unit in the attention probability distribution map of each level, use the input word with the largest corresponding probability value among each input word as the target point in the predicted correspondence relationship.

[0067] For the above step 1031, in the specific implementation, there are two ways to determine the target point using the horizontal coordinate in the attention probability distribution map. Method 1: For each output unit in the attention probability distribution map of each level, determine whether the probability value corresponding to each input word is greater than or equal to the threshold. If so, the input word is used as the target point in the predicted correspondence. If not, the input word with the largest probability value among the multiple input words can be used as the target point. Method 2: For each output unit in the attention probability distribution map of each level, the input word with the largest probability value among each input word is used as the target point in the predicted correspondence. In this way, the target point of the input word corresponding to each output unit is determined one by one along the horizontal coordinate direction in the attention probability distribution map to ensure that the predicted correspondence is continuous and complete.

[0068] Step 1032: connect multiple target points in the attention probability distribution map in sequence to obtain the predicted correspondence corresponding to the attention probability distribution map.

[0069] Regarding the above step 1032, in the specific implementation, the target points determined in the above step 1031 are sequentially connected to form the predicted corresponding relationship. Figure 2 , since a piece of text to be synthesized corresponds to a pronunciation length when generating speech, for example, 1 word 1 second, and one second of generated speech corresponds to multiple frames, for example 10 frames, that is, multiple speech output units, for example 10 speech output units. Of course, when generating speech, the pronunciation length of each word is long or short, so the predicted correspondence is presented as follows Figure 2 The irregular curve shown in .

[0070] As another optional embodiment, with respect to the above step S103, determining multiple predicted correspondences based on multiple attention probability distribution maps includes:

[0071] a: For each input word in the attention probability distribution map of each level, determine whether the probability value corresponding to each output unit is greater than or equal to the threshold. If so, use the output unit as the target point in the predicted correspondence; or, for each input word in the attention probability distribution map of each level, use the output unit with the largest corresponding probability value among each output unit as the target point in the predicted correspondence.

[0072] For step a above, in the specific implementation, there are two ways to determine the target point using the ordinate in the attention probability distribution map. Method 1: For each input word in the attention probability distribution map output at each level, determine whether the probability value corresponding to each output unit is greater than or equal to the threshold. If so, use the output unit as the target point in the predicted correspondence. If not, the output unit with the largest corresponding probability value among the multiple output units can be used as the target point. Method 2: For each input word in the attention probability distribution map at each level, use the output unit with the largest corresponding probability value among each output unit as the target point in the predicted correspondence. In this way, the target point of the output unit corresponding to each input word is determined one by one along the ordinate direction in the attention probability distribution map. The characteristic is that the predicted correspondence determined subsequently can be discontinuous to reflect the timestamps such as pauses and intervals in the process of synthesizing speech.

[0073] b: Connect multiple target points in the attention probability distribution map in sequence to obtain the predicted correspondence corresponding to the attention probability distribution map.

[0074] Here, the description of the above step b can refer to the description of the above step 1032, and can achieve the same technical effect, so it will not be repeated.

[0075] S102: Acquire a preset reference corresponding relationship, and determine a deviation value between each predicted corresponding relationship and the reference corresponding relationship.

[0076] Here, the reference correspondence is the correct correspondence between output characters and input characters, obtained manually or through other automated means. Other automated means, such as using a pre-trained speech recognition model to analyze the correspondence between speech and input word metacharacters frame by frame, can also generate the reference correspondence after manual correction.

[0077] Regarding the above step S102 , during specific implementation, a preset reference correspondence is obtained, and at least two sets of deviation values are obtained based on the multiple predicted correspondences and the reference correspondence obtained in step S101 .

[0078] As an optional embodiment, with respect to the above step S102, determining the deviation value between each predicted correspondence and the reference correspondence includes:

[0079] For each predicted correspondence, the predicted timestamp of the output unit corresponding to each input word in the predicted correspondence is subtracted from the reference timestamp of the output unit corresponding to each input word in the reference correspondence to determine the deviation value of the output unit corresponding to each input word.

[0080] In the specific implementation of the above steps, for each predicted correspondence, the predicted timestamp of the output unit corresponding to each input word in the predicted correspondence is subtracted from the reference timestamp of the output unit corresponding to each input word in the reference correspondence, and the deviation value of the output unit corresponding to each input word can be determined. Figure 3 , Figure 3 This is an example diagram comparing a reference correspondence relationship and a predicted correspondence relationship provided in an embodiment of the present application. Figure 3 As shown in , the black curve is the obtained reference correspondence, and the light curve is the predicted correspondence generated by the speech synthesis model. Then a set of deviation values may include: the deviation values before and after the output character corresponding to each input character, such as Figure 3 In the example provided, [(0,3),(3,2),...,(-2,-3)] represents the number of time units or output characters that the predicted value is ahead or behind the true value. Deviation values may also include total deviations, mean, sum of squares, variance, standard deviation, median, maximum, minimum, etc., which are not specifically limited in this application.

[0081] S103, based on the deviation value between each predicted correspondence and the reference correspondence, determining a target predicted correspondence from multiple predicted correspondences, so as to determine the timestamp information corresponding to each word in the first to-be-synthesized text based on the target predicted correspondence.

[0082] The timestamp information corresponding to each word element represents the pronunciation time of the word element in the speech when the first to-be-synthesized text is generated into speech. The pronunciation time here includes the pronunciation moment and pronunciation duration in the speech.

[0083] Regarding step S103, in specific implementation, a target predicted correspondence is determined from the multiple predicted correspondences based on the deviation between each predicted correspondence and the reference correspondence. The timestamp information corresponding to each word in the first text to be synthesized is then determined based on the target predicted correspondence. In this way, the pronunciation moment and duration of each word in the speech can be determined based on the target predicted correspondence. Subsequent digital human image synthesis is then performed based on the pronunciation time of each word.

[0084] As an optional embodiment, with respect to the above step S103, determining a target predicted correspondence relationship from a plurality of predicted correspondence relationships based on a deviation value between each predicted correspondence relationship and the reference correspondence relationship includes:

[0085] I: For each predicted correspondence, a target value is determined based on a deviation between the predicted correspondence and the reference correspondence.

[0086] II: Determine the prediction correspondence corresponding to the extreme value of the target value in the multiple prediction correspondences as the target prediction correspondence.

[0087] Regarding the above steps I to II, during the specific implementation, for each predicted correspondence, a target value is determined based on the deviation value between the predicted correspondence and the reference correspondence. Here, the target value is at least one of the maximum deviation value, the minimum deviation value, the average deviation value, the median deviation value, the standard deviation, and the variance of the deviation value of the multiple deviation values. Then, the predicted correspondence corresponding to the extreme value of the target value in the multiple predicted correspondences is determined as the target predicted correspondence. In this way, based on the target value in each group of deviation values, a group of predicted correspondences corresponding to the extreme value of the target value between the predicted correspondence and the reference correspondence is determined as the target predicted correspondence.

[0088] Alternatively, the target prediction correspondence may also be determined according to the following steps:

[0089] i: Add the deviation values corresponding to any number of prediction correspondences to determine the total number of deviations.

[0090] ii: Determine the multiple prediction correspondences corresponding to the smallest total deviation among the multiple total deviations as the target prediction correspondence.

[0091] In the specific implementation of steps i-ii above, the deviation values corresponding to any multiple prediction correspondences are summed to determine a total deviation. The multiple prediction correspondences corresponding to the minimum total deviation among the multiple total deviations are determined as target prediction correspondences. Alternatively, the target prediction correspondences may be determined as the multiple prediction correspondences whose total deviation is minimized by summing up the multiple sets of deviation values.

[0092] According to the determination method provided in the present application, as an optional embodiment, determining the timestamp information corresponding to each word in the first to-be-synthesized text based on the target prediction correspondence relationship includes:

[0093] The obtained first text to be synthesized is input into a pre-trained speech synthesis model to generate a first synthesized speech; the target prediction correspondence in the speech synthesis model is obtained, and based on the target prediction correspondence and the interval time of the output unit, the timestamp corresponding to each word of the first text to be synthesized in the first synthesized speech is obtained.

[0094] In the specific implementation of the above two steps, the first text to be synthesized is input into a pre-trained speech synthesis model to generate a first synthesized speech. Then, a target prediction correspondence is obtained from the speech synthesis model, and based on the target prediction correspondence and the interval time of the output unit, the timestamp corresponding to each word of the first text to be synthesized in the first synthesized speech is obtained.

[0095] According to the determination method provided in this application, as an optional embodiment, the determination method further includes:

[0096] The obtained second text to be synthesized is input into the speech synthesis model to generate a second synthesized speech; the attention probability distribution map of the target level in the speech synthesis model is obtained, and based on the attention probability distribution map of the target level and the output interval time, the timestamp corresponding to each word of the second text to be synthesized in the first synthesized speech is obtained.

[0097] Here, the target level is one or more levels of the target prediction correspondence output in the speech synthesis model.

[0098] For the above two steps, in the specific implementation, the obtained second text to be synthesized is input into the speech synthesis model to generate a second synthesized speech. Then, the attention probability distribution map of the target level in the speech synthesis model is obtained, where the target level is one or more levels of the output target prediction correspondence in the speech synthesis model. Based on the attention probability distribution map of the target level and the output interval time, the timestamp corresponding to each word element of the second text to be synthesized in the second synthesized speech is obtained. In this way, only the attention probability distribution map of the level with the best prediction effect in the speech synthesis model is obtained, and the word element timestamp is determined using the level with the best prediction effect. In the subsequent speech synthesis process, there is no need to calculate the deviation. It is only necessary to mark the timestamp directly in the synthesized speech according to the estimated target prediction correspondence based on the attention probability distribution map of the target level.

[0099] The timestamp determination method provided in the embodiment of the present application is as follows: first, the acquired first text to be synthesized is input into a pre-trained speech synthesis model, multiple attention probability distribution maps are acquired based on the speech synthesis model, and multiple prediction correspondences are determined based on the multiple attention probability distribution maps; wherein the prediction correspondence represents the predicted timestamp of each word in the first text to be synthesized; then, a pre-set reference correspondence is acquired, and the deviation value between each prediction correspondence and the reference correspondence is determined; finally, based on the deviation value between each prediction correspondence and the reference correspondence, a target prediction correspondence is determined from the multiple prediction correspondences, so as to determine the timestamp information corresponding to each word in the first text to be synthesized based on the target prediction correspondence; wherein the timestamp information corresponding to the word represents the pronunciation time of the word in the speech when the first text to be synthesized generates speech.

[0100] This application only needs to obtain multiple groups of corresponding prediction correspondences based on the attention probability distribution maps output by multiple levels in the speech synthesis model, determine the target prediction correspondences for accurately predicting timestamps based on the manually labeled data, and accurately mark the timestamps based on the target prediction correspondences. In this way, in the subsequent speech synthesis process, predictions can be made directly based on the target level to which the target prediction correspondences belong. The complexity of timestamp marking is greatly simplified, costs are reduced, and marking efficiency is improved. At the same time, the accuracy of timestamp marking can also be improved to the level of each frame. In this way, through the method for determining character timestamps provided in the embodiment of the present application, when performing subsequent speech synthesis, there is no need to deploy additional speech recognition models or phoneme timestamp prediction models to recognize the synthesized speech and mark the timestamps one by one, which reduces the system complexity and computational costs.

[0101] See also Figure 4 , Figure 4 This is a structural diagram of a device for determining a character timestamp provided in an embodiment of the present application. Figure 4 As shown in , the determining device 400 includes:

[0102] Prediction correspondence determination module 401 is configured to input the acquired first text to be synthesized into a pre-trained speech synthesis model, acquire multiple attention probability distribution maps based on the speech synthesis model, and determine multiple prediction correspondences based on the multiple attention probability distribution maps; wherein the prediction correspondences represent the predicted timestamps of each word in the first text to be synthesized;

[0103] The deviation value determination module 402 is used to obtain a preset reference correspondence relationship and determine a deviation value between each predicted correspondence relationship and the reference correspondence relationship;

[0104] The timestamp information determination module 403 is used to determine a target prediction correspondence from multiple prediction correspondences based on the deviation value between each prediction correspondence and the reference correspondence, so as to determine the timestamp information corresponding to each word in the first text to be synthesized based on the target prediction correspondence; wherein the timestamp information corresponding to the word represents the pronunciation time of the word in the speech when the first text to be synthesized generates speech.

[0105] Furthermore, the speech synthesis model includes multiple levels, each level generates an attention probability distribution map corresponding to the first text to be synthesized through an attention mechanism, and each level focuses on different dimensions and / or granularity.

[0106] Furthermore, the first text to be synthesized is split into word units to determine a plurality of word units carried in the first text to be synthesized; for each level, the predicted correspondence determination module 401 is further configured to generate the corresponding attention probability distribution map through the following steps:

[0107] Inputting a plurality of the word units into the hierarchy, and determining a probability value of the input word unit corresponding to each output unit;

[0108] The probability values of each output unit are arranged in the order of generation to generate an attention probability distribution matrix on the time dimension to obtain the attention probability distribution graph; wherein the horizontal axis of the attention probability distribution graph represents each output unit, the vertical axis of the attention probability distribution graph represents each input word, and each intersection point of the horizontal and vertical axes in the attention probability distribution graph represents the probability value of the output unit corresponding to the input word.

[0109] Furthermore, when the predicted correspondence relationship determination module 401 is used to determine multiple predicted correspondence relationships based on multiple attention probability distribution maps, the predicted correspondence relationship determination module 401 is further used to:

[0110] For each output unit in the attention probability distribution map of each level, determine whether the probability value corresponding to each input word is greater than or equal to a threshold value. If so, use the input word as the target point in the predicted correspondence relationship; or

[0111] For each output unit in the attention probability distribution map of each level, the input word with the largest corresponding probability value in each input word is used as the target point in the predicted correspondence relationship;

[0112] By connecting multiple target points in the attention probability distribution map in sequence, a predicted correspondence corresponding to the attention probability distribution map is obtained.

[0113] Furthermore, when the predicted correspondence relationship determination module 401 is used to determine multiple predicted correspondence relationships based on multiple attention probability distribution maps, the predicted correspondence relationship determination module 401 is further used to:

[0114] For each input word in the attention probability distribution map of each level, determine whether the probability value corresponding to each output unit is greater than or equal to a threshold value. If so, use the output unit as the target point in the predicted correspondence relationship; or

[0115] For each input word in the attention probability distribution map of each level, the output unit with the largest corresponding probability value in each output unit is used as the target point in the predicted correspondence relationship;

[0116] By connecting multiple target points in the attention probability distribution map in sequence, a predicted correspondence corresponding to the attention probability distribution map is obtained.

[0117] Furthermore, when the deviation value determination module 402 is used to determine the deviation value between each predicted correspondence relationship and the reference correspondence relationship, the deviation value determination module 402 is further used to:

[0118] For each predicted correspondence, the predicted timestamp of the output unit corresponding to each input word in the predicted correspondence is subtracted from the reference timestamp of the output unit corresponding to each input word in the reference correspondence to determine the deviation value of the output unit corresponding to each input word.

[0119] Furthermore, when the timestamp information determining module 403 is used to determine the target predicted corresponding relationship from the multiple predicted corresponding relationships based on the deviation value between each predicted corresponding relationship and the reference corresponding relationship, the timestamp information determining module 403 is further used to:

[0120] For each predicted correspondence, a target value is determined based on a deviation between the predicted correspondence and the reference correspondence; wherein the target value is at least one of a maximum deviation, a minimum deviation, an average deviation, a median deviation, a standard deviation, and a variance of the deviation;

[0121] Determining the prediction corresponding relationship corresponding to the extreme value of the target value among the multiple prediction corresponding relationships as the target prediction corresponding relationship;

[0122] or;

[0123] Add the deviation values corresponding to any number of prediction correspondences to determine the total number of deviations;

[0124] The multiple prediction correspondences corresponding to the smallest total deviation among the multiple total deviations are determined as the target prediction correspondences.

[0125] Furthermore, when the timestamp information determining module 403 is used to determine the timestamp information corresponding to each word in the first to-be-synthesized text based on the target prediction correspondence, the timestamp information determining module 403 is further used to:

[0126] Inputting the acquired first text to be synthesized into a pre-trained speech synthesis model to generate a first synthesized speech;

[0127] The target prediction correspondence in the speech synthesis model is obtained, and based on the target prediction correspondence and the interval time of the output unit, a timestamp corresponding to each word of the first to-be-synthesized text in the first synthesized speech is obtained.

[0128] Furthermore, the determining device 400 further includes a speech generating unit, and the speech generating unit is configured to:

[0129] Inputting the acquired second text to be synthesized into the speech synthesis model to generate a second synthesized speech;

[0130] Obtain an attention probability distribution map of the target level in the speech synthesis model, and obtain a timestamp corresponding to each word element of the second text to be synthesized in the second synthesized speech based on the attention probability distribution map of the target level and the output interval time; wherein, the target level is one or more levels of the target prediction correspondence output in the speech synthesis model.

[0131] See also Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 5 As shown in FIG, the electronic device 500 includes a processor 510, a memory 520 and a bus 530.

[0132] The memory 520 stores machine-readable instructions executable by the processor 510. When the electronic device 500 is running, the processor 510 communicates with the memory 520 via the bus 530. When the machine-readable instructions are executed by the processor 510, the above-mentioned Figure 1 The specific implementation of the steps of the method for determining the character timestamp in the illustrated method embodiment can be found in the method embodiment and will not be described in detail here.

[0133] The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the computer program can execute the above-mentioned Figure 1 The specific implementation of the steps of the method for determining the character timestamp in the illustrated method embodiment can be found in the method embodiment and will not be described in detail here.

[0134] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0135] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. There may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed may be through some communication interface, indirect coupling or communication connection of devices or units, which may be electrical, mechanical or other forms.

[0136] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0137] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0138] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0139] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The scope of protection of the present application is not limited thereto. Although the present application has been described in detail with reference to the above-mentioned embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-mentioned embodiments within the technical scope disclosed in the present application, or perform equivalent replacements for some of the technical features thereof. These modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A method for determining a timestamp, characterized in that: The determination method includes: Inputting the obtained first text to be synthesized into a pre-trained speech synthesis model, obtaining multiple attention probability distribution maps based on the speech synthesis model, and determining multiple prediction correspondences based on the multiple attention probability distribution maps; wherein the prediction correspondences represent the predicted timestamps of each word in the first text to be synthesized; Obtaining a preset reference corresponding relationship and determining a deviation value between each predicted corresponding relationship and the reference corresponding relationship; Based on the deviation value between each predicted correspondence and the reference correspondence, a target predicted correspondence is determined from multiple predicted correspondences, so as to determine the timestamp information corresponding to each word in the first text to be synthesized based on the target predicted correspondence; wherein the timestamp information corresponding to the word represents the pronunciation time of the word in the speech when the first text to be synthesized generates speech.

2. The determination method according to claim 1, characterized in that The speech synthesis model includes multiple levels, each level generates an attention probability distribution map corresponding to the first text to be synthesized through an attention mechanism, and each level focuses on different dimensions and / or granularity.

3. The determination method according to claim 2, characterized in that: The first text to be synthesized is split into word units to determine a plurality of word units carried in the first text to be synthesized; for each level, an attention probability distribution map corresponding to the level is generated through the following steps: Inputting a plurality of the word units into the hierarchy, and determining a probability value of the input word unit corresponding to each output unit; The probability values of each output unit are arranged in the order of generation to generate an attention probability distribution matrix on the time dimension to obtain the attention probability distribution graph; wherein the horizontal axis of the attention probability distribution graph represents each output unit, the vertical axis of the attention probability distribution graph represents each input word, and each intersection point of the horizontal and vertical axes in the attention probability distribution graph represents the probability value of the output unit corresponding to the input word.

4. The determination method according to claim 3, characterized in that: The determining of multiple prediction correspondences based on multiple attention probability distribution maps includes: For each output unit in the attention probability distribution map of each level, determine whether the probability value corresponding to each input word is greater than or equal to a threshold value. If so, use the input word as the target point in the predicted correspondence relationship; or For each output unit in the attention probability distribution map of each level, the input word with the largest corresponding probability value in each input word is used as the target point in the predicted correspondence relationship; By connecting multiple target points in the attention probability distribution map in sequence, a predicted correspondence corresponding to the attention probability distribution map is obtained.

5. The determination method according to claim 3, characterized in that: The determining of multiple prediction correspondences based on multiple attention probability distribution maps includes: For each input word in the attention probability distribution map of each level, determine whether the probability value corresponding to each output unit is greater than or equal to a threshold value. If so, use the output unit as the target point in the predicted correspondence relationship; or For each input word in the attention probability distribution map of each level, the output unit with the largest corresponding probability value in each output unit is used as the target point in the predicted correspondence relationship; By connecting multiple target points in the attention probability distribution map in sequence, a predicted correspondence corresponding to the attention probability distribution map is obtained.

6. The determination method according to claim 1, characterized in that: Determining the deviation value between each predicted corresponding relationship and the reference corresponding relationship includes: For each predicted correspondence, the predicted timestamp of the output unit corresponding to each input word in the predicted correspondence is subtracted from the reference timestamp of the output unit corresponding to each input word in the reference correspondence to determine the deviation value of the output unit corresponding to each input word.

7. The determination method according to claim 1, characterized in that: The determining a target predicted corresponding relationship from a plurality of predicted corresponding relationships based on a deviation value between each predicted corresponding relationship and the reference corresponding relationship includes: For each predicted correspondence, a target value is determined based on a deviation between the predicted correspondence and the reference correspondence; wherein the target value is at least one of a maximum deviation, a minimum deviation, an average deviation, a median deviation, a standard deviation, and a variance of the deviation; Determining the prediction corresponding relationship corresponding to the extreme value of the target value among the multiple prediction corresponding relationships as the target prediction corresponding relationship; or; Add the deviation values corresponding to any number of prediction correspondences to determine the total number of deviations; The multiple prediction correspondences corresponding to the smallest total deviation among the multiple total deviations are determined as the target prediction correspondences.

8. The determination method according to claim 1, characterized in that: The determining, based on the target prediction correspondence, the timestamp information corresponding to each word in the first to-be-synthesized text includes: Inputting the acquired first text to be synthesized into a pre-trained speech synthesis model to generate a first synthesized speech; The target prediction correspondence in the speech synthesis model is obtained, and based on the target prediction correspondence and the interval time of the output unit, a timestamp corresponding to each word of the first to-be-synthesized text in the first synthesized speech is obtained.

9. The determination method according to claim 2, characterized in that: The determination method further includes: Inputting the acquired second text to be synthesized into the speech synthesis model to generate a second synthesized speech; Obtain an attention probability distribution map of the target level in the speech synthesis model, and obtain a timestamp corresponding to each word element of the second text to be synthesized in the second synthesized speech based on the attention probability distribution map of the target level and the output interval time; wherein, the target level is one or more levels of the target prediction correspondence output in the speech synthesis model.

10. A device for determining a timestamp, characterized in that: The determining device comprises: a predicted correspondence determination module, configured to input the acquired first text to be synthesized into a pre-trained speech synthesis model, acquire a plurality of attention probability distribution maps based on the speech synthesis model, and determine a plurality of predicted correspondences based on the plurality of attention probability distribution maps; wherein the predicted correspondences represent predicted timestamps for each word in the first text to be synthesized; a deviation value determination module, configured to obtain a preset reference corresponding relationship and determine a deviation value between each predicted corresponding relationship and the reference corresponding relationship; A timestamp information determination module is used to determine a target prediction correspondence from multiple prediction correspondences based on the deviation value between each prediction correspondence and the reference correspondence, so as to determine the timestamp information corresponding to each word in the first text to be synthesized based on the target prediction correspondence; wherein the timestamp information corresponding to the word represents the pronunciation time of the word in the speech when the first text to be synthesized is generated.

Images