A word granularity timestamp determination method, electronic device, and storage medium

Abstract

This invention discloses a method, electronic device, and storage medium for determining character-level timestamps. The method determines the starting point of the current character by identifying the time point corresponding to the probability peak of the current character, and determines the ending point of the current character based on the difference between the time points corresponding to the next adjacent probability peak. The timestamp of the current character is then determined based on the starting point and the ending point. Therefore, this embodiment can solve the technical problems of time offset in character-level timestamps and the duration of ineffective speech inclusion characters, thereby improving the accuracy of character-level timestamps.

Description

Technical Field

[0001] This invention relates to the field of speech recognition technology, specifically to a method for determining word-level timestamps, an electronic device, and a storage medium. Background Technology

[0002] With the development of speech recognition technology, people have increasingly higher requirements for the accuracy of word-level timestamps in speech recognition. Existing technology uses the CTC (Connectionist Temporal Classification) algorithm to take the start time of a word from the time of the previous probability peak and the end time of a word from the time of the current word's probability peak, thus using the time of the probability peak as the word's timestamp.

[0003] However, since the probability peak predicted by CTC is slightly delayed in time from the actual pronunciation time, directly using the time of the probability peak as the word timestamp will lead to inaccurate start and end times of the word. In addition, since the audio may contain silence, noise, etc., this method will also include the time of such ineffective speech in the word timestamp, thus affecting the accuracy of word-granular timestamp. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a method for determining word-granular timestamps, an electronic device, and a storage medium to solve the technical problems of time offset of word-granular timestamp information and the duration of ineffective speech encapsulated words, and to improve the accuracy of word-granular timestamp information.

[0005] Firstly, a method for determining character-level timestamps is provided, the method comprising:

[0006] Determine the peak probability of each word in the target audio;

[0007] The time point corresponding to the peak probability of the current character is determined as the starting point of the current character;

[0008] In response to the fact that the current character is not the last character of the target audio, the termination point of the current character is determined based on the difference between the probability peak of the current character and the time point corresponding to the next adjacent probability peak.

[0009] The timestamp of the current word is determined based on the starting point and the ending point.

[0010] In some embodiments, determining the probability peak value of each word in the target audio includes:

[0011] The target audio is decoded using a predetermined decoding algorithm to determine the probability peak value of each word in the target audio.

[0012] In some embodiments, determining the termination point of the current character based on the difference between the time point corresponding to the probability peak of the current character and the next adjacent probability peak includes:

[0013] In response to the difference not being greater than a first predetermined duration, the time point corresponding to the next adjacent probability peak is determined as the termination point of the current character.

[0014] In some embodiments, determining the timestamp of the current word based on the start point and the end point includes:

[0015] The starting point and the ending point are moved forward by a third predetermined time period, respectively.

[0016] The timestamp of the current word is determined based on the starting and ending points after the movement.

[0017] In some embodiments, determining the termination point of the current character based on the difference between the time point corresponding to the probability peak of the current character and the next adjacent probability peak includes:

[0018] In response to the difference being greater than a first predetermined duration, the time point obtained after moving the starting point backward by a second predetermined duration is determined as the termination point of the current character.

[0019] In some embodiments, determining the timestamp of the current word based on the start point and the end point includes:

[0020] Move the starting point forward for a third predetermined time period;

[0021] The timestamp of the current word is determined based on the starting point and the ending point after the movement.

[0022] In some embodiments, the length of the third predetermined duration is less than the difference between the start point and the end point.

[0023] In some embodiments, the method further includes:

[0024] In response to the fact that the current character is the last character of the target audio, the difference between the time point corresponding to the probability peak of the current character and the time point at which the target audio ends is determined;

[0025] In response to the fact that the difference between the time point corresponding to the probability peak of the current character and the time point at which the target audio ends is not greater than a second predetermined duration, the time point at which the target audio ends is determined as the termination point of the current character;

[0026] If the difference between the time point corresponding to the probability peak of the current character and the time point at which the target audio ends is greater than a second predetermined duration, the time point obtained after moving the starting point backward by the second predetermined duration is determined as the ending point of the current character.

[0027] Secondly, a granularity timestamp determination device is provided, the device comprising:

[0028] The first determining module is configured to determine the peak probability of each word in the target audio.

[0029] The second determining module is configured to determine the time point corresponding to the probability peak of the current character as the starting point of the current character;

[0030] The third determining module is configured to, in response to the current character not being the last character of the target audio, determine the termination point of the current character based on the difference between the probability peak of the current character and the time point corresponding to the next adjacent probability peak.

[0031] The fourth determining module is configured to determine the timestamp of the current word based on the starting point and the ending point.

[0032] Thirdly, an electronic device is provided, including a memory and a processor, the memory being used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as described in the first aspect.

[0033] Fourthly, a computer-readable storage medium is provided, wherein a computer program is stored therein, and the computer program, when executed by a processor, implements the method described in the first aspect.

[0034] This invention addresses the issue of time offset in character-level timestamps and the duration of ineffective speech inclusion characters by determining the time point corresponding to the probability peak of the current character as its starting point and the time point corresponding to the next adjacent probability peak as its ending point. Finally, it determines the timestamp of the current character based on the starting and ending points. This approach solves the technical problems of time offset in character-level timestamp information and the impact on the accuracy of character-level timestamps by improving the accuracy of these timestamps. Attached Figure Description

[0035] The above and other objects, features and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which:

[0036] Figure 1 This is a flowchart of a method for determining word-granularity timestamps according to an embodiment of the present invention;

[0037] Figure 2This is a flowchart of the method for determining the termination point of the last character according to an embodiment of the present invention;

[0038] Figure 3 This is a flowchart of a method for determining the granularity timestamp of a non-tail character according to an embodiment of the present invention;

[0039] Figure 4 This is a flowchart of another method for determining the granularity timestamp of a non-tail character according to an embodiment of the present invention;

[0040] Figure 5 This is a flowchart of a method for determining the granularity timestamp of the last character according to an embodiment of the present invention;

[0041] Figure 6 This is a schematic diagram illustrating the process of determining the character-level timestamp in a target audio file according to an embodiment of the present invention;

[0042] Figure 7 This is a flowchart of another method for determining the granularity timestamp of a non-tail character according to an embodiment of the present invention;

[0043] Figure 8 This is a schematic diagram of the character granularity timestamp determination device according to an embodiment of the present invention;

[0044] Figure 9 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0045] The present application is described below based on embodiments, but it is not limited to these embodiments. In the detailed description of the present application below, certain specific details are described in detail. Those skilled in the art can fully understand the present application without these details. To avoid obscuring the substance of the present application, well-known methods, processes, flows, elements, and circuits are not described in detail.

[0046] Furthermore, those skilled in the art should understand that the accompanying drawings provided herein are for illustrative purposes only and are not necessarily drawn to scale.

[0047] Unless the context explicitly requires it, words such as "including" or "contains" throughout the application should be interpreted as including rather than exclusive or exhaustive; that is, meaning "including but not limited to".

[0048] In the description of this application, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0049] Figure 1This is a flowchart of a method for determining word-level timestamps according to an embodiment of the present invention. Figure 1 As shown, the word-granularity timestamp determination method in this embodiment includes the following steps:

[0050] Step S110: Determine the probability peak value of each word in the target audio.

[0051] In some embodiments, the target audio can be decoded using a predetermined decoding algorithm (e.g., the CTC algorithm) to determine the probability peak of each word in the target audio. CTC is a fully end-to-end acoustic model training method that does not require pre-alignment of data; it only requires an input sequence and an output sequence for training. This eliminates the need for data alignment and annotation, and CTC directly outputs the predicted probabilities of the sequence without external post-processing. Therefore, CTC can calculate the probability of any output value based on a given input. In this embodiment, CTC determines the probability peak of a word by decoding the audio and calculating the probability of the word within its duration frames.

[0052] Step S120: Determine the time point corresponding to the probability peak of the current character as the starting point of the current character.

[0053] Step S130: In response to the fact that the current character is not the last character of the target audio, the termination point of the current character is determined based on the difference between the probability peak value of the current character and the time point corresponding to the next adjacent probability peak value.

[0054] Since the probability peak of a character corresponds one-to-one with the character, when the current character is not the last character in the target audio, there is a next probability peak adjacent to the probability peak of the current character. Therefore, in this embodiment, the termination point of the current character can be determined based on the difference between the probability peak of the current character and the time point corresponding to the next adjacent probability peak.

[0055] Specifically, the termination point of the current character is determined based on whether the difference between the time point corresponding to the probability peak of the current character and the time point corresponding to the next adjacent probability peak is greater than a first predetermined duration. The first predetermined duration includes the actual duration of a character and a reserved time, where the reserved time is the duration of reasonable pauses caused by individual speech speed or phrasing habits. Therefore, when the difference is not greater than the first predetermined duration, it can be considered that the target audio contains valid speech information within the time period between the two probability peaks, i.e., there are no invalid speech segments such as silence or noise. In this case, the time point corresponding to the next adjacent probability peak can be directly determined as the termination point of the current character.

[0056] When the difference is greater than the first predetermined duration (i.e., the duration exceeds the actual duration of a character and the reserved time), it indicates that the target audio may contain ineffective speech segments such as silence or noise between the two probability peaks. In this case, the time point obtained after shifting the starting point backward by the second predetermined duration is determined as the ending point of the current character. The second predetermined duration is not less than the length of a character. This reduces ineffective speech within the duration of the included character. It should be understood that both the first and second predetermined durations can be preset according to actual needs. In some implementations, the first and second predetermined durations can be set to fixed values ​​based on experience. In other implementations, corresponding first and second predetermined durations can be set for the language characteristics of each target audio segment. The language characteristics are used to characterize each person's speaking habits, including the duration of each character and the time interval between characters.

[0057] In some implementations, the first predetermined duration is longer than the second predetermined duration. In other implementations, the first predetermined duration is equal to the second predetermined duration. When the current word is not the last word of the target audio, the termination point of the current word is determined based on the relationship between the difference between two probability peaks and the first predetermined duration. When the first predetermined duration is equal to the second predetermined duration, since the second predetermined duration only includes the duration of one word, the termination point of the current word is determined based on the relationship between the difference between two probability peaks and the duration of one word, without considering the reserved time for reasonable pauses caused by each person's speaking speed or phrasing habits. However, in real-world scenarios, the language habits of the speakers in the target audio can affect the actual termination point of the word to some extent. Therefore, in practical applications, it is preferable for the first predetermined duration to be longer than the second predetermined duration.

[0058] For example, a first predetermined duration can be set to 350ms, and a second predetermined duration to 200ms. Since the first predetermined duration includes the actual duration of a word and the duration of reasonable pauses due to individual speech patterns such as pace or phrasing, if the difference between the probability peak of the current word and the time point corresponding to the next adjacent probability peak is greater than 350ms, it indicates that there may be a segment of ineffective speech after the actual duration of the word. This could include noise, long pauses during speech, etc. The determination of the termination point can reduce the corresponding ineffective speech in the target audio. Specifically, by determining the termination point of the current word at the time point obtained 200ms backward from the starting point, ineffective speech can be excluded from the word's duration, thereby reducing ineffective speech in the timestamps of the included words and improving the accuracy of word-level timestamps.

[0059] If the difference between the time point corresponding to the probability peak of the current character and the time point corresponding to the next adjacent probability peak is no greater than 350ms, it can be assumed that there is no invalid speech after the actual duration of the current character ends. Therefore, the time point corresponding to the next adjacent probability peak can be directly determined as the termination point of the current character. Thus, by determining the difference between the time point corresponding to the probability peak of the current character and the time point corresponding to the next probability peak, and then determining the termination point of the current character based on this difference, the accuracy of the subsequent character-level timestamp information determined based on the termination point can be improved.

[0060] When the current character is the last character in the target audio, there is no next probability peak adjacent to the probability peak of the current character. In this case, the termination point of the current character (i.e., the last character) can be determined by, for example... Figure 2 The steps shown are determined, and the specific steps include:

[0061] Step S210: In response to the current character being the last character of the target audio, determine the difference between the time point corresponding to the probability peak of the current character and the time point at which the target audio ends.

[0062] Specifically, if the current character is the last character in the target audio, then the difference between the time point corresponding to the probability peak of the current character and the time point at which the target audio ends is calculated.

[0063] Step S220: In response to the fact that the difference between the time point corresponding to the probability peak of the current character and the time point at which the target audio ends is not greater than a second predetermined duration, the time point at which the target audio ends is determined as the termination point of the current character.

[0064] Since the second predetermined duration is the duration of a single character, if the difference is not greater than the second predetermined duration, it indicates that there is no silence, noise, or other invalid speech after the last character in the target audio. Or, even if there is invalid speech, its impact on the accuracy of the subsequently determined character-level timestamp is very small due to the short duration, and can be ignored. Therefore, the end time of the target audio can be directly determined as the termination point of the last character.

[0065] Step S230: In response to the difference between the time point corresponding to the probability peak of the current character and the time point at which the target audio ends being greater than a second predetermined duration, the time point obtained after moving the starting point backward by the second predetermined duration is determined as the end point of the current character.

[0066] Specifically, if the difference is greater than the second predetermined duration, it indicates that there is non-valid speech such as silence after the last word in the target audio. By moving the starting point backward by the second predetermined duration and determining the time point obtained as the end point of the last word, the non-valid speech in the duration of the enclosed word can be reduced, making the accuracy of the word-granularity timestamp information determined based on the end point higher.

[0067] For example, a second predetermined duration (i.e., the duration of a single character) can be preset to 200ms. If the current character is the last character, it is determined whether the difference between the time point corresponding to the probability peak of the last character and the time point at the end of the target audio is greater than 200ms. If it is not greater than 200ms, it can be assumed that there is no silence, noise, or other invalid speech after the duration of the last character ends. Therefore, the time point at the end of the target audio can be directly determined as the termination point of the last character. If the difference between the time point corresponding to the probability peak of the last character and the time point at the end of the target audio is greater than 200ms, it indicates that there is silence or other invalid speech after the last character in the target audio. By moving the starting point backward by 200ms and determining the time point obtained as the termination point of the last character, invalid speech in the duration of the last character can be reduced. Thus, by judging the difference between the time point corresponding to the probability peak of the last character and the time point at the end of the target audio, and then determining the termination point of the last character based on the above difference, the accuracy of the subsequent character-level timestamp information determined based on the termination point can be improved.

[0068] Step S140: Determine the timestamp of the current word based on the start point and the end point.

[0069] The timestamp of the current character includes the start and end times of the current character in the audio. Specifically, because the termination point of the current character is affected by the difference between the time point corresponding to the probability peak of the current character and the next adjacent probability peak, as well as the position of the character, the specific steps for determining the timestamp of the current character based on the start and end points differ in different situations.

[0070] If the current character is not the last character in the target audio, there is a next probability peak adjacent to the probability peak of the current character. Therefore, in this embodiment, the termination point of the current character can be determined based on the time difference between the probability peak of the current character and the next probability peak.

[0071] In some implementations, if the difference between the time point corresponding to the probability peak of the current character and the time point corresponding to the next adjacent probability peak is not greater than a first predetermined duration, the step of determining the timestamp of the current character based on the start point and the end point is as follows: Figure 3 As shown, it specifically includes:

[0072] Step S141: Move the starting point and the ending point forward by a third predetermined time period, respectively.

[0073] The starting point is the time point corresponding to the probability peak of the current character, and the ending point is the time point corresponding to the next probability peak adjacent to the probability peak of the current character. The length of the third predetermined duration is less than the difference between the starting point and the ending point, and the third predetermined duration can be preset. For example, based on experience, the third predetermined duration can be set to 50ms.

[0074] Step S142: Determine the timestamp of the current character based on the starting point and ending point after the movement.

[0075] Specifically, the timestamp of the current character is determined based on the starting point and ending point after moving forward for a third predetermined time. Thus, the start and end times of the character are determined by offsetting the time point corresponding to the probability peak. The timestamp of the current character is then determined based on the start and end times of the character. This can compensate for the offset of the start and end times of the character in the character-granular timestamp caused by the probability peak predicted by CTC being slightly delayed in time compared to the actual pronunciation time point, thereby improving the accuracy of the character-granular timestamp.

[0076] In some implementations, if the difference between the time point corresponding to the probability peak of the current character and the time point corresponding to the next adjacent probability peak is greater than a first predetermined duration, the step of determining the timestamp of the current character based on the start point and the end point is as follows: Figure 4 As shown, it specifically includes:

[0077] Step S143: Move the starting point forward for a third predetermined time period.

[0078] The starting point is the time point corresponding to the probability peak of the current character. Moving the starting point forward by a third predetermined time period can compensate for the offset of the character's start time in the character-granular timestamp caused by the slight time delay between the probability peak predicted by CTC and the actual pronunciation time point. Subsequently, the character-granular timestamp can be determined based on the moved starting point, which can improve the accuracy of the character-granular timestamp.

[0079] Step S144: Determine the timestamp of the current character based on the starting point and the ending point after the movement.

[0080] The termination point is the time point obtained after moving the starting point backward by a second predetermined time period.

[0081] If the current character is the last character in the target audio, the step of determining the timestamp of the current character based on the start point and the end point is as follows: Figure 5 As shown, it specifically includes:

[0082] Step S145: Move the starting point forward by a third preset duration.

[0083] Wherein, the starting point is the time point corresponding to the probability peak of the last word.

[0084] Step S146: Determine the time stamp of the last word according to the moved starting point and the ending point.

[0085] Wherein, the specific determination method of the ending point can refer to steps S210 - S230, which will not be elaborated here.

[0086] Figure 6 Illustrates the process of determining the word - granularity time stamp for a target audio with an audio length of 1000 ms and the audio content of "Hello everyone". The first preset duration is preset to be 350 ms, the second preset duration is 200 ms, and the third preset duration is 50 ms. As Figure 6 shown, the total length T of the target audio is 1000 ms. Among them, the time point t1 corresponding to the probability peak of the word "da" is at the 200th ms of the target audio, the time point t2 corresponding to the probability peak of the word "jia" is at the 380th ms of the target audio, and the time point t3 corresponding to the probability peak of the word "hao" is at the 750th ms of the target audio. Obviously, the last word of this paragraph of the target audio is "hao", and "da" and "jia" are not the last words of this paragraph of audio.

[0087] First, determine the time point t1 corresponding to the probability peak of the word "da" as the starting point of the word "da". Since the difference between the time point t1 corresponding to the probability peak of the word "da" and the time point t2 corresponding to the next adjacent probability peak (i.e., the probability peak of the word "jia") is 180 ms, which is less than 350 ms, the time point t2 can be directly determined as the ending point of the word "da". Since the probability peak predicted by CTC is slightly delayed in time compared to the time point corresponding to the real pronunciation, in this embodiment, both the time point t1 corresponding to the starting point of the word "da" and the time point t2 corresponding to the ending point are moved forward by 50 ms. Then, based on the time points t10 (i.e., the 150th ms of the target audio) and t​​Next, the time point t2 corresponding to the probability peak of the character "家" is determined as the starting point of the character "家". Since the difference between the time point t2 corresponding to the probability peak of the character "家" and the time point t3 corresponding to the next adjacent probability peak (i.e., the probability peak of the character "好") is 370 ms, which is greater than 350 ms, the time point t21 obtained by moving the time point t2 backward by 200 ms (i.e., the 580 ms of the target audio) is determined as the ending point of the character "家". Since only the time point t2 corresponding to the starting point is the time corresponding to the probability peak among the starting point and the ending point of the character "家" at this time, only the time point t2 corresponding to the starting point needs to be moved forward by 50 ms, and then the character granularity timestamp of the character "家" is determined based on the time point t21 corresponding to the ending point and the time point t20 obtained after moving the starting point. That is to say, in the character granularity timestamp of the character "家" finally determined in this embodiment, the start time of the character "家" is t20, that is, the 330 ms of the target audio, and the end time is t21, that is, the 580 ms of the target audio.

[0089] Finally, the time point t3 corresponding to the probability peak of the character "好" is determined as the starting point of the character "好". Since the difference between the time point t3 corresponding to the probability peak of the character "好" and the time point at the end of the target audio is 250 ms, which is greater than 200 ms, the time point t31 obtained by moving the time point t3 backward by 200 ms (i.e., the 950 ms of the target audio) is determined as the ending point of the character "好". Since only the time point t3 corresponding to the starting point is the time corresponding to the probability peak among the starting point and the ending point of the character "好" at this time, only the time point t3 corresponding to the starting point needs to be moved forward by 50 ms, and then the character granularity timestamp of the character "好" is determined based on the time point t31 corresponding to the ending point and the time point t30 obtained after moving the starting point (i.e., the 700 ms of the target audio). That is to say, in the character granularity timestamp of the character "好" finally determined in this embodiment, the start time of the character "好" is t3, that is, the 700 ms of the target audio, and the end time is t31, that is, the 950 ms of the target audio. Thus, in this embodiment, by shifting both the predicted start time and end time corresponding to the probability peak forward by the third predetermined duration and then determining the character granularity timestamp based on the time obtained after the shift, the offset of the start time and end time of the character in the character granularity timestamp caused by the fact that the probability peak predicted by CTC is slightly delayed in time compared to the time point corresponding to the real pronunciation is reduced, and the accuracy of the character granularity timestamp is improved.

[0090] In some implementations, after processing the target audio according to the steps of the word-granularity timestamp determination method described in this embodiment, the number of characters in the target audio and the total duration of effective speech can be obtained. Based on the number of characters and the total duration of effective speech in the target audio, the average duration of a single character in the corresponding audio is determined, thereby determining a new second predetermined duration. The steps of the word-granularity timestamp determination method described in this embodiment are then re-executed based on the determined new second predetermined duration to determine the final word-granularity timestamp. Thus, by setting a corresponding second predetermined duration for a specific target audio, the second predetermined duration is correlated with the speaker's speech rate and other language habits in the target audio, thereby increasing the accuracy of the set second predetermined duration and improving the accuracy of the word-granularity timestamp.

[0091] This invention addresses the issue of time offset in character-level timestamps and the duration of ineffective speech inclusion characters by determining the time point corresponding to the probability peak of the current character as its starting point and the time point corresponding to the next adjacent probability peak as its ending point. Finally, it determines the timestamp of the current character based on the starting and ending points. This approach solves the technical problems of time offset in character-level timestamp information and the impact on the accuracy of character-level timestamps by improving the accuracy of these timestamps.

[0092] Figure 7 This is a flowchart illustrating another method for determining the granularity timestamp of a non-tailed character according to an embodiment of the present invention. For example... Figure 7 As shown, the word-granularity timestamp determination method in this embodiment includes the following steps:

[0093] Step S701: Determine the probability peak value of each word in the target audio.

[0094] In some embodiments, the target audio can be decoded using a predetermined decoding algorithm to determine the probability peak value of each word in the target audio. The decoding algorithm is the CTC decoding algorithm.

[0095] Step S702: Determine the time point corresponding to the probability peak of the current character as the starting point of the current character.

[0096] Step S703: Determine whether the current character is the last character of the target audio.

[0097] If the current character is the last character of the target audio, then the character granularity timestamp of the last character is determined according to steps S704-S708; otherwise, the corresponding character granularity timestamp is determined according to steps S707-S712 and step S705.

[0098] Since there is a one-to-one correspondence between the probability peaks of characters, when the current character is not the last character in the target audio, a next probability peak adjacent to the probability peak of the current character exists. Therefore, in this embodiment, the termination point of the current character can be determined based on the time difference between the probability peak of the current character and the time point corresponding to the next adjacent probability peak. When the current character is the last character in the target audio, a next probability peak adjacent to the probability peak of the current character does not exist. In this case, the termination point of the last character needs to be determined based on the relationship between the probability peak of the current character (i.e., the last character) and the end time of the target audio.

[0099] Specifically, if the result of step S703 is that the current character is the last character in the target audio, the specific process of determining the character-granularity timestamp of the last character according to steps S704-S708 is as follows:

[0100] Step S704: Determine whether the difference between the time point corresponding to the probability peak of the current word and the time point at which the target audio ends is greater than the second predetermined duration.

[0101] If the difference between the time point corresponding to the probability peak of the current word and the time point at which the target audio ends is greater than the second predetermined duration, proceed to step S705; otherwise, proceed to step S706.

[0102] Step S705: The time point obtained after moving the starting point backward by a second predetermined time period is determined as the end point of the last character.

[0103] The second predetermined duration is the duration of one character.

[0104] Specifically, since the second predetermined duration is the duration of a single character, if the difference is greater than the second predetermined duration, it indicates that there is non-valid speech such as silence after the last character in the target audio. By moving the starting point backward by the second predetermined duration and determining the time point obtained as the end point of the last character, the non-valid speech in the duration of the enclosed character can be reduced, making the accuracy of the subsequent character-granular timestamp information determined based on the end point higher.

[0105] Step S706: Determine the end time of the target audio as the termination point of the current word.

[0106] Specifically, if the above difference is not greater than the second predetermined duration, it indicates that there is no silence, noise or other invalid speech after the last word in the target audio, or even if there is invalid speech, its impact on the accuracy of the subsequently determined word-granularity timestamp is very small due to the short duration, and can be ignored. Therefore, the end time of the target audio can be directly determined as the termination point of the last word.

[0107] Based on the difference between the time point corresponding to the probability peak of the current character and the time point at which the target audio ends, selecting to execute either step S705 or step S706 above can determine the termination point of the last character accordingly.

[0108] Step S707: Move the starting point forward by a third predetermined time period.

[0109] Step S708: Determine the timestamp of the current character based on the corresponding termination point and the starting point after the movement.

[0110] The starting point is the time point corresponding to the probability peak of the current character, and the ending point is the ending point determined in step S705 or step S706. Since the starting point corresponds to the probability peak, it may be slightly delayed in time from the time point corresponding to the actual pronunciation. This step determines the timestamp of the current character based on the ending point and the moved starting point, which can reduce the impact of the character's starting time offset on the accuracy of the character-level timestamp.

[0111] This enables the determination of the character-level timestamp when the current character is the last character.

[0112] If the result of step S703 is that the current character is not the last character in the target audio, the process of determining the corresponding character granularity timestamp according to steps S707-S712 and step S705 is as follows:

[0113] If the result of step S703 is that the current character is not the last character in the target audio, proceed to step S709.

[0114] Step S709: Determine whether the difference between the probability peak of the current character and the time point corresponding to the next adjacent probability peak is greater than the first predetermined duration.

[0115] The first predetermined duration includes the actual duration of a word and a reserved time, where the reserved time is the duration of reasonable pauses caused by each person's speaking speed or phrasing habits. If the difference between the probability peak of the current word and the time point corresponding to the next adjacent probability peak is greater than the first predetermined duration, it indicates that there may be a long period of ineffective speech after the actual duration of the word, such as noise or long pauses during speech. In this case, proceed to step S705, and execute steps S705, S707, and S712 in sequence to determine the corresponding word granularity timestamp. Otherwise, proceed to step S710, and execute steps S710-S712 in sequence to determine the corresponding word granularity timestamp.

[0116] Specifically, if the result of step S709 is that the difference between the time point corresponding to the current probability peak and the next adjacent probability peak is greater than the first predetermined duration, then the following steps are executed sequentially to determine the corresponding word granularity timestamp:

[0117] Step S705: The time point obtained after moving the starting point backward by a second predetermined time is determined as the end point of the current character.

[0118] Step S707: Move the starting point forward by a third predetermined time period.

[0119] Step S708: Determine the timestamp of the current character based on the corresponding termination point and the starting point after the movement.

[0120] The starting point is the time point corresponding to the probability peak of the current character. Since the starting point corresponds to the probability peak, it may be slightly delayed in time from the time point corresponding to the actual pronunciation. This step determines the timestamp of the current character based on the ending point and the moved starting point, which can reduce the impact of the offset of the character's starting time on the accuracy of the character granular timestamp.

[0121] If the result of step S709 is that the difference between the time point corresponding to the current word's probability peak and the next adjacent probability peak is not greater than the first predetermined duration, then the following steps are executed sequentially to determine the corresponding word granularity timestamp:

[0122] Step S710: Determine the time point corresponding to the next adjacent probability peak as the termination point of the current character.

[0123] If the difference between the probability peak of the current character and the time point corresponding to the next adjacent probability peak is not greater than the first predetermined duration, it can be considered that there is no invalid speech after the actual duration of the current character ends. Therefore, the time point corresponding to the next adjacent probability peak can be directly determined as the termination point of the current character.

[0124] Step S711: Move the starting point and the ending point forward by a third predetermined time period, respectively.

[0125] Wherein, the length of the third predetermined duration is less than the difference between the starting point and the ending point.

[0126] Step S712: Determine the timestamp of the current character based on the starting point and ending point after the movement.

[0127] The starting point is the time point corresponding to the probability peak of the current character, and the ending point is the ending point determined in step S710. Since both the starting point and the ending point correspond to the probability peak, they may be slightly delayed in time from the time point corresponding to the actual pronunciation. This step reduces the impact of the offset of the character's start and end times on the accuracy of the character-level timestamp by determining the timestamp of the current character based on the moved starting and ending points.

[0128] This invention addresses the issue of time offset in character-level timestamps and the duration of ineffective speech inclusion characters by determining the time point corresponding to the probability peak of the current character as its starting point and the time point corresponding to the next adjacent probability peak as its ending point. Finally, it determines the timestamp of the current character based on the starting and ending points. This approach solves the technical problems of time offset in character-level timestamp information and the impact on the accuracy of character-level timestamps by improving the accuracy of these timestamps.

[0129] Figure 8 This is a schematic diagram of a character-granularity timestamp determination device according to an embodiment of the present invention. Figure 8 As shown, the device includes a first determining module 81, a second determining module 82, a third determining module 83, and a fourth determining module 84.

[0130] The first determining module 81 is configured to determine the probability peak value of each character in the target audio. The second determining module 82 is configured to determine the time point corresponding to the probability peak value of the current character as the starting point of the current character. The third determining module 83 is configured to determine the ending point of the current character based on the difference between the time point corresponding to the probability peak value of the current character and the next adjacent probability peak value, in response to the current character not being the last character of the target audio. The fourth determining module 84 is configured to determine the timestamp of the current character based on the starting point and the ending point.

[0131] Furthermore, the third determining module 83 further includes a first termination point determining unit and a second termination point determining unit, wherein the first termination point determining unit is configured to determine the time point corresponding to the adjacent next probability peak as the termination point of the current character in response to the difference not being greater than a first predetermined duration. The second termination point determining unit is configured to determine the time point obtained after moving the starting point backward by a second predetermined duration as the termination point of the current character in response to the difference being greater than the first predetermined duration.

[0132] This invention addresses the issue of time offset in character-level timestamps and the duration of ineffective speech inclusion characters by determining the time point corresponding to the probability peak of the current character as its starting point and the time point corresponding to the next adjacent probability peak as its ending point. Finally, it determines the timestamp of the current character based on the starting and ending points. This approach solves the technical problems of time offset in character-level timestamp information and the impact on the accuracy of character-level timestamps by improving the accuracy of these timestamps.

[0133] The fourth determining module 84 further includes a first moving unit, a first timestamp determining unit, a second moving unit, and a second timestamp determining unit. The first moving unit is configured to move the starting point and the ending point forward by a third predetermined time period, the first timestamp determining unit is configured to determine the timestamp of the current character based on the moved starting point and ending point, and the second moving unit is configured to move the starting point forward by a third predetermined time period. The second timestamp determining unit is configured to determine the timestamp of the current character based on the moved starting point and the ending point.

[0134] Optionally, the length of the third predetermined duration is less than the difference between the starting point and the ending point.

[0135] Optionally, the character-granularity timestamp determination device further includes a fifth determination module, a sixth determination module, and a seventh determination module. The fifth determination module is configured to, in response to the current character being the last character of the target audio, determine the difference between the time point corresponding to the probability peak of the current character and the time point at the end of the target audio. The sixth determination module is configured to, in response to the difference between the time point corresponding to the probability peak of the current character and the time point at the end of the target audio not being greater than a second predetermined duration, determine the time point at the end of the target audio as the termination point of the current character. The seventh determination module is configured to, in response to the difference between the time point corresponding to the probability peak of the current character and the time point at the end of the target audio being greater than a second predetermined duration, determine the time point obtained after moving the starting point backward by a second predetermined duration as the termination point of the current character.

[0136] This invention addresses the issue of time offset in character-level timestamps and the duration of ineffective speech inclusion characters by determining the time point corresponding to the probability peak of the current character as its starting point and the time point corresponding to the next adjacent probability peak as its ending point. Finally, it determines the timestamp of the current character based on the starting and ending points. This approach solves the technical problems of time offset in character-level timestamp information and the impact on the accuracy of character-level timestamps by improving the accuracy of these timestamps.

[0137] This invention also provides an electronic device. Figure 9 This is a schematic diagram of an electronic device according to an embodiment of the present invention. (For example...) Figure 9 As shown, Figure 9The illustrated electronic device is a general address lookup device, comprising a general computer hardware architecture, including at least a processor 91 and a memory 92. The processor 91 and memory 92 are connected via a bus 93. The memory 92 is adapted to store instructions or programs executable by the processor 91. The processor 91 can be a standalone microprocessor or a collection of one or more microprocessors. Thus, the processor 91 executes the instructions stored in the memory 92, thereby performing the method flow of the embodiments of the present invention as described above to process data and control other devices. The bus 93 connects the aforementioned components together, and also connects these components to a display controller 94, a display device, and an input / output (I / O) device 95. The input / output (I / O) device 95 can be a mouse, keyboard, modem, network interface, touch input device, motion-sensing input device, printer, and other devices known in the art. Typically, the input / output device 95 is connected to the system via an input / output (I / O) controller 96.

[0138] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus (devices), or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0139] This application is described with reference to flowchart illustrations of methods, apparatus (devices), and computer program products according to embodiments of this application. It should be understood that each step in the flowchart can be implemented by computer program instructions.

[0140] These computer program instructions may be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction means, the implementation process of which is described in the instruction means. Figure 1 The function specified in one or more processes.

[0141] These computer program instructions may also be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, produce instructions for implementing processes. Figure 1 A device for a function specified in one or more processes.

[0142] Another embodiment of the present invention relates to a non-volatile storage medium for storing a computer-readable program for use by a computer to execute some or all of the above-described method embodiments.

[0143] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program specifying the relevant hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0144] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for determining character-level timestamps, characterized in that, The method includes: Determine the peak probability of each word in the target audio; The time point corresponding to the peak probability of the current character is determined as the starting point of the current character; In response to the fact that the current character is not the last character of the target audio, the termination point of the current character is determined based on the difference between the probability peak of the current character and the time point corresponding to the next adjacent probability peak. The timestamp of the current character is determined based on the start point and the end point; Determining the termination point of the current character based on the difference between the time point corresponding to the probability peak of the current character and the next adjacent probability peak includes: In response to the difference not being greater than a first predetermined duration, the time point corresponding to the next adjacent probability peak is determined as the termination point of the current character; Determining the timestamp of the current word based on the start point and the end point includes: The starting point and the ending point are moved forward by a third predetermined time period, respectively. The timestamp of the current character is determined based on the starting and ending points after the movement; Wherein, the starting point is the time point corresponding to the probability peak of the current character, the ending point is the time point corresponding to the next probability peak adjacent to the probability peak of the current character, and the length of the third predetermined duration is less than the difference between the starting point and the ending point.

2. The method according to claim 1, characterized in that, The determination of the peak probability of each word in the target audio includes: The target audio is decoded using a predetermined decoding algorithm to determine the probability peak value of each word in the target audio.

3. The method according to claim 1, characterized in that, Determining the termination point of the current character based on the difference between the time point corresponding to the probability peak of the current character and the next adjacent probability peak includes: In response to the difference being greater than a first predetermined duration, the time point obtained after moving the starting point backward by a second predetermined duration is determined as the termination point of the current character.

4. The method according to claim 3, characterized in that, Determining the timestamp of the current word based on the start point and the end point includes: Move the starting point forward for a third predetermined time period; The timestamp of the current word is determined based on the starting point and the ending point after the movement.

5. The method according to claim 1, characterized in that, The method further includes: In response to the fact that the current character is the last character of the target audio, the difference between the time point corresponding to the probability peak of the current character and the time point at which the target audio ends is determined; In response to the fact that the difference between the time point corresponding to the probability peak of the current character and the time point at which the target audio ends is not greater than a second predetermined duration, the time point at which the target audio ends is determined as the termination point of the current character; If the difference between the time point corresponding to the probability peak of the current character and the time point at which the target audio ends is greater than a second predetermined duration, the time point obtained after moving the starting point backward by the second predetermined duration is determined as the ending point of the current character.

6. An electronic device comprising a memory and a processor, characterized in that, The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as described in any one of claims 1-5.

7. A computer-readable storage medium storing computer program instructions thereon, characterized in that, The computer program instructions, when executed by a processor, implement the method as described in any one of claims 1-5.

Images