Method, device, electronic device and storage medium for generating word-level timestamp

Abstract

The present application relates to a method for generating a word-level timestamp, an electronic device, and a storage medium, the method comprising: determining the probability peak of each word in a frame-by-frame decoding process; determining the time corresponding to the end point of each word according to the probability peak of each word; determining the time corresponding to the beginning point of each word according to the time corresponding to the end point of each word; and generating a word-level timestamp according to the time corresponding to the beginning point of each word and the time corresponding to the end point of each word. The present application determines the probability peak of each word based on the output score of a deep neural network and the changing rule of the score when each word is output during the decoding process, determines the time corresponding to the beginning point of each word and the time corresponding to the end point of each word according to the probability peak of each word, and provides a method for obtaining a word-level timestamp, which can output accurate timestamp information at the word level, thereby obtaining high-precision boundary information and improving the user experience.

Description

Technical Field

[0001] The present application relates to the technical field of timestamps, and in particular to a method, device, electronic device and storage medium for generating a word-level timestamp. Background Art

[0002] Traditional Kaldi-based speech recognition systems can obtain the boundary information of each word based on the grid. Although the popular end-to-end speech recognition systems in the industry have surpassed traditional systems in terms of recognition rate, many of them do not provide timestamp information, or only provide very rough timestamps, such as directly judging the boundary information of words based on neural network scores. Currently, there is no relatively mature algorithm that can obtain the timestamp information of each word. Summary of the invention

[0003] Based on the problem that a relatively mature algorithm can currently obtain the timestamp information of each word, the present application provides a method for generating a timestamp at a word level, an electronic device, and a storage medium.

[0004] In a first aspect, an embodiment of the present application provides a method for generating a word-level timestamp, comprising:

[0005] During the frame-by-frame decoding process, the probability peak of each word is determined;

[0006] Determine the time corresponding to the end point of each word according to the probability peak of each word;

[0007] Determine the time corresponding to the first endpoint of each word according to the time corresponding to the last endpoint of each word;

[0008] Generate a word-level timestamp based on the time corresponding to the first endpoint of each word and the time corresponding to the last endpoint of each word.

[0009] Furthermore, in the above-mentioned method for generating a word-level timestamp, determining the time corresponding to the end point of each word according to the probability peak of each word includes:

[0010] Compare the probability peak of each word with the current probability value of each word;

[0011] If the comparison result is that the difference between the probability peak value of each word and the current probability value of each word is greater than or equal to the preset threshold;

[0012] Determine the time corresponding to the current probability value as the time corresponding to the tail endpoint.

[0013] Furthermore, in the above-mentioned method for generating a word-level timestamp, determining the time corresponding to the end point of each word according to the probability peak of each word includes:

[0014] If after the current word ends, there is a silent segment, and the difference between the probability peak of each word and the current probability value of each word is less than the preset threshold;

[0015] The time corresponding to the probability peak of each word is postponed to a first preset time, and the time corresponding to the tail end point of each word is determined.

[0016] Furthermore, in the above-mentioned method for generating a word-level timestamp, the end point of each word determines the time corresponding to the beginning point of each word, including:

[0017] The time corresponding to the tail endpoint of each word is extended forward by a second preset time to determine the time corresponding to the head endpoint of each word.

[0018] Furthermore, the above-mentioned method for generating a word-level timestamp further includes:

[0019] The time corresponding to the beginning endpoint of each word is determined according to the probability peak of each word.

[0020] Furthermore, in the above-mentioned method for generating a word-level timestamp, determining the time corresponding to the beginning endpoint of each word according to the probability peak value of each word includes:

[0021] The time corresponding to the probability peak of each word is extended forward by a first preset time to determine the time corresponding to the beginning endpoint of each word.

[0022] Furthermore, in the above-mentioned method for generating a word-level timestamp, the probability peak is the log probability.

[0023] In a second aspect, an embodiment of the present application provides a device for generating a word-level timestamp, comprising:

[0024] The first determination module is used to determine the probability peak of each word during the frame decoding process;

[0025] The second determination module is used to determine the time corresponding to the end point of each word according to the probability peak of each word;

[0026] The third determining module is used to determine the time corresponding to the beginning endpoint of each word according to the time corresponding to the end endpoint of each word;

[0027] The fourth determination module is used to generate a word-level timestamp according to the time corresponding to the first endpoint of each word and the time corresponding to the last endpoint of each word.

[0028] In a third aspect, an embodiment of the present application further provides an electronic device, characterized in that it includes: a processor and a memory;

[0029] The processor is used to execute the above-mentioned method for generating a word-level timestamp by calling the program or instruction stored in the memory.

[0030] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores a program or instruction, and the program or instruction enables the computer to implement the above-mentioned method for generating a word-level timestamp.

[0031] The advantages of the embodiments of the present application are: the present application relates to a method for generating a timestamp at the word level, an electronic device and a storage medium, the method comprising: determining the probability peak of each word in the frame-by-frame decoding process; determining the time corresponding to the end point of each word according to the probability peak of each word; determining the time corresponding to the beginning point of each word according to the time corresponding to the end point of each word; generating a timestamp at the word level according to the time corresponding to the beginning point of each word and the time corresponding to the end point of each word. The present application determines the probability peak of each word based on the output score of a deep neural network and the changing law of the score when each word is output during the decoding process, determines the time corresponding to the beginning point of each word and the time corresponding to the end point of each word according to the probability peak of each word, provides a method for obtaining a timestamp at the word level, can output accurate timestamp information at the word level, obtains high-precision boundary information, and improves the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the conventional technology, the drawings required for use in the embodiments or the conventional technology descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0033] Figure 1 A method for generating a word-level timestamp provided in an embodiment of the present application is shown in FIG. Figure 1 ;

[0034] Figure 2 A method for generating a word-level timestamp provided in an embodiment of the present application is shown in FIG. Figure 2 ;

[0035] Figure 3 A method for generating a word-level timestamp provided in an embodiment of the present application is shown in FIG. Figure 3 ;

[0036] Figure 4 A schematic diagram of a device for generating a word-level timestamp provided in an embodiment of the present application;

[0037] Figure 5 It is a schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.

[0039] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0040] The following first introduces the technical background of this application:

[0041] The neural network model of the end-to-end speech recognition engine generally outputs a T x M matrix. T represents the number of audio frames, and M represents the size of the dictionary. The element ixj in the matrix represents the probability of the model outputting j at time i, and log probability is generally used. Subsequently, based on this matrix, a decoding algorithm (such as CTC Prefix Beam Search, Time SyncDecoding, Align length Sync Decoding, etc.) is used to obtain the final recognition result. The boundary information timestamp of each word is obtained during the decoding process.

[0042] In the frame-by-frame decoding process, each frame has an optimal path, and its score is the sum of the log probabilities of all moments along the path. Under normal circumstances, a word will cover several frames, and the probability of the entire path will have a general pattern from the beginning to the complete appearance: from small to large, and then stable or jump to the next word. Because there is little information at the beginning, the probability of matching this word will not be very high. As the decoding time progresses, it will become more and more "similar" to this word, that is, the probability gradually increases. The subsequent probability may be relatively stable to encounter silence or jump to the next word.

[0043] Figure 1 A method for generating a word-level timestamp provided in an embodiment of the present application is shown in FIG. Figure 1 .

[0044] In a first aspect, the present application embodiment provides a method for generating a word-level timestamp, combining Figure 1 , including four steps from S101 to S104:

[0045] S101: During the frame-by-frame decoding process, determine the probability peak of each word.

[0046] Specifically, in the embodiment of the present application, in the frame-by-frame decoding process, determining the probability peak of each word is to determine the maximum log probability score of each word when it appears as the latest word.

[0047] S102: Determine the time corresponding to the end point of each word according to the probability peak of each word.

[0048] Specifically, in the embodiment of the present application, after determining the maximum log probability score of each word, the time corresponding to the tail endpoint of each word is determined according to the maximum log probability score of each word. The time corresponding to the tail endpoint of each word is introduced below in combination with specific steps.

[0049] S103: Determine the time corresponding to the beginning endpoint of each word according to the time corresponding to the end endpoint of each word.

[0050] Specifically, in the embodiment of the present application, after determining the time corresponding to the tail endpoint of each word, the time corresponding to the tail endpoint of each word can be moved forward by approximately one word time, and the time corresponding to the head endpoint of each word can be determined, which is introduced below in conjunction with specific examples.

[0051] S104: Generate a word-level timestamp according to the time corresponding to the first endpoint of each word and the time corresponding to the last endpoint of each word.

[0052] Specifically, in the embodiment of the present application, the time corresponding to the first endpoint of each word and the time corresponding to the last endpoint of each word are determined, and the timestamp of each word can be determined based on the time between the first endpoint and the last endpoint.

[0053] Figure 2 A method for generating a word-level timestamp provided in an embodiment of the present application is shown in FIG. Figure 2 .

[0054] Furthermore, in the above-mentioned method for generating a word-level timestamp, the time corresponding to the end point of each word is determined according to the probability peak of each word, combined with Figure 2 , including two steps S201 to S202:

[0055] S201: Compare the probability peak value of each word with the current probability value of each word;

[0056] S202: If the comparison result is that the difference between the probability peak value of each word and the current probability value of each word is greater than or equal to a preset threshold;

[0057] S203: Determine that the time corresponding to the current probability value is the time corresponding to the tail endpoint.

[0058] Specifically, in the embodiment of the present application, if the current word lasts for a period of time and then immediately jumps to the next word. At this time, it is necessary to find the jump point of the current word, compare the probability peak of each word with the current probability value of each word, set a relative threshold of 0.1%, and compare the probability value of the current word with the probability peak. If the comparison result is that the difference between the probability peak of each word and the current probability value of each word is greater than or equal to the preset threshold; if the difference is the preset threshold such as 0.1%, determine that the time corresponding to the current probability value is the time corresponding to the end point.

[0059] Figure 3 A method for generating a word-level timestamp provided in an embodiment of the present application is shown in FIG. Figure 3 .

[0060] Furthermore, in the above-mentioned method for generating a word-level timestamp, the time corresponding to the end point of each word is determined according to the probability peak of each word, combined with Figure 3 , including two steps S301 to S302:

[0061] S301: if after the current word ends, there is a silent segment, and the difference between the probability peak value of each word and the current probability value of each word is less than a preset threshold;

[0062] S302: Delay the time corresponding to the probability peak of each word by a first preset time to determine the time corresponding to the end point of each word.

[0063] Specifically, in the embodiment of the present application, if a period of silence is started after the current word continues to end, the score is lower than the probability peak by no more than 0.1% of the preset threshold value, and it will last for a long time, then the time of the probability peak is postponed by a first preset time such as 120ms, which is about half a word, to determine the time corresponding to the end point of each word.

[0064] Furthermore, in the above-mentioned method for generating a word-level timestamp, the end point of each word determines the time corresponding to the beginning point of each word, including:

[0065] The time corresponding to the tail endpoint of each word is extended forward by a second preset time to determine the time corresponding to the head endpoint of each word.

[0066] Specifically, in an embodiment of the present application, after determining the time corresponding to the tail endpoint of each word, the time corresponding to the tail endpoint can be moved forward by a second preset time such as 240ms, which is approximately the time of one word, to determine the time corresponding to the head endpoint of each word.

[0067] Furthermore, the above-mentioned method for generating a word-level timestamp further includes:

[0068] The time corresponding to the beginning endpoint of each word is determined according to the probability peak of each word.

[0069] Specifically, in the embodiment of the present application, in addition to comparing the probability peak value of each word with the current probability value of each word as described above, the time corresponding to the starting endpoint of each word can also be determined by the time corresponding to the probability peak value of each word.

[0070] Furthermore, in the above-mentioned method for generating a word-level timestamp, determining the time corresponding to the beginning endpoint of each word according to the probability peak value of each word includes:

[0071] The time corresponding to the probability peak of each word is extended forward by a first preset time to determine the time corresponding to the beginning endpoint of each word.

[0072] Specifically, in the embodiment of the present application, the time corresponding to the probability peak of each word is shifted forward by a first preset time of 120 ms, and the time corresponding to the beginning endpoint of each word is determined by half a word.

[0073] Furthermore, in the above-mentioned method for generating a word-level timestamp, the probability peak is the log probability.

[0074] Specifically, in the embodiment of the present application, the neural network model of the end-to-end speech recognition engine generally outputs a matrix of Tx M. T represents the number of audio frames, and M represents the size of the dictionary. The element ixj in the matrix represents the probability of the model outputting j at time i, and log probability is generally used, so the probability peak is log probability.

[0075] Figure 4 A schematic diagram of a device for generating a word-level timestamp provided in an embodiment of the present application.

[0076] In a second aspect, the present application embodiment provides a device for generating a word-level timestamp, combining Figure 4 ,include:

[0077] The first determination module 401 is used to determine the probability peak value of each word during the frame decoding process.

[0078] Specifically, in the embodiment of the present application, during the frame-by-frame decoding process, the first determination module 401 determines the probability peak of each word, which is to determine the maximum log probability score of each word when it appears as the latest word.

[0079] The second determination module 402 is used to determine the time corresponding to the end point of each word according to the probability peak value of each word.

[0080] Specifically, in the embodiment of the present application, after determining the maximum log probability score of each word, the second determination module 402 determines the time corresponding to the tail endpoint of each word according to the maximum log probability score of each word. The time corresponding to the tail endpoint of each word has been introduced above in combination with the specific steps.

[0081] The third determining module 403 is used to determine the time corresponding to the beginning endpoint of each word according to the time corresponding to the end endpoint of each word.

[0082] Specifically, in the embodiment of the present application, after determining the time corresponding to the tail endpoint of each word, the time corresponding to the tail endpoint of each word can be moved forward by approximately one word time, and the third determination module 403 can determine the time corresponding to the head endpoint of each word, which has been introduced above in conjunction with specific examples.

[0083] The fourth determining module 404 is used to generate a word-level timestamp according to the time corresponding to the first endpoint of each word and the time corresponding to the last endpoint of each word.

[0084] Specifically, in the embodiment of the present application, the time corresponding to the first endpoint of each word and the time corresponding to the last endpoint of each word are determined, and the fourth determination module 404 can determine the timestamp of each word according to the time between the first endpoint and the last endpoint.

[0085] In a third aspect, an embodiment of the present application further provides an electronic device, characterized in that it includes: a processor and a memory;

[0086] The processor is used to execute the above-mentioned method for generating a word-level timestamp by calling the program or instruction stored in the memory.

[0087] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores a program or instruction, and the program or instruction enables the computer to implement the above-mentioned method for generating a word-level timestamp.

[0088] Figure 5 It is a schematic block diagram of an electronic device provided by an embodiment of the present disclosure.

[0089] like Figure 5 As shown, the electronic device includes: at least one processor 501, at least one memory 502 and at least one communication interface 503. The various components in the electronic device are coupled together through a bus system 504. The communication interface 503 is used to transmit information between external devices. It can be understood that the bus system 504 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 504 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 5 Various buses are labeled as bus system 504 .

[0090] It can be understood that the memory 502 in this embodiment can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories.

[0091] In some implementations, the memory 502 stores the following elements, executable units or data structures, or a subset thereof, or an extended set thereof: an operating system and application programs.

[0092] The operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application includes various application programs, such as a media player (Media Player), a browser (Browser), etc., which are used to implement various application services. A program that implements any method of the method for generating a word-level timestamp provided in an embodiment of the present application can be included in the application.

[0093] In an embodiment of the present application, the processor 501 calls a program or instruction stored in the memory 502, specifically, a program or instruction stored in an application, and the processor 501 is used to execute the steps of each embodiment of a method for generating a word-level timestamp provided in an embodiment of the present application.

[0094] During the frame-by-frame decoding process, the probability peak of each word is determined;

[0095] Determine the time corresponding to the end point of each word according to the probability peak of each word;

[0096] Determine the time corresponding to the first endpoint of each word according to the time corresponding to the last endpoint of each word;

[0097] Generate a word-level timestamp based on the time corresponding to the first endpoint of each word and the time corresponding to the last endpoint of each word.

[0098] Any of the methods for generating a word-level timestamp provided in an embodiment of the present application can be applied to the processor 501, or implemented by the processor 501. The processor 501 can be an integrated circuit chip having signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor 501 or an instruction in the form of software. The above-mentioned processor 501 can be a general-purpose processor, a digital signal processor (DigitalSignal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0099] The steps of any method in the method for generating a word-level timestamp provided in an embodiment of the present application can be directly embodied as being executed by a hardware decoding processor, or being executed by a combination of hardware and software units in a decoding processor. The software unit can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 502, and the processor 501 reads the information in the memory 502, and completes the steps of a method for generating a word-level timestamp in combination with its hardware.

[0100] Those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features from different embodiments is meant to be within the scope of the present application and to form different embodiments.

[0101] Those skilled in the art will appreciate that the description of each embodiment has its own emphasis, and for parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0102] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for generating a word-level timestamp, characterized in that: include: In the frame-by-frame decoding process, the probability peak of each word is determined. The probability peak of each word is to determine the maximum log probability score of each word when it appears as the latest word; Determine the time corresponding to the tail endpoint of each word according to the probability peak value of each word; Determine the time corresponding to the beginning endpoint of each word according to the time corresponding to the end endpoint of each word; A word-level timestamp is generated according to the time corresponding to the first endpoint of each word and the time corresponding to the last endpoint of each word.

2. The method for generating a word-level timestamp according to claim 1, characterized in that: The step of determining the time corresponding to the tail endpoint of each word according to the probability peak value of each word includes: Comparing the probability peak value of each word with the current probability value of each word; If the comparison result is that the difference between the probability peak value of each word and the current probability value of each word is greater than or equal to the preset threshold; The time corresponding to the current probability value is determined as the time corresponding to the tail endpoint.

3. The method for generating a word-level timestamp according to claim 1, characterized in that: The step of determining the time corresponding to the tail endpoint of each word according to the probability peak value of each word includes: If after the current word ends, there is a silent segment, and the difference between the probability peak value of each word and the current probability value of each word is less than a preset threshold; The time corresponding to the probability peak of each word is postponed by a first preset time, and the time corresponding to the tail end point of each word is determined.

4. The method for generating a word-level timestamp according to claim 1, characterized in that: Determining the time corresponding to the beginning endpoint of each word according to the end endpoint of each word includes: The time corresponding to the tail endpoint of each word is extended forward by a second preset time to determine the time corresponding to the head endpoint of each word.

5. The method for generating a word-level timestamp according to claim 1, characterized in that: The method further comprises: The time corresponding to the beginning endpoint of each word is determined according to the probability peak value of each word.

6. The method for generating a word-level timestamp according to claim 5, characterized in that: The step of determining the time corresponding to the beginning endpoint of each word according to the probability peak value of each word includes: The time corresponding to the probability peak of each word is extended forward by a first preset time to determine the time corresponding to the beginning endpoint of each word.

7. The method for generating a word-level timestamp according to claim 1, characterized in that: The peak probability is the log probability.

8. A device for generating a word-level timestamp, characterized in that: include: The first determination module is used to determine the probability peak of each word during the frame decoding process. The probability peak of each word is to determine the maximum log probability score of each word when it appears as the latest word. A second determination module is used to determine the time corresponding to the tail end point of each word according to the probability peak value of each word; A third determining module: used to determine the time corresponding to the beginning endpoint of each word according to the time corresponding to the end endpoint of each word; The fourth determining module is used to generate a word-level timestamp according to the time corresponding to the first endpoint of each word and the time corresponding to the last endpoint of each word.

9. An electronic device, characterized in that: include: Processor and memory; The processor is used to execute the method for generating a word-level timestamp as claimed in any one of claims 1 to 7 by calling the program or instruction stored in the memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program or an instruction, wherein the program or the instruction enables a computer to execute the method for generating a word-level time stamp according to any one of claims 1 to 7.

Images