Method, apparatus, electronic device, and storage medium for evaluating timestamp accuracy

Abstract

The present application relates to a method, apparatus, electronic device, and storage medium for evaluating the accuracy of timestamps. The method includes: obtaining an identified text result corresponding to an audio and an annotated text corresponding to the audio; matching the identified text result and the annotated text to determine the number of matching words; determining an accuracy rate based on the number of matching words and the identified text result; determining a recall rate based on the number of matching words and the annotated text; and determining an F1-score based on the accuracy rate and the recall rate, where the F1-score is used to evaluate the accuracy of timestamps. In the present application, the accuracy of timestamps is first evaluated by determining the matching rate between the identified text and the annotated text. The matching rate between the identified text result and the annotated text is regarded as a retrieval task. By borrowing the accuracy rate and recall rate commonly used in retrieval tasks, as well as the harmonic mean F1-score that combines the two, the accuracy of word-level timestamps is evaluated. The word can be a Chinese character or an English word, and finally, the evaluation of word-level timestamps is achieved, improving the user experience.

Description

Technical Field

[0001] This application relates to the technical field of timestamp evaluation, and particularly to a method, apparatus, electronic device, and storage medium for evaluating the accuracy of timestamps. Background Art

[0002] In tasks such as voice activity detection, only the start point and end point of a sentence need to be given, and some errors are allowed (such as 100 ms, which has little impact on the user experience). However, in dictation applications, ASR needs to provide timestamps at a finer granularity (such as word level) to provide a better user experience during playback. For segment-level timestamps, we can use metrics such as frame precision, recall, and accuracy to evaluate them. For word-level timestamps, they are mainly judged by subjective feelings, and there is no objective and useful evaluation metric and method yet. This may lead to relatively large deviations in the timestamps given by the recognition engine, affecting the user experience. Summary of the Invention

[0003] Based on the above problems, this application provides a method, electronic device, and storage medium for evaluating the accuracy of timestamps.

[0004] In a first aspect, an embodiment of this application provides a method for evaluating the accuracy of timestamps, including:

[0005] Obtaining the recognition text result corresponding to the audio and the annotated text corresponding to the audio;

[0006] Matching the recognition text result and the annotated text to determine the number of matching words;

[0007] Determining the accuracy rate according to the number of matching words and the recognition text result;

[0008] Determining the recall rate according to the number of matching words and the annotated text;

[0009] Determining the F1-score according to the accuracy rate and the recall rate, and the F1-score is used to evaluate the accuracy of the timestamp.

[0010] Further, the above method for evaluating the accuracy of timestamps further includes:

[0011] Obtaining the recognition text result corresponding to the audio and the annotated text corresponding to the audio;

[0012] Calculating the matching degree at the word level between the recognition text result and the annotated text;

[0013] Determining the accuracy of the timestamp according to the matching degree at the word level.

[0014] Further, the above method for evaluating the accuracy of timestamps further includes:

[0015] Obtain the recognized text result corresponding to the audio and the annotated text corresponding to the audio;

[0016] Calculate the character-level matching deviation between the recognized text result and the annotated text;

[0017] Determine the accuracy of the timestamp according to the character-level matching deviation.

[0018] Further, the above method for evaluating the accuracy of the timestamp further includes:

[0019] Determine the F1-score according to the precision and recall rate, including:

[0020] Determine the F1-score according to the harmonic mean of the precision and recall rate.

[0021] Further, in the above method for evaluating the accuracy of the timestamp, calculating the character-level matching degree between the recognized text result and the annotated text includes:

[0022] Determine the start time point and end time point of each character in the recognized text result to obtain the first interval;

[0023] Determine the start time point and end time point of each character in the annotated text to obtain the second interval;

[0024] Calculate the overlap rate of the first interval and the second interval through the intersection over union;

[0025] The overlap rate of the first interval and the second interval is the character-level matching degree.

[0026] Further, in the above method for evaluating the accuracy of the timestamp, calculating the character-level matching deviation between the recognized text result and the annotated text includes:

[0027] Determine the timestamp of each character in the recognized text result;

[0028] Determine the timestamp of each character in the annotated text;

[0029] Determine the character-level matching deviation between the recognized text result and the annotated text by calculating the timestamp of each character in the recognized text result and the timestamp of each character in the annotated text.

[0030] Further, in the above method for evaluating the accuracy of the timestamp, determining the character-level matching deviation between the recognized text result and the annotated text by calculating the timestamp of each character in the recognized text result and the timestamp of each character in the annotated text includes:

[0031] Calculate the mean absolute error and the root mean square error.

[0032] In a second aspect, an embodiment of the present application provides a device for evaluating the accuracy of timestamps, including:

[0033] An acquisition module: configured to acquire the recognized text result corresponding to the audio and the annotated text corresponding to the audio;

[0034] A matching module: configured to match the recognized text result and the annotated text to determine the number of matching words;

[0035] A first determination module: configured to determine the accuracy rate according to the number of matching words and the recognized text result;

[0036] A second determination module: configured to determine the recall rate according to the number of matching words and the annotated text;

[0037] A third determination module: configured to determine the F1-score according to the accuracy rate and the recall rate, and the F1-score is used to evaluate the accuracy of the timestamp.

[0038] In a third aspect, an embodiment of the present application further provides an electronic device, which is characterized by including: a processor and a memory;

[0039] The processor is configured to execute the above method for evaluating the accuracy of timestamps by calling the program or instruction stored in the memory.

[0040] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which is characterized in that the computer-readable storage medium stores a program or instruction, and the program or instruction enables the computer to execute the above method for evaluating the accuracy of timestamps.

[0041] The advantages of the embodiments of the present application are as follows: The present application relates to a method, an electronic device, and a storage medium for evaluating the accuracy of timestamps. The method includes: acquiring the recognized text result corresponding to the audio and the annotated text corresponding to the audio; matching the recognized text result and the annotated text to determine the number of matching words; determining the accuracy rate according to the number of matching words and the recognized text result; determining the recall rate according to the number of matching words and the annotated text; determining the F1-score according to the accuracy rate and the recall rate, and the F1-score is used to evaluate the accuracy of the timestamp. In the present application, the accuracy of the timestamp is first evaluated by determining the matching rate between the recognized text and the annotated text. The matching rate between the recognized text result and the annotated text is regarded as a retrieval task, and the accuracy rate and recall rate commonly used in the retrieval task, as well as the harmonic mean F1-score that combines the two, are used to evaluate the accuracy of the word-level timestamp. The word can be a Chinese character or an English word, and finally, the evaluation of the word-level timestamp is realized, improving the user experience. Description of the Drawings

[0042] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0043] Figure 1 Schematic diagram of a method for evaluating the accuracy of timestamps provided by an embodiment of the present application Figure 1 ;

[0044] Figure 2 Schematic diagram of a method for evaluating the accuracy of timestamps provided by an embodiment of the present application Figure 2 ;

[0045] Figure 3 Schematic diagram of a method for evaluating the accuracy of timestamps provided by an embodiment of the present application Figure 3 ;

[0046] Figure 4 Schematic diagram of a device for evaluating the accuracy of timestamps provided by an embodiment of the present application;

[0047] Figure 5 Schematic block diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0048] To make the above objects, features, and advantages of the present application more obvious and understandable, the following will provide a detailed description of the specific implementation manners of the present application in conjunction with the drawings. Many specific details are set forth in the following description to fully understand the present application. However, the present application can be implemented in many other ways different from those described herein. Those skilled in the art can make similar improvements without departing from the connotation of the present application. Therefore, the present application is not limited by the specific implementations disclosed below.

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which this application belongs. The terms used in the description of the present application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0050] Figure 1 Schematic diagram of a method for evaluating the accuracy of timestamps provided by an embodiment of the present application Figure 1 。

[0051] In the first aspect, an embodiment of the present application provides a method for evaluating the accuracy of timestamps, in combination with Figure 1, including five steps from S101 to S105:

[0052] S101: Obtain the recognized text result corresponding to the audio and the annotated text corresponding to the audio.

[0053] Specifically, in the embodiments of the present application, the audio of the recognized text result and the annotated text obtained is the same audio. The recognized text result can be a recognized Chinese text, etc., and the annotated text can be a text with some words manually annotated.

[0054] S102: Match the recognized text result and the annotated text to determine the number of matching words.

[0055] Specifically, in the embodiments of the present application, when matching the recognized text result and the annotated text to determine the number of matching words, it should be understood that the matching situation may be that all the words in the recognized text result and the annotated text are matched, or all the words in the recognized text result and the annotated text are off by 5 ms to be matched, or there may be other situations. Here, they are not listed one by one and do not limit the protection scope of the present application.

[0056] S103: Determine the accuracy rate according to the number of matching words and the recognized text result.

[0057] Specifically, in the embodiments of the present application, the accuracy rate is determined by dividing the number of all matching words in the recognized text result and the annotated text by the number of all words in the recognized text result.

[0058] S104: Determine the recall rate according to the number of matching words and the annotated text.

[0059] Specifically, in the embodiments of the present application, the recall rate is determined by dividing the number of all matching words in the recognized text result and the annotated text by the number of all words in the annotated text.

[0060] S105: Determine the F1-score according to the accuracy rate and the recall rate. The F1-score is used to evaluate the accuracy of the time stamp.

[0061] Specifically, in the embodiments of the present application, determining the F1-score according to the accuracy rate and the recall rate is to determine the F1-score according to the harmonic mean of the accuracy rate and the recall rate. The F1-score is used to evaluate the accuracy of the word-level time stamp.

[0062] The above five steps from S101 to S105 evaluate the accuracy of the word-level time stamp through the number of matching words in the recognized text result and the annotated text. The following combines three steps from S201 to S203 to evaluate the accuracy of the word-level time stamp through the word-level matching degree.

[0063] Figure 2Schematic diagram of a method for evaluating the accuracy of timestamps provided by an embodiment of the present application Figure 2 。

[0064] Further, the above method for evaluating the accuracy of timestamps, in combination with Figure 2 , further includes three steps S201 to S203:

[0065] S201: Obtain the recognized text result corresponding to the audio and the annotated text corresponding to the audio.

[0066] Specifically, in the embodiment of the present application, the audio of the recognized text result and the annotated text obtained is the same audio. The recognized text result can be a recognized Chinese text, etc., and the annotated text can be a text with some words manually annotated.

[0067] S202: Calculate the character-level matching degree between the recognized text result and the annotated text.

[0068] Specifically, in the embodiment of the present application, calculating the character-level matching degree between the recognized text result and the annotated text is to calculate whether the time interval of each character in the recognized text result matches the time interval of each character in the annotated text through the intersection over union.

[0069] S203: Determine the accuracy of the timestamp according to the character-level matching degree.

[0070] Specifically, in the embodiment of the present application, the higher the intersection over union, the higher the overlap degree between the time interval of the recognized text result and the time interval of the annotated text. If the intersection over union is 0, it indicates that the time interval of the recognized text result and the time interval of the annotated text do not match at all.

[0071] The following evaluates the accuracy of the character-level timestamp through the character-level matching deviation in combination with three steps S301 to S303.

[0072] Figure 3 Schematic diagram of a method for evaluating the accuracy of timestamps provided by an embodiment of the present application Figure 3 。

[0073] Further, the above method for evaluating the accuracy of timestamps, in combination with Figure 3 , further includes three steps S301 to S303:

[0074] S301: Obtain the recognized text result corresponding to the audio and the annotated text corresponding to the audio.

[0075] Specifically, in the embodiment of the present application, the audio of the recognized text result and the annotated text obtained is the same audio. The recognized text result can be a recognized Chinese text, etc., and the annotated text can be a text with some words manually annotated.

[0076] S302: Calculate the character-level matching deviation between the recognized text result and the annotated text.

[0077] Specifically, in the embodiments of the present application, the character-level matching deviation between the recognized text result and the annotated text is calculated through two metrics: the mean absolute error and the root mean square error. The mean absolute error is the average of the absolute values of the errors, and the root mean square error is the square root of the average of the squared errors. They have the same dimension and both represent the deviation magnitude between the timestamp of the recognized text result and the timestamp of the annotated text.

[0078] S303: Determine the accuracy of the timestamp based on the character-level matching deviation.

[0079] Specifically, in the embodiments of the present application, the lower the mean absolute error and the root mean square error, the higher the accuracy of the timestamp. If the mean absolute error and the root mean square error are higher, it indicates that the recognized text result does not match the annotated text at all.

[0080] Furthermore, the above method for evaluating the accuracy of the timestamp further includes:

[0081] Determine the F1-score according to the precision and recall rate, including:

[0082] Determine the F1-score according to the harmonic mean of the precision and recall rate.

[0083] Specifically, in the embodiments of the present application, the F1-score is determined according to the harmonic mean of the precision and recall rate as follows:

[0084] F1-score = 2 * precision * recall / (precision + recall).

[0085] Furthermore, in the above method for evaluating the accuracy of the timestamp, calculating the character-level matching degree between the recognized text result and the annotated text includes:

[0086] Determine the start time point and end time point of each character in the recognized text result to obtain the first interval;

[0087] Determine the start time point and end time point of each character in the annotated text to obtain the second interval;

[0088] Calculate the overlap rate of the first interval and the second interval through the intersection-over-union ratio;

[0089] The overlap rate of the first interval and the second interval is the character-level matching degree.

[0090] Specifically, in the embodiments of the present application, the character-level matching degree between the recognized text result and the annotated text is determined by calculating the intersection-over-union ratio of the time interval of each character in the recognized text result and the time interval of each character in the annotated text.

[0091] Further, in the above method for evaluating the accuracy of timestamps, calculating the character-level matching deviation between the recognized text result and the annotated text includes:

[0092] Determine the timestamp of each character in the recognized text result;

[0093] Determine the timestamp of each character in the annotated text;

[0094] Determine the character-level matching deviation between the recognized text result and the annotated text by calculating the timestamp of each character in the recognized text result and the timestamp of each character in the annotated text.

[0095] Specifically, in the embodiments of the present application, calculating the character-level matching deviation between the recognized text result and the annotated text is calculated through two metrics: the mean absolute error and the root mean square error. The mean absolute error is the average of the absolute values of the errors, and the root mean square error is the square root of the average of the squared errors. They have the same dimension and both represent the deviation between the timestamp of the recognized text result and the timestamp of the annotated text.

[0096] Figure 4 It is a schematic diagram of a device for evaluating the accuracy of timestamps provided by the embodiments of the present application.

[0097] In a second aspect, the embodiments of the present application provide a device for evaluating the accuracy of timestamps, in combination with Figure 4 , including:

[0098] An acquisition module 401: configured to acquire the recognized text result corresponding to the audio and the annotated text corresponding to the audio.

[0099] Specifically, in the embodiments of the present application, the audio of the recognized text result and the annotated text acquired by the acquisition module 401 is the same audio segment. The recognized text result may be the recognized Chinese text, etc., and the annotated text may be the text with some characters manually annotated.

[0100] A matching module 402: configured to match the recognized text result and the annotated text to determine the number of matching characters.

[0101] Specifically, in the embodiments of the present application, the matching module 402 matches the recognized text result and the annotated text to determine the number of matching characters. It should be understood that the matching situation may be that all the characters in the recognized text result and the annotated text are matched, or all the characters in the recognized text result and the annotated text are off by 5 ms to be matched, or there may be other situations. Here, they are not listed one by one and do not limit the protection scope of the present application.

[0102] A first determination module 403: configured to determine the accuracy rate according to the number of matching characters and the recognized text result.

[0103] Specifically, in the embodiments of the present application, the first determination module 403 determines the accuracy rate by dividing the number of words that match in the recognized text result and the labeled text by the total number of words in the recognized text result.

[0104] The second determination module 404: is used to determine the recall rate according to the number of matching words and the labeled text.

[0105] Specifically, in the embodiments of the present application, the second determination module 404 determines the recall rate by dividing the number of words that match in the recognized text result and the labeled text by the total number of words in the labeled text.

[0106] The third determination module 405: is used to determine the F1-score according to the accuracy rate and the recall rate, and the F1-score is used to evaluate the accuracy of the timestamp.

[0107] Specifically, in the embodiments of the present application, the third determination module 405 determines the F1-score as the harmonic mean of the accuracy rate and the recall rate, and the F1-score is used to evaluate the accuracy of the character-level timestamp.

[0108] In a third aspect, the embodiments of the present application further provide an electronic device, which is characterized by including: a processor and a memory;

[0109] The processor is used to execute the above method for evaluating the accuracy of the timestamp by calling the program or instruction stored in the memory.

[0110] In a fourth aspect, the embodiments of the present application further provide a computer-readable storage medium, which is characterized in that the computer-readable storage medium stores a program or instruction, and the program or instruction enables the computer to execute the above method for evaluating the accuracy of the timestamp.

[0111] Figure 5 is a schematic block diagram of an electronic device provided by the embodiments of the present disclosure.

[0112] As Figure 5 shown, the electronic device includes: at least one processor 501, at least one memory 502, and at least one communication interface 503. Each component in the electronic device is coupled together through a bus system 504. The communication interface 503 is used for information transmission with external devices. It can be understood that the bus system 504 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 504 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clear illustration, in Figure 5 all the various buses are labeled as the bus system 504.

[0113] It can be understood that the memory 502 in this embodiment may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories.

[0114] In some embodiments, the memory 502 stores the following elements, executable units or data structures, or subsets or supersets thereof: an operating system and application programs.

[0115] Among them, the operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., and is used to implement various basic services and process hardware-based tasks. The application programs include various application programs, such as a Media Player, a Browser, etc., and are used to implement various application services. The program for implementing any of the methods for evaluating the accuracy of a timestamp provided in the embodiments of the present application may be included in the application programs.

[0116] In the embodiments of the present application, the processor 501, by calling the programs or instructions stored in the memory 502, specifically, the programs or instructions stored in the application programs, is used to execute the steps of the various embodiments of the method for evaluating the accuracy of a timestamp provided in the embodiments of the present application.

[0117] Obtain the recognized text result corresponding to the audio and the annotated text corresponding to the audio;

[0118] Match the recognized text result and the annotated text to determine the number of matching words;

[0119] Determine the accuracy rate according to the number of matching words and the recognized text result;

[0120] Determine the recall rate according to the number of matching words and the annotated text;

[0121] Determine the F1-score according to the accuracy rate and the recall rate, and the F1-score is used to evaluate the accuracy of the timestamp.

[0122] Any of the methods for evaluating the accuracy of timestamps provided by the embodiments of the present application can be applied to or implemented by the processor 501. The processor 501 may be an integrated circuit chip with signal processing capabilities. During implementation, the steps of the above methods can be completed by the integrated logic circuit in hardware or instructions in software form in the processor 501. The above-mentioned processor 501 may be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0123] The steps of any of the methods for evaluating the accuracy of timestamps provided by the embodiments of the present application can be directly embodied as being executed and completed by a hardware decoding processor, or by a combination of hardware and software units in the decoding processor. The software units may be located in mature storage media in the art such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers, etc. This storage media is located in the memory 502, and the processor 501 reads the information in the memory 502 and combines its hardware to complete the steps of a method for evaluating the accuracy of timestamps.

[0124] Those skilled in the art can understand that although some of the embodiments described herein include certain features included in other embodiments rather than other features, the combination of the features of different embodiments means that it is within the scope of the present application and forms different embodiments.

[0125] Those skilled in the art can understand that the descriptions of the various embodiments have their own emphases. For parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0126] The above is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of various equivalent modifications or substitutions, and these modifications or substitutions should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A method for evaluating the accuracy of timestamps, characterized in that, Including: Obtain the recognized text result corresponding to the audio and the annotated text corresponding to the audio; Match the recognized text result and the annotated text to determine the number of matching words; Determine the accuracy rate according to the number of matching words and the recognized text result; Determine the recall rate according to the number of matching words and the annotated text; Determine the F1-score according to the accuracy rate and the recall rate, and the F1-score is used to evaluate the accuracy of the timestamp; Also including: Obtain the recognized text result corresponding to the audio and the annotated text corresponding to the audio; Calculate the matching degree at the word level between the recognized text result and the annotated text; Determine the accuracy of the timestamp according to the matching degree at the word level; The said also including: Obtain the recognized text result corresponding to the audio and the annotated text corresponding to the audio; Calculate the matching deviation at the word level between the recognized text result and the annotated text; Determine the accuracy of the timestamp according to the matching deviation at the word level; The said determining the F1-score according to the accuracy rate and the recall rate includes: Determine the F1-score according to the harmonic mean of the accuracy rate and the recall rate; The said calculating the matching degree at the word level between the recognized text result and the annotated text includes: Determine the start time point and end time point of each word in the recognized text result to obtain the first interval; Determine the start time point and end time point of each word in the annotated text to obtain the second interval; Calculate the overlapping rate of the first interval and the second interval through the intersection over union; The overlapping rate of the first interval and the second interval is the matching degree at the word level; The said calculating the matching deviation at the word level between the recognized text result and the annotated text includes: Determine the timestamp of each word in the recognized text result; Determine the timestamp of each word in the annotated text; Determine the matching deviation at the word level between the recognized text result and the annotated text by calculating the timestamp of each word in the recognized text result and the timestamp of each word in the annotated text; Determine the matching deviation at the word level between the recognized text result and the annotated text by calculating the timestamp of each word in the recognized text result and the timestamp of each word in the annotated text includes: Calculate the mean absolute error and the root mean square error.

2. An apparatus for evaluating the accuracy of timestamps, which adopts the method for evaluating the accuracy of timestamps described in claim 1, wherein Including: Obtaining module: used to obtain the recognized text result corresponding to the audio and the annotated text corresponding to the audio; Matching module: used to match the recognized text result and the annotated text to determine the number of matching words; First determination module: used to determine the accuracy rate according to the number of matching words and the recognized text result; Second determination module: used to determine the recall rate according to the number of matching words and the annotated text; Third determination module: used to determine the F1-score according to the accuracy rate and the recall rate, and the F1-score is used to evaluate the accuracy of the timestamp.

3. An electronic device, characterized in that, Including: Processor and memory; The processor is used to execute the method for evaluating the accuracy of the timestamp as described in claim 1 by calling the program or instruction stored in the memory.

4. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores the program or instruction, and the program or instruction causes the computer to execute the method for evaluating the accuracy of the timestamp as described in claim 1.

Images