Data correction method, system and equipment and storage medium

Abstract

The invention relates to the technical field of computers, and discloses a data correction method, system and device and a storage medium, the data correction method comprises the following steps: obtaining a voice recognition result and a subtitle recognition result of a multimedia file, the subtitle recognition result comprising a subtitle array recognized from the multimedia file at a plurality of time points; extracting a voice sub-character string from the voice recognition result, and extracting a target subtitle array from the subtitle recognition result, wherein the target subtitle array and the voice sub-character string are located in the same time period; combining the target subtitle array into different subtitle sub-character strings, and searching a target subtitle sub-character string having the maximum similarity with the voice sub-character string in the subtitle sub-character strings; and correcting the voice sub-character string according to the target subtitle sub-character string.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a data correction method, system, device and storage medium. Background Art

[0002] Multimedia files refer to files composed of multiple forms of expression such as text, images, sounds, videos, animations, etc., including but not limited to audio files, video files, etc. When converting multimedia files into text, the simplest way is manual conversion, that is, manually listening to the content of the multimedia file and then recording it in a document. This manual conversion method is inefficient. Therefore, automatic conversion technology has emerged. Automatic conversion technology mainly automatically recognizes the voice played by multimedia files to obtain the corresponding text. However, when recognizing the voice played by multimedia files, some proper nouns, idioms, names, etc. are prone to errors.

[0003] Therefore, a method that can improve the conversion accuracy is urgently needed. Summary of the invention

[0004] In view of this, the embodiments of the present disclosure provide a data correction method, a data correction system, an electronic device, and a computer-readable storage medium, which can improve the conversion accuracy when converting multimedia files into text.

[0005] In one aspect, the present disclosure provides a data correction method, the method comprising:

[0006] Acquire a speech recognition result and a subtitle recognition result of a multimedia file, wherein the subtitle recognition result includes an array of subtitles recognized from the multimedia file at multiple time points;

[0007] Extracting a speech substring from the speech recognition result, and extracting a target subtitle array in the same time period as the speech substring from the subtitle recognition result;

[0008] Combining the target subtitle array into different subtitle substrings, and searching the subtitle substrings for a target subtitle substring having the greatest similarity to the voice substring;

[0009] The speech substring is corrected according to the target subtitle substring.

[0010] Another aspect of the present disclosure further provides a data correction system, the system comprising:

[0011] A result acquisition module, used to acquire a speech recognition result and a subtitle recognition result of a multimedia file, wherein the subtitle recognition result includes an array of subtitles recognized from the multimedia file at multiple time points;

[0012] A speech extraction module, used to extract a speech substring from the speech recognition result, and to extract a target subtitle array in the same time period as the speech substring from the subtitle recognition result;

[0013] A subtitle extraction module, used for combining the target subtitle array into different subtitle substrings, and searching the subtitle substrings for a target subtitle substring having the greatest similarity to the voice substring;

[0014] The correction module is used to correct the speech substring according to the target subtitle substring.

[0015] Another aspect of the present disclosure further provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0016] Another aspect of the present disclosure provides an electronic device, which includes a processor and a memory, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the method described above is implemented.

[0017] In the technical solutions of some embodiments of the present application, a speech substring is extracted from the speech recognition result of a multimedia file, and a target subtitle array in the same time period as the speech substring is extracted from the subtitle recognition result of the multimedia file, and then a target subtitle substring with the greatest similarity to the speech substring is searched in the subtitle substring obtained by combining the target subtitle array. Since the text in the target subtitle substring has a high accuracy, based on the target subtitle substring, the homophones, proper nouns, idioms, names, etc. that are incorrectly recognized in the speech substring can be corrected, thereby improving the conversion accuracy when converting multimedia files into text. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The features and advantages of the present disclosure will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present disclosure in any way. In the accompanying drawings:

[0019] Figure 1 A schematic diagram showing a flow chart of a data correction method provided by an embodiment of the present application is shown;

[0020] Figure 2 A schematic diagram of a multimedia file playback interface provided by an embodiment of the present application is shown;

[0021] Figure 3 A schematic diagram of frame extraction scanning provided by an embodiment of the present application is shown;

[0022] Figure 4 A schematic diagram of a module of a data correction system provided by an embodiment of the present application is shown;

[0023] Figure 5 A schematic diagram of an electronic device provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0025] Before describing the technical solution of the present application, the related concepts are first explained.

[0026] Automatic Speech Recognition (ASR), also known as speech recognition, computer speech recognition, and speech-to-text, refers to the technology that converts human speech into text through an application.

[0027] Optical Character Recognition (OCR) refers to the technology of recognizing and processing image files of text materials to obtain text and layout information, or recognizing text in image files and returning text.

[0028] Based on the above description, the present application provides a data correction method, which can improve the conversion accuracy when converting multimedia files into text. The data correction method can be applied to electronic devices. Electronic devices include but are not limited to tablet computers, laptop computers, desktop computers, servers, etc. Figure 1 , which is a flow chart of a data correction method provided in one embodiment of the present application. Figure 1 In the data correction method, the data correction method comprises the following steps:

[0029] Step S11, obtaining speech recognition results and subtitle recognition results of the multimedia file, where the subtitle recognition results include an array of subtitles recognized from the multimedia file at multiple time points.

[0030] Specifically, the speech recognition result refers to the text obtained after recognizing the speech played by the multimedia file based on the automatic speech recognition technology. The speech recognition result can have accurate causal relationships and logic, but it is easy to have recognition errors of homophones, proper nouns, idioms, names, etc. For example, taking a speech recognition result "Hello everyone, welcome to my direct doctoral study room" as an example, in this speech recognition result, there is an accurate causal relationship and logic between the sentences. However, the "direct doctoral study room" in the result should be "live broadcast room", that is, the speech recognition result has a recognition error of the proper noun.

[0031] Subtitle recognition results refer to the text obtained by recognizing subtitles displayed in multimedia files based on optical character recognition technology. A subtitle array in the subtitle recognition result can be a row or a column of subtitles displayed in the playback interface of the multimedia file. For example, see Figure 2 , which is a schematic diagram of a multimedia file playback interface provided by an embodiment of the present application. Figure 2 In the example, a line of subtitles can be used as a subtitle array, that is, "Hello everyone", "Thanks for coming", and "My live broadcast room" are each used as a subtitle array.

[0032] When identifying subtitles of multimedia files, the playback interface of the multimedia files can be scanned frame by frame to obtain multiple image frames. After identifying the subtitles in the image frames, the multiple subtitle arrays obtained are combined to obtain the subtitle recognition results. Based on optical character recognition technology, the subtitles in the image frames can be accurately identified, thereby preventing recognition errors of homophones, proper nouns, idioms, names, etc. However, there is a large amount of redundant information in the subtitle recognition results and there are no segmentation punctuation marks, resulting in a lack of accurate causal relationship and logic between sentences. Therefore, the subtitle recognition results cannot be directly used as the text recognition results of the multimedia files. For example, suppose that the playback interface of the multimedia file is scanned once every 0.5 seconds (i.e., frame scanning). Combined with reference Figure 3 , which is a schematic diagram of frame extraction scanning provided in one embodiment of the present application. Figure 3In the figure, the image frames of a multimedia file at 0.5 seconds, 1 second, 1.5 seconds and 2 seconds are exemplarily shown. From left to right, in the image frame at 0.5 seconds, "Hello everyone" can be recognized; in the image frame at 1 second, "Hello everyone", "Thanks for coming", and "My live studio" can be recognized; in the image frames at 1.5 seconds and 2 seconds, "Thanks for coming" and "My live studio" can be recognized. The recognition results of the four image frames are combined as: "Hello everyone", "Hello everyone", "Thanks for coming", "My live studio", "Thanks for coming", "My live studio", "Thanks for coming", and "My live studio". Obviously, this recognition result cannot be directly used as the text recognition result of a multimedia file.

[0033] Through the above analysis of the advantages and disadvantages of the speech recognition results and the subtitle recognition results, it can be understood that, according to the causal relationship and logic of the speech recognition results, the causal relationship and logic of the subtitle recognition results are corrected, or according to the subtitle array in the subtitle recognition results, the recognition errors of homophones, proper nouns, idioms, names, etc. in the speech recognition results are corrected, which can improve the conversion accuracy when converting multimedia files into text. The following is explained in sequence through steps S12 to S14.

[0034] Step S12, extracting a speech substring from the speech recognition result, and extracting a target subtitle array in the same time period as the speech substring from the subtitle recognition result.

[0035] In this embodiment, the speech recognition result includes segmentation punctuation, and the speech substring is a substring located between two adjacent segmentation punctuation. The segmentation punctuation may be a comma. That is, in the speech recognition result, the substring between two adjacent commas may be used as the speech substring. In some other embodiments, the speech substring may be divided in the speech recognition result according to other rules, for example, starting from the starting position of the speech recognition result, substrings of a preset length are sequentially used as the speech substring.

[0036] For each speech substring, subsequent operations may be performed separately.

[0037] The following describes how to extract a target subtitle array that is in the same time period as a speech substring from the subtitle recognition result based on the speech substring.

[0038] Specifically, in a speech substring, each character can have a timestamp. The timestamp of a character represents the time point when the character is played in a multimedia file. For example, taking "Hello everyone, welcome to my live streaming room" as an example, assume that in the multimedia file, the character "大" (big) is played at the 0.007 second mark, and the character "家" (family) is played at the 0.008 second mark. Then the timestamp of the character "大" is 0.007 seconds, and the timestamp of the character "家" is 0.008 seconds.

[0039] Similarly, in the subtitle recognition result, each subtitle array can also have a timestamp. The timestamp of a subtitle array refers to the time point when the image frame where the subtitle array is located is played in the multimedia file. For example, taking Figure 3 as an example, assume that in the multimedia file, the first image frame starting from the left is played at the 1 second mark, and the second image frame starting from the left is played at the 2 second mark. Then, among the subtitle recognition results "Hello everyone", "Hello everyone", "Thank you for coming to", "My live streaming room", the timestamp of the first subtitle array "Hello everyone" is 1, and the timestamps of the second subtitle array "Hello everyone", the third subtitle array "Thank you for coming to", and the fourth subtitle array "My live streaming room" are 2.

[0040] In the speech substring, the characters can be sorted and stored in the order in which they are recognized, and in the subtitle recognition result, the subtitle arrays can be sorted in the order in which they are recognized.

[0041] Combining the above examples of the subtitle recognition result and the timestamps of the speech substring, it can be understood that since the timestamps of the subtitle arrays in the subtitle recognition result are determined based on the time points when the image frames are played, and the timestamps of the characters in the speech substring are determined based on the time points when the characters are played, therefore, the timestamp accuracy of the subtitle arrays is lower than that of the characters. For example, the timestamp of a character can be accurate to milliseconds, but the timestamp of a subtitle array can only be accurate to seconds. In view of this, based on the timestamps of the characters in the speech substring, the target subtitle arrays in the same time period as the speech substring can be extracted from the subtitle recognition result, so as to improve the accuracy of the found target subtitle arrays.

[0042] Specifically, the timestamp of the speech substring can include a minimum timestamp and a maximum timestamp. According to the order in which the characters in the speech substring are played, the minimum timestamp is the timestamp of the starting character in the speech substring, and the maximum timestamp is the timestamp of the ending character in the speech substring. In the subtitle recognition result, the subtitle arrays whose timestamps are between the minimum timestamp and the maximum timestamp are the target subtitle arrays.

[0043] In this embodiment, in order to make the speech substring completely located in the target subtitle array in the same time period, the above-mentioned extraction of the target subtitle array located in the same time period as the speech substring may include:

[0044] Determine a first timestamp that is less than a minimum timestamp and a second timestamp that is greater than a maximum timestamp;

[0045] In the subtitle recognition result, a subtitle array whose timestamp is between the first timestamp and the second timestamp is extracted as a target subtitle array.

[0046] In simple terms, the time range of the voice substring is expanded, and the target subtitle array is searched within the expanded time range. For example, assuming that the timestamp accuracy of each character in the voice recognition result is milliseconds, the timestamp accuracy of each character in the subtitle recognition result is seconds, and the minimum timestamp of the voice substring is 2.225 seconds, and the maximum timestamp is 4.450 seconds, if the target subtitle array is searched according to the minimum timestamp and the maximum timestamp, the subtitle array with the timestamp located at 2 seconds, 4 seconds, and between 2 seconds and 4 seconds is searched. After the time range of the voice substring is expanded, the first timestamp less than the minimum timestamp can be determined as 1.5 seconds, and the second timestamp greater than the maximum timestamp can be determined as 5.5 seconds. The target subtitle array is searched according to the first timestamp and the second timestamp, that is, the subtitle array with the timestamp located at 1 second, 5 seconds, and between 1 second and 5 seconds is searched. In this way, it can be ensured that the voice substring is completely located in the target subtitle array to improve the subsequent correction accuracy.

[0047] Step S13, combining the target subtitle array into different subtitle substrings, and searching the subtitle substrings for a target subtitle substring having the greatest similarity to the voice substring.

[0048] Specifically, when combining the target subtitle arrays into subtitle substrings, the target subtitle arrays can be accumulated in order according to the order in which the target subtitle arrays are identified in the multimedia file to obtain subtitle substrings of different combinations. The number of subtitle substrings obtained is n*(n+1) / 2, where n is the number of target subtitle arrays. For example, according to the order in which they are identified, the target subtitle arrays are: "Thanks", "Come", "My Live Studio". The combined subtitle substrings may include:

[0049] "grateful"

[0050] “Thanks for coming”

[0051] “Thanks for coming to my live studio”

[0052] "come"

[0053] "Come to my live studio"

[0054] "My Live Room"

[0055] Assume that the voice substring is "Thank you for coming to my live broadcast room". The similarity between the voice substring and each of the above subtitle substrings can be calculated respectively, and then the subtitle substring with the greatest similarity to the voice substring is used as the target subtitle substring. For example, among the subtitle substrings obtained by the above combination, the subtitle substring "Thank you for coming to my live broadcast room" has the greatest similarity with the voice substring "Thank you for coming to my live broadcast room", so the subtitle substring "Come to my live broadcast room" is used as the target subtitle substring.

[0056] Specifically, in this embodiment, searching for a target subtitle substring having the greatest similarity to a speech substring in the subtitle substring may include:

[0057] Based on the length difference and character difference between the voice substring and each subtitle substring, respectively calculate the similarity score between the voice substring and each subtitle substring;

[0058] The subtitle substring having the highest similarity score with the speech substring is taken as the target subtitle substring.

[0059] For any subtitle substring, the smaller the length difference between the voice substring and the subtitle substring, and the smaller the character difference, the higher the similarity score. The subtitle substring with the highest similarity score to the voice substring is the target subtitle substring with the greatest similarity to the voice substring.

[0060] Furthermore, for any subtitle substring, the similarity score between the voice substring and the subtitle substring is calculated, including:

[0061] Determine a length difference score based on a length difference between the speech substring and the subtitle substring, and determine a character similarity score based on a character difference between the speech substring and the subtitle substring;

[0062] The length difference score and the character similarity score are weightedly summed to obtain the similarity score.

[0063] The length difference score represents the length difference between the voice substring and the subtitle substring, and the character similarity score represents the character difference between the voice substring and the subtitle substring.

[0064] The length difference can be determined based on expression (1):

[0065]

[0066] Wherein, distance(cand, target) represents the length difference between the speech substring and the subtitle substring, cand represents the length of the subtitle substring, and lenth(target) represents the length of the speech substring.

[0067] The character similarity score can be calculated based on the SW (Smith-Waterman algorithm). The character similarity score can be determined based on expression (2):

[0068]

[0069] Among them, sw_string represents the optimal matching sequence between the speech substring and the subtitle substring, and target represents the speech substring.

[0070] The character similarity score and the length difference score may have their own corresponding weights. The larger the weight, the greater the influence of the corresponding score on the similarity of the two character strings. For example, the weight of the character similarity score is 0.4, and the weight of the length difference score is 0.6, which means that the length difference score has a greater influence on the similarity of the two character strings. The formula for weighted summation of the length difference score and the character similarity score can be shown as expression (3).

[0071]

[0072] Among them, score represents the similarity score between the voice substring and the subtitle substring, length_ratio represents the weight of the length difference score, and match_ratio represents the weight of the character similarity score.

[0073] In this way, based on the similarity score, the target subtitle substring can be determined.

[0074] Step S14, correcting the speech substring according to the target subtitle substring.

[0075] Specifically, based on the relevant description of step S11, it can be learned that since the voice substring has an accurate causal relationship and logic, the text in the target subtitle substring is relatively accurate. Therefore, based on the text of the target subtitle substring, the homophones, proper nouns, idioms, names, etc. that are misrecognized in the voice substring can be corrected, so that the corrected voice substring has an accurate causal relationship and logic while reducing the recognition errors of homophones, proper nouns, idioms, and names, thereby improving the conversion accuracy when converting multimedia files into text.

[0076] In summary, in the technical solutions of some embodiments of the present application, a speech substring is extracted from the speech recognition results of a multimedia file, and a target subtitle array in the same time period as the speech substring is extracted from the subtitle recognition results of the multimedia file, and then a target subtitle substring with the greatest similarity to the speech substring is searched in the subtitle substring obtained by combining the target subtitle array. Since the text in the target subtitle substring has a high accuracy, based on the target subtitle substring, the homophones, proper nouns, idioms, names, etc. that are incorrectly recognized in the speech substring can be corrected, thereby improving the conversion accuracy when converting the multimedia file into text.

[0077] The following describes in detail how to correct the speech substring based on the target subtitle substring.

[0078] In some embodiments, before correcting the speech substring, the following pre-processing operations may be performed on the target subtitle substring and the speech substring:

[0079] 1) If the target subtitle substring and the speech substring include string punctuation, delete the string punctuation. Specifically, take the speech substring as an example. String punctuation refers to punctuation included in the speech substring with a smaller segmentation granularity than the segmentation punctuation. For example, when the segmentation punctuation is a comma, the string punctuation can include a semicolon (i.e., a semicolon can be located between two adjacent commas).

[0080] 2) If the target subtitle substring and voice substring include non-Chinese or English characters, delete the non-Chinese or English characters.

[0081] 3) Convert the numeric characters in the target subtitle substring. Specifically, convert the numeric characters into text characters. For example, convert the Arabic numerals 123 into one, two, three.

[0082] After the preprocessing operation is completed, the following correction operation may be performed based on the target subtitle substring and speech substring after the preprocessing operation is completed.

[0083] In some embodiments, correcting the speech substring according to the target subtitle substring may include:

[0084] If the target subtitle substring and the voice substring have the same length, and the front character of the target subtitle substring and the front character of the voice substring have the same pronunciation, and the back character of the target subtitle substring and the back character of the voice substring have the same pronunciation, the voice substring is replaced with the target subtitle substring, where the same pronunciation means that the similarity of the character pronunciation exceeds the pronunciation threshold.

[0085] Specifically, the front-end characters refer to the first quantity of characters sequentially found in the direction from the start position of the substring towards the end position of the substring; the back-end characters refer to the second quantity of characters sequentially found in the direction from the end position of the substring towards the start position of the substring. For example, taking the voice substring "Thank you for coming to my live streaming room" as an example, assuming the first quantity is 2 and the second quantity is 3, then the front-end characters are "Thank you", and the back-end characters are "live streaming room".

[0086] When the lengths of the target subtitle substring and the voice substring are the same, and the front-end and back-end characters are both the same, it can be considered that there are only character differences between the two substrings. Also, since the characters in the target subtitle substring are relatively accurate, the target subtitle substring is directly used to replace the voice substring to correct recognition errors such as homophones, proper nouns, idioms, and personal names in the voice substring. Compared with finding and correcting the misrecognized characters in the voice substring by comparing the characters of the target subtitle substring with the characters in the voice substring, the present application directly uses the target subtitle substring to replace the voice substring, which can reduce the comparison workload.

[0087] In some embodiments, correcting the voice substring according to the target subtitle substring may include:

[0088] If the length of the target subtitle substring is greater than the length of the voice substring, and in the target subtitle substring, there is a first character with the same pronunciation as the front-end character of the voice substring, and there is a second character with the same pronunciation as the back-end character of the voice substring, use the string composed of the first character, the second character, and the characters between the first character and the second character to replace the voice substring.

[0089] Specifically, in these embodiments, the target subtitle substring includes a string that only has character differences from the voice substring. In this case, only need to extract the corresponding string from the target subtitle substring and replace the voice substring with the extracted string.

[0090] For example, assume the target subtitle substring is "Hello everyone, thank you for coming to my live streaming room", and the voice substring is "Thank you for coming to my live streaming room". In the target subtitle substring, there is a first character with the same pronunciation as the front-end character of the voice substring, such as "Thank you", and there is a second character with the same pronunciation as the back-end character of the voice substring, such as "live streaming room". The string composed of the first character, the second character, and the characters between the first character and the second character is "Thank you for coming to my live streaming room". Use the string "Thank you for coming to my live streaming room" extracted from the target subtitle substring to replace the voice substring "Thank you for coming to my live streaming room".

[0091] In some embodiments, when considering the recognition of subtitles in a multimedia file based on optical character recognition technology, there may be problems with the recognition of some artistic characters or special fonts in the subtitles. That is, there may also be misrecognized characters in the target subtitle substring or the string extracted from the target subtitle substring, but these misrecognized characters may be correct in the speech substring. In this case, if the target subtitle substring or the string extracted from the target subtitle substring is directly used to replace the speech substring, the speech substring may be modified incorrectly. For example, assume that:

[0092] The target subtitle substring is:以至干仅仅在这里呆上三天

[0093] The speech substring is:以至于仅仅在这里呆上三天

[0094] In the above two substrings, compared with "以至干" in the target subtitle substring and "以至于" in the speech substring, obviously, "以至于" in the speech substring is correct. If the target subtitle substring is directly used to replace the speech substring, the characters in the speech substring will be modified incorrectly.

[0095] To solve the above problems, it can be processed by combining the recognition confidence of each character in the target subtitle substring. Among them, the recognition confidence represents the confidence corresponding to each character when the subtitle in the multimedia file is recognized as each character in the target subtitle substring. The recognition confidence can be an attribute自带 by each character in the target subtitle substring obtained based on optical character recognition technology. In other words, each character in the target subtitle substring itself has a recognition confidence. The higher the recognition confidence of a character, the higher the confidence of the character. Based on the above description, the above use of the target subtitle substring or the string extracted from the target subtitle substring to replace the speech substring can include:

[0096] In the string used to replace the speech substring, if there are low-confidence characters with a recognition confidence lower than the confidence threshold, according to the position of the low-confidence characters in the string, find alternative characters corresponding to the low-confidence characters in the speech substring;

[0097] If the shape similarity between the low-confidence character and the alternative character exceeds the shape threshold, and / or the low-confidence character and the alternative character have the same pronunciation, use the alternative character to replace the low-confidence character;

[0098] Use the string with the low-confidence characters replaced to replace the speech substring.

[0099] Among them, the alternative character and the low-confidence character can be characters corresponding in position. In other words, in the string used to replace the speech substring, assuming that the k-th character starting from the starting position is the low-confidence character, then in the speech substring, the k-th character starting from the starting position is the alternative character. Here, k is an integer. For example, assume that:

[0100] The target subtitle substring is:以至干仅仅在这里呆上三天

[0101] The speech substring is:以至于仅仅在这里呆上三天

[0102] In the target subtitle substring, assuming that the 3rd character '干' starting from the starting position is the low-confidence character, then in the speech substring, the 3rd character '于' starting from the starting position is the alternative character corresponding to the low-confidence character.

[0103] After determining the low-confidence character and the alternative character, it is possible to judge whether to use the alternative character to replace the low-confidence character according to the shape similarity or pronunciation between the low-confidence character and the alternative character. Specifically, the shape similarity between the low-confidence character and the alternative character can be judged from aspects such as the number of strokes, Chinese character structure, four-corner code, and radical. If the shape similarity between the low-confidence character and the alternative character is relatively high or the pronunciation is relatively similar, it can indicate that the low-confidence character is a misrecognized character. At this time, the alternative character can be used to replace the low-confidence character to correct the characters in the target subtitle substring. After completing the character correction, the corrected target subtitle substring can be used to replace the speech substring. In this way, it is possible to avoid the incorrect introduction of the speech substring into the target subtitle substring and improve the reliability of the data correction method.

[0104] In some embodiments, correcting the speech substring according to the target subtitle substring may include:

[0105] If the characters in the target subtitle substring do not have the same pronunciation as the front and back end characters of the speech substring, according to the character correspondence between the target subtitle substring and the speech substring, find the difference character pairs where the target subtitle substring and the speech substring are different;

[0106] In the difference character pairs, if the characters in the target subtitle substring meet the preset conditions with the characters in the speech substring, use the characters in the target subtitle substring to replace the characters in the speech substring.

[0107] Specifically, in these embodiments, it is impossible to compare the two substrings based on the length and pronunciation of the target subtitle substring and the voice substring, that is, the lengths of the two substrings are different, and the pronunciations of the front and back characters are also different. In this case, the edit distance between the target subtitle substring and the voice substring can be calculated to find the difference character pairs in the target subtitle substring and the voice substring, and then determine whether the characters in each difference character pair meet the preset conditions. The preset conditions may include the shape similarity, pronunciation similarity, etc. of the characters. If the characters in the difference character pair meet the preset conditions, the characters in the target subtitle substring are used to replace the characters in the voice substring.

[0108] That is, in these embodiments, instead of directly replacing the speech substring with the target subtitle substring, the difference character pair between the target subtitle substring and the speech substring is searched, and then when the characters in the difference character pair meet the preset conditions, the characters in the target subtitle substring are used to replace the characters in the speech substring. In this way, the speech subcharacters are corrected, so as to improve the conversion accuracy when converting multimedia files into text.

[0109] Furthermore, in some embodiments, after replacing the characters in the speech substring with the characters in the target subtitle substring, the data correction method of the present application further includes:

[0110] Inputting the first voice substring without character replacement and the second voice substring with character replacement into the trained language model respectively, and the language model outputs the first language score of the first voice substring and the second language score of the second voice substring;

[0111] If the second language score is higher than the first language score, the second phonetic substring is used as the corrected phonetic substring; if the second language score is not higher than the first language score, the first phonetic substring is used as the corrected phonetic substring.

[0112] Specifically, the language score is used to evaluate whether the voice substring conforms to the language norms, such as the fluency of the language. The higher the language score, the higher the probability that the voice substring conforms to the language norms. If the second language score is not higher than the first language score, it means that the replacement of the voice substring is wrong, that is, the second voice substring that has completed the character replacement does not conform to the language norms as the first voice substring that has not undergone character replacement. In this case, the first voice substring can be used as the corrected voice substring. On the contrary, if the second language score is higher than the first language score, it means that the replacement of the voice substring is correct, that is, the second voice substring that has completed the character replacement conforms to the language norms better than the first voice substring that has not undergone character replacement. In this case, the second voice substring can be used as the corrected voice substring. Based on the language model, the incorrect replacement of the voice substring can be corrected to ensure the accuracy of the correction.

[0113] Furthermore, in some embodiments, when the second voice substring is used as the corrected voice substring, the data correction method further includes:

[0114] Record the difference character pairs;

[0115] According to the character correspondence between the subtitle recognition result and the speech recognition result, if a character pair that matches the difference character pair is found, the characters in the subtitle recognition result are used to replace the characters in the speech recognition result.

[0116] In simple terms, based on the target subtitle substring and speech substring, after determining the character pairs that may differ between the subtitle recognition results and the speech recognition results, the speech recognition results are fully replaced based on the recorded difference character pairs. In this way, there is no need to replace the same difference character pairs in other speech substring corrections, reducing the correction workload.

[0117] In summary, the correction of the speech substring is completed. The following describes how to add punctuation marks to the corrected speech substring (it can be known from the previous preprocessing that the corrected speech substring does not include any punctuation marks).

[0118] In some embodiments, since the speech substring is a string located between two segmentation punctuation marks, the segmentation punctuation marks can be directly added after the speech substring is corrected. In addition, the string punctuation marks (such as semicolons) in the speech substring deleted during preprocessing can be ignored and no longer added, so as to avoid the problem of misplacement of string punctuation marks due to the different number of words in the second speech substring after correction and the first speech substring before correction. For example, suppose:

[0119] The first voice substring before correction is: I like to eat cake, biscuits, ice cream. The second voice substring after correction is: I like to eat cheesecake, biscuits, ice cream.

[0120] In the first phonetic substring, the comma is located after the sixth and eighth characters. After correction, if a comma is added after the sixth and eighth characters of the second phonetic substring, the second phonetic substring becomes: I like to eat cheese, cake, biscuits and ice cream. Obviously, this is wrong. Therefore, by ignoring string punctuation, more correction problems can be avoided.

[0121] Further, in some embodiments, it is considered that the punctuation in the target subtitle substring may be relatively rich. For example, a target subtitle substring may be expressed as "Don't be confused by this strange way of drinking~~", where "~~" can increase the semantic information of the string. In view of this, the punctuation of the target subtitle substring can be added to the corrected voice substring to make the punctuation of the voice substring richer. However, considering that the punctuation in the target subtitle substring will be deleted in the previous preprocessing process, therefore, in order to retain these punctuation, the target subtitle substring before the preprocessing operation can be saved as the original subtitle substring before performing the preprocessing operation. Furthermore, in the case of using the target subtitle substring to replace the voice substring, if the target subtitle substring has the same length as the voice substring, and the length of the target subtitle substring is greater than the length of the corresponding original subtitle substring, and the recognition confidence of the characters of the target subtitle substring is higher than the confidence threshold, the original subtitle substring corresponding to the target subtitle substring is used to replace the voice substring. In this way, the punctuation richness of the corrected voice substring is improved.

[0122] In summary, the entire description of the data correction method of the present application is completed. Based on the data correction method of the present application, the conversion accuracy can be improved when converting multimedia files into text.

[0123] See also Figure 4 , which is a module diagram of a data correction system provided in one embodiment of the present application. Figure 4 In the data correction system, the data correction system includes:

[0124] A result acquisition module, used to acquire speech recognition results and subtitle recognition results of multimedia files, where the subtitle recognition results include an array of subtitles recognized from multimedia files at multiple time points;

[0125] A speech extraction module is used to extract a speech substring from the speech recognition result, and to extract a target subtitle array in the same time period as the speech substring from the subtitle recognition result;

[0126] A subtitle extraction module, used for combining the target subtitle array into different subtitle substrings, and searching the subtitle substrings for the target subtitle substring having the greatest similarity with the speech substring;

[0127] The correction module is used to correct the speech substring according to the target subtitle substring.

[0128] See also Figure 5 , is a schematic diagram of an electronic device provided by an embodiment of the present application. The electronic device includes a processor and a memory, the memory is used to store a computer program, and when the computer program is executed by the processor, the above method is implemented.

[0129] The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0130] As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the method in the embodiment of the present invention. The processor executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory, that is, implementing the method in the above method embodiment.

[0131] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0132] One embodiment of the present application further provides a computer-readable storage medium, which is used to store a computer program. When the computer program is executed by a processor, the above method is implemented.

[0133] Although the embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A data correction method, characterized in that: The method comprises: Acquire a speech recognition result and a subtitle recognition result of a multimedia file, wherein the subtitle recognition result includes an array of subtitles recognized from the multimedia file at multiple time points; Extracting a speech substring from the speech recognition result, and extracting a target subtitle array in the same time period as the speech substring from the subtitle recognition result; Combining the target subtitle array into different subtitle substrings, and searching the subtitle substrings for a target subtitle substring having the greatest similarity to the voice substring; The speech substring is corrected according to the target subtitle substring.

2. The method according to claim 1, characterized in that The voice substring and the subtitle array have timestamps, and the timestamp of the voice substring includes a minimum timestamp and a maximum timestamp; The step of extracting a target subtitle array that is in the same time period as the voice substring comprises: Determining a first timestamp that is less than the minimum timestamp, and a second timestamp that is greater than the maximum timestamp; In the subtitle recognition result, a subtitle array whose timestamp is between the first timestamp and the second timestamp is extracted as the target subtitle array.

3. The method according to claim 1, characterized in that The correcting the speech substring according to the target subtitle substring comprises: If the target subtitle substring has the same length as the voice substring, and the front character of the target subtitle substring has the same pronunciation as the front character of the voice substring, and the back character of the target subtitle substring has the same pronunciation as the back character of the voice substring, the voice substring is replaced with the target subtitle substring, wherein the same pronunciation means that the similarity of the character pronunciation exceeds the pronunciation threshold.

4. The method according to claim 3, characterized in that The correcting the speech substring according to the target subtitle substring comprises: If the length of the target subtitle substring is greater than the length of the voice substring, and in the target subtitle substring, there is a first character having the same pronunciation as the front character of the voice substring, and there is a second character having the same pronunciation as the back character of the voice substring, the voice substring is replaced with a character string consisting of the first character, the second character, and the characters between the first character and the second character.

5. The method according to claim 3 or 4, characterized in that The characters in the target subtitle substring have recognition confidence; and replacing the speech substring comprises: In the character string used to replace the speech substring, if there is a low-confidence character whose recognition confidence is lower than the confidence threshold, searching the speech substring for an alternative character corresponding to the low-confidence character according to the position of the low-confidence character in the character string; If the shape similarity between the low-confidence character and the candidate character exceeds a shape threshold, and / or the low-confidence character and the candidate character have the same pronunciation, the candidate character is used to replace the low-confidence character; The voice substring is replaced by a string in which low-confidence characters have been replaced.

6. The method according to claim 3 or 4, characterized in that The correcting the speech substring according to the target subtitle substring comprises: If the characters in the target subtitle substring do not have the same pronunciation as the front and rear characters of the voice substring, searching for a difference character pair between the target subtitle substring and the voice substring according to the character correspondence between the target subtitle substring and the voice substring; In the difference character pair, if the characters in the target subtitle substring and the characters in the voice substring meet a preset condition, the characters in the target subtitle substring are used to replace the characters in the voice substring.

7. The method according to claim 6, characterized in that After replacing the characters in the speech substring with the characters in the target subtitle substring, the method further includes: Inputting the first voice substring without character replacement and the second voice substring with character replacement into the trained language model respectively, and the language model outputs the first language score of the first voice substring and the second language score of the second voice substring; If the second language score is higher than the first language score, the second speech substring is used as the corrected speech substring; if the second language score is not higher than the first language score, the first speech substring is used as the corrected speech substring.

8. The method according to claim 7, characterized in that When the second speech substring is used as the corrected speech substring, the method further includes: Recording the difference character pairs; According to the character correspondence between the subtitle recognition result and the speech recognition result, if a character pair that matches the difference character pair is found, the characters in the subtitle recognition result are used to replace the characters in the speech recognition result.

9. The method according to claim 3 or 4, characterized in that Before correcting the speech substring, the method further includes: The following preprocessing operations are performed on the target subtitle substring and the speech substring: If the target subtitle substring and the speech substring include string punctuations, deleting the string punctuations; and / or If the target subtitle substring and the voice substring include non-Chinese or English characters, delete the non-Chinese or English characters; and / or Convert the numeric characters in the target subtitle substring.

10. The method according to claim 9, characterized in that The characters in the target subtitle substring have recognition confidence; before performing the preprocessing operation, the method further includes: Saving the target subtitle substring before the preprocessing operation as an original subtitle substring; The step of replacing the speech substring with the target subtitle substring comprises: If the target subtitle substring is the same length as the speech substring, and the length of the target subtitle substring is greater than the length of the corresponding original subtitle substring, and the recognition confidence of the characters of the target subtitle substring is higher than the confidence threshold, the speech substring is replaced with the original subtitle substring corresponding to the target subtitle substring.

11. The method according to claim 1, characterized in that The speech recognition result includes segmentation punctuation marks, and the speech substring is a substring located between two adjacent segmentation punctuation marks; After completing the correction of the speech substring, the method further includes: After the corrected speech substring is completed, the segmentation punctuation is added.

12. The method according to claim 1, characterized in that The step of searching the subtitle substring for a target subtitle substring having the greatest similarity to the voice substring includes: Based on the length difference and character difference between the voice substring and each of the subtitle substrings, respectively calculating the similarity score between the voice substring and each of the subtitle substrings; The subtitle substring having the highest similarity score with the speech substring is used as the target subtitle substring.

13. The method according to claim 12, characterized in that For any of the subtitle substrings, calculating a similarity score between the voice substring and the subtitle substring includes: Determining a length difference score based on a length difference between the speech substring and the subtitle substring, and determining a character similarity score based on a character difference between the speech substring and the subtitle substring; The length difference score and the character similarity score are weightedly summed to obtain the similarity score.

14. A data correction system, characterized in that: The system comprises: A result acquisition module, used to acquire a speech recognition result and a subtitle recognition result of a multimedia file, wherein the subtitle recognition result includes an array of subtitles recognized from the multimedia file at multiple time points; A speech extraction module, used to extract a speech substring from the speech recognition result, and to extract a target subtitle array in the same time period as the speech substring from the subtitle recognition result; A subtitle extraction module, used for combining the target subtitle array into different subtitle substrings, and searching the subtitle substrings for a target subtitle substring having the greatest similarity to the voice substring; The correction module is used to correct the speech substring according to the target subtitle substring.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.

16. An electronic device, characterized in that: The electronic device comprises a processor and a memory, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the method according to any one of claims 1 to 13 is implemented.

Images