Data correction method and system, device, and storage medium

Abstract

The present disclosure relates to the technical field of computers. Disclosed are a data correction method and system, a device, and a storage medium. The data correction method comprises: acquiring a speech recognition result and a subtitle recognition result of a multimedia file, the subtitle recognition result comprising subtitle arrays obtained by recognizing the multimedia file at a plurality of time points; extracting from the speech recognition result a speech substring, and extracting from the subtitle recognition result a target subtitle array located in the same time period as the speech substring; combining the target subtitle array into different subtitle substrings, and searching among the subtitle substrings a target subtitle substring having the maximum similarity with the speech substring; and, according to the target subtitle substring, correcting the speech substring.

Description

Data correction method, system, device and storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese invention patent application with application number 202311182602.9, entitled “Data correction method, system, device and storage medium” and filing date September 13, 2023, and the entire application is incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the field of computer technology, and in particular to a data correction method, system, device, and storage medium. Background Art

[0004] Multimedia files refer to files composed of a variety of formats, including text, images, sound, video, and animation, and include but are not limited to audio and video files. The simplest way to convert multimedia files into text is manual conversion, where a person listens to the content of the multimedia file and records it in a document. This manual conversion method is inefficient, leading to the emergence of automatic conversion technology. Automatic conversion technology primarily involves automatically recognizing the speech played in multimedia files to produce the corresponding text. However, when recognizing speech played in multimedia files, errors can easily occur in certain proper nouns, idioms, and personal names.

[0005] Therefore, a method that can improve the conversion accuracy is urgently needed.

[0006] Summary of the Invention

[0007] In view of this, embodiments of the present disclosure provide a data correction method, a data correction system, an electronic device, and a computer-readable storage medium, which can improve the conversion accuracy when converting multimedia files into text.

[0008] In one aspect, the present disclosure provides a data correction method, the method comprising:

[0009] Obtaining a speech recognition result and a subtitle recognition result of a multimedia file, wherein the subtitle recognition result includes an array of subtitles recognized from the multimedia file at multiple time points;

[0010] Extracting a speech substring from the speech recognition result, and extracting a target subtitle array in the same time period as the speech substring from the subtitle recognition result;

[0011] Combining the target subtitle array into different subtitle substrings, and searching the subtitle substrings for a target subtitle substring having the greatest similarity to the voice substring;

[0012] The speech substring is corrected according to the target subtitle substring.

[0013] Another aspect of the present disclosure provides a data correction system, the system comprising:

[0014] A result acquisition module, configured to acquire speech recognition results and subtitle recognition results of a multimedia file, wherein the subtitle recognition results include an array of subtitles recognized from the multimedia file at multiple time points;

[0015] A speech extraction module, configured to extract a speech substring from the speech recognition result, and to extract a target subtitle array in the same time period as the speech substring from the subtitle recognition result;

[0016] a subtitle extraction module, configured to combine the target subtitle array into different subtitle substrings, and search the subtitle substrings for a target subtitle substring having the greatest similarity to the speech substring;

[0017] The correction module is used to correct the speech substring according to the target subtitle substring.

[0018] On the other hand, the present disclosure further provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0019] On the other hand, the present disclosure further provides an electronic device, which includes a processor and a memory, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the method described above is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The features and advantages of the present disclosure will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present disclosure in any way. In the accompanying drawings:

[0021] FIG1 is a schematic diagram showing a flow chart of a data correction method provided by an embodiment of the present application;

[0022] FIG2 is a schematic diagram showing a multimedia file playback interface provided by an embodiment of the present application;

[0023] FIG3 shows a schematic diagram of frame extraction scanning provided by an embodiment of the present application;

[0024] FIG4 shows a module diagram of a data correction system provided by an embodiment of the present application;

[0025] FIG5 shows a schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0026] To make the purpose, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0027] Before explaining the technical solution of this application, the relevant concepts are first explained.

[0028] Automatic Speech Recognition (ASR), also known as speech recognition, computer speech recognition, and speech-to-text, refers to the technology that converts human speech into text through applications.

[0029] Optical Character Recognition (OCR) refers to the technology of identifying and processing image files of text materials to obtain text and layout information, or the technology of identifying text in image files and returning text.

[0030] Based on the above description, the present application provides a data correction method that can improve the conversion accuracy when converting multimedia files into text. The data correction method can be applied to electronic devices. Electronic devices include but are not limited to tablet computers, laptop computers, desktop computers, servers, etc. Please refer to Figure 1, which is a flow chart of the data correction method provided in one embodiment of the present application. In Figure 1, the data correction method includes the following steps:

[0031] Step S11 , obtaining a speech recognition result and a subtitle recognition result of a multimedia file, wherein the subtitle recognition result includes an array of subtitles recognized from the multimedia file at multiple time points.

[0032] Specifically, the speech recognition result refers to the text obtained after recognizing the speech played by multimedia files based on automatic speech recognition technology. The speech recognition result can have accurate causal relationships and logic, but it is prone to recognition errors of homophones, proper nouns, idioms, names, etc. For example, taking the speech recognition result "Hello everyone, welcome to my direct doctoral room" as an example, in this speech recognition result, there is an accurate causal relationship and logic between the sentences. However, the "direct doctoral room" in the result should be "live broadcast room", that is, the speech recognition result has a recognition error of a proper noun.

[0033] The subtitle recognition result refers to the text obtained after identifying the subtitles displayed in the multimedia file based on optical character recognition technology. A subtitle array in the subtitle recognition result can be a row or a column of subtitles displayed in the playback interface of the multimedia file. For example, please refer to Figure 2, which is a schematic diagram of a playback interface of a multimedia file provided in an embodiment of the present application. In Figure 2, a row of subtitles can be used as a subtitle array, that is, "Hello everyone", "Thanks for coming", and "My live broadcast room" are each used as a subtitle array.

[0034] When identifying the subtitles of a multimedia file, the playback interface of the multimedia file can be scanned frame by frame to obtain multiple image frames. After identifying the subtitles in the image frame, the multiple subtitle arrays obtained are combined to obtain the subtitle recognition result. Based on optical character recognition technology, the subtitles in the image frame can be accurately identified, thereby preventing recognition errors of homophones, proper nouns, idioms, names, etc. However, there is a large amount of redundant information in the subtitle recognition result and there are no segmentation punctuation marks, resulting in no accurate causal relationship and logic between sentences. Therefore, the subtitle recognition result cannot be directly used as the text recognition result of the multimedia file. For example, suppose that the playback interface of the multimedia file is scanned once every 0.5 seconds (i.e., frame scanning). Refer to Figure 3 for a schematic diagram of frame scanning provided by an embodiment of the present application. Figure 3 exemplarily shows the image frames of a multimedia file at 0.5 seconds, 1 second, 1.5 seconds and 2 seconds. From left to right, the image frame at 0.5 seconds recognizes "Hello everyone"; the image frame at 1 second recognizes "Hello everyone," "Thanks for coming," and "My live studio"; and the image frames at 1.5 and 2 seconds recognize "Thanks for coming," and "My live studio." Combining the recognition results of these four frames yields "Hello everyone," "Hello everyone," "Thanks for coming," "My live studio," "Thanks for coming," "My live studio," "Thanks for coming," and "My live studio." Obviously, this recognition result cannot be directly used as the text recognition result for a multimedia file.

[0035] Through the above analysis of the advantages and disadvantages of speech recognition and subtitle recognition results, it can be understood that correcting the causal relationship and logic of the subtitle recognition results based on the causal relationship and logic of the speech recognition results, or correcting recognition errors such as homophones, proper nouns, idioms, and personal names in the speech recognition results based on the subtitle array in the subtitle recognition results, can improve the conversion accuracy when converting multimedia files to text. The following describes this in sequence through steps S12 to S14.

[0036] Step S12: Extract voice substrings from the voice recognition result, and extract an array of target subtitles from the subtitle recognition result that is in the same time period as the voice substring.

[0037] In this embodiment, the voice recognition result includes punctuation marks for segmentation, and the voice substring is a substring located between two adjacent punctuation marks for segmentation. Among them, the punctuation mark for segmentation can be a comma. That is, in the voice recognition result, the substring between two adjacent commas can be used as the voice substring. In some other embodiments, the voice substring can also be divided in the voice recognition result according to other rules. For example, starting from the starting position of the voice recognition result, substrings of a preset length are sequentially used as the voice substring.

[0038] For each voice substring, subsequent operations can be performed separately.

[0039] The following describes how to extract an array of target subtitles from the subtitle recognition result that is in the same time period as the voice substring based on the voice substring.

[0040] Specifically, in the voice substring, each character can have a timestamp. The timestamp of a character represents the time point when the character is played in the multimedia file. For example, taking "Hello everyone, welcome to my live stream room" as an example, assuming that in the multimedia file, the character "大" is played at the 0.007th second and the character "家" is played at the 0.008th second, then the timestamp of the character "大" is 0.007 seconds, and the timestamp of the character "家" is 0.008 seconds.

[0041] Similarly, in the subtitle recognition result, each subtitle array can also have a timestamp. The timestamp of the subtitle array refers to the time point when the image frame where the subtitle array is located is played in the multimedia file. For example, taking Figure 3 as an example, assuming that in the multimedia file, the first image frame starting from the left is played at the 1st second and the second image frame starting from the left is played at the 2nd second, then in the subtitle recognition result "大家好", "大家好", "感谢来到", "我的直播间", the timestamp of the first subtitle array "大家好" is 1, and the timestamps of the second subtitle array "大家好", the third subtitle array "感谢来到", and the fourth subtitle array "我的直播间" are 2.

[0042] In the voice substring, the characters can be sorted and stored in the order of recognition, and in the subtitle recognition result, the subtitle arrays can be sorted in the order of recognition.

[0043] Combining the above-mentioned examples of the timestamps of the subtitle recognition results and the speech substring, it can be understood that since the timestamps of each subtitle array in the subtitle recognition results are determined based on the time point when the image frame is played, and the timestamps of each character in the speech substring are determined based on the time point when the character is played, the timestamp accuracy of the subtitle array is lower than the timestamp accuracy of the character. For example, the timestamp of the character can be accurate to milliseconds, but the timestamp of the subtitle array can only be accurate to seconds. In view of this, based on the timestamps of each character in the speech substring, the target subtitle array that is in the same time period as the speech substring can be extracted from the subtitle recognition results, thereby improving the accuracy of the target subtitle array found.

[0044] Specifically, the timestamps of the speech substring can include a minimum timestamp and a maximum timestamp. Based on the order in which the characters in the speech substring are played, the minimum timestamp is the timestamp of the starting character in the speech substring, and the maximum timestamp is the timestamp of the ending character in the speech substring. In the subtitle recognition results, the subtitle array with timestamps between the minimum and maximum timestamps is the target subtitle array.

[0045] In this embodiment, in order to ensure that the speech substring is completely located in the target subtitle array in the same time period, the above-mentioned extraction of the target subtitle array in the same time period as the speech substring may include:

[0046] determining a first timestamp that is less than a minimum timestamp and a second timestamp that is greater than a maximum timestamp;

[0047] In the subtitle recognition result, a subtitle array whose timestamp is between the first timestamp and the second timestamp is extracted as a target subtitle array.

[0048] Simply put, the time range of the speech substring is expanded, and the target subtitle array is searched within the expanded time range. For example, assuming the timestamp accuracy of each character in the speech recognition results is milliseconds, and the timestamp accuracy of each character in the subtitle recognition results is seconds, and the minimum timestamp of the speech substring is 2.225 seconds and the maximum timestamp is 4.450 seconds, searching the target subtitle array based on the minimum and maximum timestamps will result in searching for subtitle arrays with timestamps of 2 seconds, 4 seconds, and between 2 seconds and 4 seconds. After expanding the time range of the speech substring, the first timestamp less than the minimum timestamp can be determined to be 1.5 seconds, and the second timestamp greater than the maximum timestamp can be determined to be 5.5 seconds. Searching the target subtitle array based on the first and second timestamps will result in searching for subtitle arrays with timestamps of 1 second, 5 seconds, and between 1 second and 5 seconds. This ensures that the speech substring is completely within the target subtitle array, thereby improving subsequent correction accuracy.

[0049] Step S13: Combining the target subtitle array into different subtitle substrings, and searching the subtitle substrings for a target subtitle substring having the greatest similarity to the speech substring.

[0050] Specifically, when combining the target subtitle arrays into subtitle substrings, the target subtitle arrays can be sequentially accumulated according to the order in which each target subtitle array is identified in the multimedia file to obtain subtitle substrings of different combinations. The number of subtitle substrings obtained is n*(n+1) / 2, where n is the number of target subtitle arrays. For example, in the order in which they are identified, the target subtitle arrays are: "Thanks", "Come", "My Live Room". The subtitle substrings obtained by the combination may include:

[0051] "grateful"

[0052] “Thanks for coming”

[0053] “Thanks for coming to my live studio”

[0054] "come"

[0055] Come to my live studio

[0056] My Live Room

[0057] Assume the speech substring is "Thank you for coming to my live broadcast room." The similarity between this speech substring and each of the aforementioned subtitle substrings can be calculated separately, and the subtitle substring with the greatest similarity to the speech substring is selected as the target subtitle substring. For example, among the subtitle substrings obtained from the above combination, the subtitle substring "Thank you for coming to my live broadcast room" has the greatest similarity to the speech substring "Thank you for coming to my live broadcast room." Therefore, the subtitle substring "Come to my live broadcast room" is selected as the target subtitle substring.

[0058] Specifically, in this embodiment, searching for a target subtitle substring having the greatest similarity to a speech substring in the subtitle substrings may include:

[0059] Calculate the similarity scores of the speech substring and each subtitle substring based on the length and character differences between the speech substring and each subtitle substring;

[0060] The subtitle substring with the highest similarity score to the speech substring is taken as the target subtitle substring.

[0061] For any subtitle substring, the smaller the length difference between the speech substring and the subtitle substring, and the smaller the character difference, the higher the similarity score. The subtitle substring with the highest similarity score to the speech substring is the target subtitle substring with the greatest similarity to the speech substring.

[0062] Furthermore, for any subtitle substring, a similarity score between the voice substring and the subtitle substring is calculated, including:

[0063] Determining a length difference score based on a length difference between the speech substring and the subtitle substring, and determining a character similarity score based on a character difference between the speech substring and the subtitle substring;

[0064] The length difference score and the character similarity score are weighted and summed to obtain the similarity score.

[0065] The length difference score represents the length difference between the speech substring and the subtitle substring, and the character similarity score represents the character difference between the speech substring and the subtitle substring.

[0066] The length difference can be determined based on expression (1):

[0067] Wherein, distance(cand, target) represents the length difference between the speech substring and the subtitle substring, cand represents the length of the subtitle substring, and lenth(target) represents the length of the speech substring.

[0068] The character similarity score can be calculated based on the SW (Smith-Waterman algorithm). The character similarity score can be determined based on expression (2):

[0069] Among them, sw_string represents the optimal matching sequence between the speech substring and the subtitle substring, and target represents the speech substring.

[0070] The character similarity score and the length difference score can each have corresponding weights. The larger the weight, the greater the influence of the corresponding score on the similarity between the two strings. For example, if the weight of the character similarity score is 0.4 and the weight of the length difference score is 0.6, it means that the length difference score has a greater influence on the similarity between the two strings. The formula for the weighted sum of the length difference score and the character similarity score can be shown as expression (3).

[0071] Among them, score represents the similarity score between the voice substring and the subtitle substring, length_ratio represents the weight of the length difference score, and match_ratio represents the weight of the character similarity score.

[0072] In this way, based on the similarity score, the target subtitle substring can be determined.

[0073] Step S14: Correct the speech substring according to the target subtitle substring.

[0074] Specifically, based on the relevant description of step S11, it can be known that since the voice substring has accurate causal relationships and logic, the text in the target subtitle substring is relatively accurate. Therefore, based on the text of the target subtitle substring, the homophones, proper nouns, idioms, names, etc. that are recognized incorrectly in the voice substring can be corrected. While the corrected voice substring has accurate causal relationships and logic, it reduces the recognition errors of homophones, proper nouns, idioms, and names, thereby improving the conversion accuracy when converting multimedia files into text.

[0075] In summary, in the technical solutions of some embodiments of the present application, a speech substring is extracted from the speech recognition results of a multimedia file, and a target subtitle array that is in the same time period as the speech substring is extracted from the subtitle recognition results of the multimedia file. Then, among the subtitle substrings obtained by combining the target subtitle arrays, a target subtitle substring with the greatest similarity to the speech substring is searched. Because the text in the target subtitle substring has a high degree of accuracy, homophones, proper nouns, idioms, personal names, etc. that were incorrectly recognized in the speech substring can be corrected based on the target subtitle substring, thereby improving the conversion accuracy when converting multimedia files to text.

[0076] The following describes in detail how to correct the speech substring based on the target subtitle substring.

[0077] In some embodiments, before correcting the speech substring, the following pre-processing operations may be performed on the target subtitle substring and the speech substring:

[0078] 1) If the target subtitle substring and speech substring include string punctuation, remove the string punctuation. Specifically, taking the speech substring as an example, string punctuation refers to punctuation included in the speech substring that has a smaller segmentation granularity than the segmentation punctuation. For example, if the segmentation punctuation is a comma, the string punctuation can include a semicolon (i.e., a semicolon can be placed between two adjacent commas).

[0079] 2) If the target subtitle substring and the speech substring include non-Chinese and non-English characters, delete the non-Chinese and non-English characters.

[0080] 3) Convert the numeric characters in the target subtitle substring. Specifically, convert the numeric characters to text characters. For example, convert the Arabic numerals 123 to 一二三.

[0081] After completing the preprocessing operation, based on the target subtitle substring and the speech substring after the preprocessing operation, the following correction operations can be performed.

[0082] In some embodiments, correcting the speech substring according to the target subtitle substring may include:

[0083] If the lengths of the target subtitle substring and the speech substring are the same, and the front-end characters of the target subtitle substring and the front-end characters of the speech substring have the same pronunciation, and the back-end characters of the target subtitle substring and the back-end characters of the speech substring have the same pronunciation, replace the speech substring with the target subtitle substring, where the same pronunciation means that the similarity of the character pronunciation exceeds the pronunciation threshold.

[0084] Specifically, the front-end characters refer to the first number of characters sequentially found in the direction from the start position of the substring to the end position of the substring; the back-end characters refer to the second number of characters sequentially found in the direction from the end position of the substring to the start position of the substring. For example, taking the speech substring "感谢来到我的直博间" as an example, assuming the first number is 2 and the second number is 3, then the front-end characters are 感谢, and the back-end characters are 直博间.

[0085] When the lengths of the target subtitle substring and the speech substring are the same and the front-end and back-end characters are the same, it can be considered that there are only character differences between the two substrings. Also, since the characters in the target subtitle substring are more accurate, directly replace the speech substring with the target subtitle substring to correct the recognition errors such as homophones, proper nouns, idioms, and personal names in the speech substring. Compared with finding and correcting the misrecognized characters in the speech substring by comparing the characters of the target subtitle substring with the characters in the speech substring, directly replacing the speech substring with the target subtitle substring in this application can reduce the comparison workload.

[0086] In some embodiments, correcting the speech substring according to the target subtitle substring may include:

[0087] If the length of the target subtitle substring is greater than the length of the speech substring, and in the target subtitle substring, there is a first character with the same pronunciation as the front-end character of the speech substring, and there is a second character with the same pronunciation as the back-end character of the speech substring, use the string composed of the first character, the second character, and the characters between the first character and the second character to replace the speech substring.

[0088] Specifically, in these embodiments, the target subtitle substring includes a string that only has character differences from the speech substring. In this case, only need to extract the corresponding string from the target subtitle substring and replace the speech substring with the extracted string.

[0089] For example, assume that the target subtitle substring is "Hello everyone. Thank you for coming to my live stream room", and the speech substring is "Thank you for coming to my live stream room". In the target subtitle substring, there is a first character with the same pronunciation as the front-end character of the speech substring, such as "Thank you", and there is a second character with the same pronunciation as the back-end character of the speech substring, such as "stream room". The string composed of the first character, the second character, and the characters between the first character and the second character is "Thank you for coming to my live stream room". Use the string "Thank you for coming to my live stream room" extracted from the target subtitle substring to replace the speech substring "Thank you for coming to my live stream room".

[0090] In some embodiments, considering that when recognizing the subtitles of a multimedia file based on optical character recognition technology, some artistic words or special fonts in the subtitles may have problems with incorrect recognition. That is, the target subtitle substring or the string extracted from the target subtitle substring may also have misrecognized characters, but these misrecognized characters may be correct in the speech substring. In this case, if directly using the target subtitle substring or the string extracted from the target subtitle substring to replace the speech substring, it may instead modify the speech substring incorrectly. For example, assume:

[0091] The target subtitle substring is:以至于仅仅在这里呆上三天

[0092] The speech substring is:以至于仅仅在这里呆上三天

[0093] In the above two substrings, comparing "以至于" in the target subtitle substring with "以至于" in the speech substring, obviously, "以至于" in the speech substring is correct. If directly using the target subtitle substring to replace the speech substring, it will instead modify the characters in the speech substring incorrectly.

[0094] To solve the above problems, it can be processed by combining the recognition confidence of each character in the target subtitle substring. The recognition confidence represents the confidence corresponding to each character when the subtitle in the multimedia file is recognized as each character in the target subtitle substring. The recognition confidence can be an attribute自带的属性 of each character in the target subtitle substring obtained based on optical character recognition technology. In other words, each character in the target subtitle substring itself has a recognition confidence. The higher the recognition confidence of a character, the higher the confidence of the character. Based on the above description, the above use of the target subtitle substring or the string extracted from the target subtitle substring to replace the speech substring may include:

[0095] In the string used to replace the speech substring, if there are low-confidence characters with a recognition confidence lower than the confidence threshold, according to the position of the low-confidence characters in the string, find the alternative characters corresponding to the low-confidence characters in the speech substring;

[0096] If the shape similarity between the low-confidence character and the alternative character exceeds the shape threshold, and / or the low-confidence character and the alternative character have the same pronunciation, use the alternative character to replace the low-confidence character;

[0097] Use the string after replacing the low-confidence characters to replace the speech substring.

[0098] Among them, the alternative character and the low-confidence character can be characters corresponding in position. In other words, in the string used to replace the speech substring, assuming that the kth character from the starting position is the low-confidence character, then in the speech substring, the kth character from the starting position is the alternative character corresponding to the low-confidence character. Here, k is an integer. For example, assume:

[0099] The target subtitle substring is:以至干仅仅在这里呆上三天

[0100] The speech substring is:以至于仅仅在这里呆上三天

[0101] In the target subtitle substring, assume that the 3rd character "干" from the starting position is the low-confidence character, then in the speech substring, the 3rd character "于" from the starting position is the alternative character corresponding to the low-confidence character.

[0102] It should be noted that the phrase "自带的属性" in the original text seems to be an incorrect expression. It might be a miswriting or an unclear term. The translation is done based on the overall context as accurately as possible.After determining the low-confidence character and the alternative character, it can be determined whether to use the alternative character to replace the low-confidence character based on the shape similarity or pronunciation between the low-confidence character and the alternative character. Specifically, the shape similarity between the low-confidence character and the alternative character can be judged from aspects such as the number of strokes, the structure of Chinese characters, the four-corner code, and the radical. If the shape similarity between the low-confidence character and the alternative character is high or the pronunciation is relatively similar, it can be said that the low-confidence character is a character that was recognized incorrectly. At this time, the alternative character can be used to replace the low-confidence character to correct the characters in the target subtitle substring. After completing the character correction, the corrected target subtitle substring can be used to replace the voice substring. In this way, the incorrect introduction of the voice substring into the target subtitle substring can be avoided, and the reliability of the data correction method can be improved.

[0103] In some embodiments, correcting the speech substring according to the target subtitle substring may include:

[0104] If the characters in the target subtitle substring do not have the same pronunciation as the leading and trailing characters in the speech substring, searching for different character pairs between the target subtitle substring and the speech substring based on the character correspondence between the target subtitle substring and the speech substring;

[0105] In the difference character pair, if the characters in the target subtitle substring and the characters in the speech substring meet the preset conditions, the characters in the speech substring are replaced with the characters in the target subtitle substring.

[0106] Specifically, in these embodiments, it is impossible to compare the two substrings based on the length and pronunciation of the target subtitle substring and the voice substring, that is, the lengths of the two substrings are different, and the pronunciations of the front and back characters are also different. In this case, the edit distance between the target subtitle substring and the voice substring can be calculated to find the difference character pairs in the target subtitle substring and the voice substring, and then determine whether the characters in each difference character pair meet the preset conditions. The preset conditions may include the shape similarity, pronunciation similarity, etc. of the characters. If the characters in the difference character pair meet the preset conditions, the characters in the target subtitle substring are used to replace the characters in the voice substring.

[0107] In other words, in these embodiments, rather than directly replacing the speech substring with the target subtitle substring, a difference character pair is found between the target subtitle substring and the speech substring. Then, if the characters in the difference character pair meet a predetermined condition, the characters in the speech substring are replaced with the characters in the target subtitle substring. This corrects the speech substring, thereby improving the conversion accuracy when converting multimedia files to text.

[0108] Furthermore, in some embodiments, after replacing characters in the speech substring with characters in the target subtitle substring, the data correction method of the present application further includes:

[0109] Inputting the first phonetic substring without character replacement and the second phonetic substring with character replacement into the trained language model respectively, and the language model outputting a first language score for the first phonetic substring and a second language score for the second phonetic substring;

[0110] If the second language score is higher than the first language score, the second phonetic substring is used as the corrected phonetic substring; if the second language score is not higher than the first language score, the first phonetic substring is used as the corrected phonetic substring.

[0111] Specifically, the language score is used to evaluate whether the speech substring conforms to the language norms, such as the fluency of the language. The higher the language score, the higher the probability that the speech substring conforms to the language norms. If the second language score is not higher than the first language score, it means that the replacement of the speech substring is incorrect, that is, the second speech substring that has completed the character replacement does not conform to the language norms as well as the first speech substring that has not undergone the character replacement. In this case, the first speech substring can be used as the corrected speech substring. Conversely, if the second language score is higher than the first language score, it means that the replacement of the speech substring is correct, that is, the second speech substring that has completed the character replacement conforms to the language norms better than the first speech substring that has not undergone the character replacement. In this case, the second speech substring can be used as the corrected speech substring. Based on the language model, the incorrect replacement of the speech substring can be corrected to ensure the accuracy of the correction.

[0112] Furthermore, in some embodiments, when the second phonetic substring is used as the corrected phonetic substring, the data correction method further includes:

[0113] Record the difference character pairs;

[0114] According to the character correspondence between the subtitle recognition result and the speech recognition result, if a character pair that meets the difference character pair is found, the characters in the subtitle recognition result are used to replace the characters in the speech recognition result.

[0115] Simply put, based on the target subtitle and speech substrings, we identify character pairs that may differ between the subtitle and speech recognition results. Then, we perform a full-text replacement of the speech recognition results based on the recorded discrepancies. This eliminates the need to replace the same discrepancies in other speech substring corrections, reducing the correction workload.

[0116] In summary, the correction of the speech substring is completed. The following describes how to add punctuation to the corrected speech substring (as known from the previous preprocessing, the corrected speech substring does not include any punctuation).

[0117] In some embodiments, since the speech substring is a string located between two segmentation punctuation marks, the segmentation punctuation marks can be directly added after the speech substring is corrected. In addition, the string punctuation marks (such as semicolons) in the speech substring deleted during preprocessing can be ignored and no longer added. This can avoid the problem of misplacement of string punctuation marks due to the different number of words in the second speech substring after correction and the first speech substring before correction. For example, suppose:

[0118] The first speech substring before correction is: I like to eat cake, biscuits, ice cream. The second speech substring after correction is: I like to eat cheesecake, biscuits, ice cream.

[0119] In the first phonetic substring, the commas are placed after the sixth and eighth characters. After correction, if commas are added after the sixth and eighth characters of the second phonetic substring, the second phonetic substring becomes: "I like cheese, cake, cookies, and ice cream." This is obviously incorrect. Therefore, by ignoring string punctuation, we can avoid introducing further correction issues.

[0120] Furthermore, in some embodiments, the target subtitle substring may contain a relatively rich punctuation mark. For example, a target subtitle substring may be expressed as "Don't be fooled by this novel way of drinking..." Here, "~~" can add semantic information to the string. Given this, the punctuation marks from the target subtitle substring can be added to the corrected speech substring to enrich the punctuation mark of the speech substring. However, considering that the punctuation marks in the target subtitle substring are deleted during the previous preprocessing process, to preserve these punctuation marks, the target subtitle substring before the preprocessing operation can be saved as the original subtitle substring before performing the preprocessing operation. Furthermore, when using the target subtitle substring to replace the speech substring, if the target subtitle substring and the speech substring have the same length, the length of the target subtitle substring is greater than the length of the corresponding original subtitle substring, and the recognition confidence of the characters in the target subtitle substring is greater than a confidence threshold, the speech substring is replaced with the original subtitle substring corresponding to the target subtitle substring. In this way, the punctuation mark richness of the corrected speech substring is improved.

[0121] In summary, the entire description of the data correction method of the present application is completed. Based on the data correction method of the present application, the conversion accuracy can be improved when converting multimedia files into text.

[0122] Please refer to Figure 4, which is a module diagram of a data correction system provided by an embodiment of the present application. In Figure 4, the data correction system includes:

[0123] A result acquisition module is used to obtain the speech recognition results and subtitle recognition results of the multimedia file, where the subtitle recognition results include an array of subtitles recognized from the multimedia file at multiple time points;

[0124] A speech extraction module is used to extract a speech substring from the speech recognition result, and to extract a target subtitle array that is in the same time period as the speech substring from the subtitle recognition result;

[0125] A subtitle extraction module is used to combine the target subtitle array into different subtitle substrings and search the subtitle substrings for the target subtitle substring with the greatest similarity to the speech substring;

[0126] The correction module is used to correct the speech substring according to the target subtitle substring.

[0127] Please refer to Figure 5, which is a schematic diagram of an electronic device provided in one embodiment of the present application. The electronic device includes a processor and a memory, wherein the memory is used to store a computer program. When the computer program is executed by the processor, the above method is implemented.

[0128] The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0129] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the methods described in the embodiments of the present invention. The processor executes the non-transitory software programs, instructions, and modules stored in the memory to perform various processor functions and data processing, thereby implementing the methods described in the aforementioned method embodiments.

[0130] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include a memory remotely located relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0131] One embodiment of the present application further provides a computer-readable storage medium, which is used to store a computer program. When the computer program is executed by a processor, the above method is implemented.

[0132] Although the embodiments of the present disclosure have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A data correction method, comprising: Acquire a speech recognition result and a subtitle recognition result of a multimedia file, wherein the subtitle recognition result includes an array of subtitles recognized from the multimedia file at multiple time points; Extracting a speech substring from the speech recognition result, and extracting a target subtitle array in the same time period as the speech substring from the subtitle recognition result; Combining the target subtitle array into different subtitle substrings, and searching the subtitle substrings for a target subtitle substring having the greatest similarity to the voice substring; The speech substring is corrected according to the target subtitle substring.

2. The method of claim 1, wherein: The voice substring and the subtitle array have timestamps, and the timestamp of the voice substring includes a minimum timestamp and a maximum timestamp; The step of extracting a target subtitle array that is in the same time period as the voice substring comprises: Determining a first timestamp that is less than the minimum timestamp, and a second timestamp that is greater than the maximum timestamp; In the subtitle recognition result, a subtitle array whose timestamp is between the first timestamp and the second timestamp is extracted as the target subtitle array.

3. The method of claim 1, wherein: The correcting the speech substring according to the target subtitle substring comprises: If the target subtitle substring has the same length as the voice substring, and the front character of the target subtitle substring has the same pronunciation as the front character of the voice substring, and the back character of the target subtitle substring has the same pronunciation as the back character of the voice substring, the voice substring is replaced with the target subtitle substring, wherein the same pronunciation means that the similarity of the character pronunciation exceeds the pronunciation threshold.

4. The method of claim 3, wherein: The correcting the speech substring according to the target subtitle substring comprises: If the length of the target subtitle substring is greater than the length of the voice substring, and in the target subtitle substring, there is a first character having the same pronunciation as the front character of the voice substring, and there is a second character having the same pronunciation as the back character of the voice substring, the voice substring is replaced with a character string consisting of the first character, the second character, and the characters between the first character and the second character.

5. The method according to claim 3 or 4, wherein: The characters in the target subtitle substring have recognition confidence; and replacing the speech substring comprises: In the character string used to replace the speech substring, if there is a low-confidence character whose recognition confidence is lower than the confidence threshold, searching the speech substring for an alternative character corresponding to the low-confidence character according to the position of the low-confidence character in the character string; If the shape similarity between the low-confidence character and the candidate character exceeds a shape threshold, and / or the low-confidence character and the candidate character have the same pronunciation, the candidate character is used to replace the low-confidence character; The voice substring is replaced by a string in which low-confidence characters have been replaced.

6. The method according to claim 3 or 4, wherein: The correcting the speech substring according to the target subtitle substring comprises: If the characters in the target subtitle substring do not have the same pronunciation as the front and rear characters of the voice substring, searching for a difference character pair between the target subtitle substring and the voice substring according to the character correspondence between the target subtitle substring and the voice substring; In the difference character pair, if the characters in the target subtitle substring and the characters in the voice substring meet a preset condition, the characters in the target subtitle substring are used to replace the characters in the voice substring.

7. The method of claim 6, wherein: After replacing the characters in the speech substring with the characters in the target subtitle substring, the method further includes: Inputting the first voice substring without character replacement and the second voice substring with character replacement into the trained language model respectively, and the language model outputs the first language score of the first voice substring and the second language score of the second voice substring; If the second language score is higher than the first language score, the second speech substring is used as the corrected speech substring; if the second language score is not higher than the first language score, the first speech substring is used as the corrected speech substring.

8. The method of claim 7, wherein: When the second speech substring is used as the corrected speech substring, the method further includes: Recording the difference character pairs; According to the character correspondence between the subtitle recognition result and the speech recognition result, if a character pair that matches the difference character pair is found, the characters in the subtitle recognition result are used to replace the characters in the speech recognition result.

9. The method according to claim 3 or 4, wherein: Before correcting the speech substring, the method further includes: The following preprocessing operations are performed on the target subtitle substring and the speech substring: If the target subtitle substring and the speech substring include string punctuations, deleting the string punctuations; and / or If the target subtitle substring and the voice substring include non-Chinese or English characters, delete the non-Chinese or English characters; and / or Convert the numeric characters in the target subtitle substring.

10. The method of claim 9, wherein: The characters in the target subtitle substring have recognition confidence; before performing the preprocessing operation, the method further includes: Saving the target subtitle substring before the preprocessing operation as an original subtitle substring; The step of replacing the speech substring with the target subtitle substring comprises: If the target subtitle substring is the same length as the speech substring, and the length of the target subtitle substring is greater than the length of the corresponding original subtitle substring, and the recognition confidence of the characters of the target subtitle substring is higher than the confidence threshold, the speech substring is replaced with the original subtitle substring corresponding to the target subtitle substring.

11. The method of claim 1, wherein: The speech recognition result includes segmentation punctuation marks, and the speech substring is a substring located between two adjacent segmentation punctuation marks; After completing the correction of the speech substring, the method further includes: After the corrected speech substring is completed, the segmentation punctuation is added.

12. The method of claim 1, wherein: The step of searching the subtitle substring for a target subtitle substring having the greatest similarity to the voice substring includes: Based on the length difference and character difference between the voice substring and each of the subtitle substrings, respectively calculating the similarity score between the voice substring and each of the subtitle substrings; The subtitle substring having the highest similarity score with the speech substring is used as the target subtitle substring.

13. The method of claim 12, wherein: For any of the subtitle substrings, calculating a similarity score between the voice substring and the subtitle substring includes: Determining a length difference score based on a length difference between the speech substring and the subtitle substring, and determining a character similarity score based on a character difference between the speech substring and the subtitle substring; The length difference score and the character similarity score are weightedly summed to obtain the similarity score.

14. A data correction system, comprising: The result acquisition module is used to acquire the speech recognition result and subtitle recognition result of the multimedia file, wherein the subtitle recognition result includes the subtitle data recognized from the multimedia file at multiple time points. Group; A speech extraction module, used to extract a speech substring from the speech recognition result, and to extract a target subtitle array in the same time period as the speech substring from the subtitle recognition result; A subtitle extraction module, used for combining the target subtitle array into different subtitle substrings, and searching the subtitle substrings for a target subtitle substring having the greatest similarity to the voice substring; The correction module is used to correct the speech substring according to the target subtitle substring.

15. A computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.

16. An electronic device, comprising a processor and a memory, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the method according to any one of claims 1 to 13 is implemented.

Images