Transcript generation method, system, device, and storage medium

Abstract

The present invention relates to the technical field of computers, and disclosed are a transcript generation method, a system, a device, and a storage medium. The transcript generation method comprises: performing audio recognition on a video file to obtain an audio recognition result, and recognizing subtitles of the video file to obtain a subtitle recognition result; adding segmentation punctuations in the audio recognition result, and acquiring audio character substrings obtained by performing segmentation according to a target segmentation punctuation; for any audio character substring, on the basis of a timestamp of the audio character substring, acquiring from the subtitle recognition result a subtitle character substring present in the same time period as the audio character substring, and correcting the audio character substring on the basis of the subtitle character substring; and using the corrected audio recognition result as a video transcript of the video file for output.

Description

Manuscript generation method, system, device and storage medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on September 13, 2023, with application number 202311183290.3 and invention name “Document Generation Method, System, Device and Storage Medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present disclosure relates to the field of computer technology, and in particular to a document generation method, system, device, and storage medium. Background Art

[0003] In some application scenarios, it's necessary to output the video transcript corresponding to a video file. A video transcript is a text document corresponding to the audio in the video file. Currently, the relevant technology uses automatic speech recognition technology to recognize the audio in video files and generate the video transcript. However, automatic speech recognition technology has its shortcomings, such as frequent errors in recognizing proprietary vocabulary.

[0004] In view of this, there is an urgent need for a method to improve the accuracy of video transcripts.

[0005] Summary of the Invention

[0006] In view of this, the embodiments of the present disclosure provide a manuscript generation method, a manuscript generation system, an electronic device, and a computer-readable storage medium, which can improve the accuracy of video manuscripts.

[0007] In one aspect, the present disclosure provides a method for generating a document, the method comprising:

[0008] Performing audio recognition on the video file to obtain an audio recognition result, and recognizing the subtitles of the video file to obtain a subtitle recognition result;

[0009] Adding segmentation punctuation marks to the audio recognition result, and obtaining audio substrings segmented according to target segmentation punctuation marks;

[0010] For any of the audio substrings, obtaining, in the subtitle recognition result, a subtitle substring that is in the same time period as the audio substring according to the timestamp of the audio substring, and correcting the audio substring according to the subtitle substring;

[0011] The corrected audio recognition result is output as the video document of the video file.

[0012] Another aspect of the present disclosure provides a document generation system, comprising:

[0013] The recognition module is used to perform audio recognition on the video file to obtain an audio recognition result, and to recognize the subtitles of the video file to obtain a subtitle recognition result;

[0014] A segmentation module, configured to add segmentation punctuation marks to the audio recognition result and obtain audio substrings segmented according to target segmentation punctuation marks;

[0015] a correction module configured to, for any of the audio substrings, obtain, in the subtitle recognition result, a subtitle substring that is in the same time period as the audio substring based on a timestamp of the audio substring, and correct the audio substring based on the subtitle substring;

[0016] The document output module is used to output the corrected audio recognition result as a video document of the video file.

[0017] On the other hand, the present disclosure further provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0018] On the other hand, the present disclosure further provides an electronic device, which includes a processor and a memory, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the method described above is implemented.

[0019] In the technical solutions of some embodiments of the present application, segmentation punctuation is added to the audio recognition results of a video file, and an audio substring segmented according to the target segmentation punctuation is obtained. Furthermore, based on the timestamp of the audio substring, a subtitle substring that is in the same time period as the audio substring is obtained from the subtitle recognition results of the video file, and the audio substring is corrected based on the subtitle substring. Because the text in the subtitle substring has a high degree of accuracy, homophones, proper nouns, idioms, personal names, etc. that are incorrectly recognized in the voice substring can be corrected based on the subtitle substring, thereby improving the accuracy of the generated video transcript. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The features and advantages of the present disclosure will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present disclosure in any way. In the accompanying drawings:

[0021] FIG1 is a schematic diagram showing a flow chart of a method for generating a document according to an embodiment of the present application;

[0022] FIG2 is a schematic diagram showing a playback interface of a video file provided by an embodiment of the present application;

[0023] FIG3 shows a schematic diagram of frame extraction scanning provided by an embodiment of the present application;

[0024] FIG4 is a schematic diagram showing a process of obtaining a subtitle substring according to an embodiment of the present application;

[0025] FIG5 shows a module diagram of a document generation system provided by an embodiment of the present application;

[0026] FIG6 shows a schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0027] To make the purpose, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0028] Before explaining the technical solution of this application, the relevant concepts are first explained.

[0029] Automatic Speech Recognition (ASR), also known as speech recognition, computer speech recognition, and speech-to-text, refers to the technology that converts human speech into text through applications.

[0030] Optical Character Recognition (OCR) refers to the technology of identifying and processing image files of text materials to obtain text and layout information, or the technology of identifying text in image files and returning text.

[0031] Based on the above description, the present application provides a manuscript generation method that can improve the accuracy of video manuscripts. The manuscript generation method can be applied to electronic devices. Electronic devices include but are not limited to tablet computers, laptop computers, desktop computers, servers, etc. Please refer to Figure 1, which is a flow chart of the manuscript generation method provided in one embodiment of the present application. In Figure 1, the manuscript generation method includes the following steps:

[0032] Step S11 , performing audio recognition on the video file to obtain an audio recognition result, and performing recognition on the subtitles of the video file to obtain a subtitle recognition result.

[0033] Specifically, the audio recognition result refers to the text obtained by recognizing the audio of a video file based on automatic speech recognition technology. In the audio recognition result, the characters can have an accurate causal relationship and logic, but it is easy to make recognition errors in homophones, proper nouns, idioms, and personal names. For example, taking the audio recognition result "Welcome to my direct-to-learn room" as an example, in this audio recognition result, the characters have an accurate causal relationship and logic. However, the "direct-to-learn room" in the result should be "live broadcast room", that is, the audio recognition result has a proper noun recognition error.

[0034] The subtitle recognition result refers to the text obtained after identifying the subtitles displayed in the video file based on optical character recognition technology. The subtitle recognition result may include a subtitle array identified from the video file at multiple time points. Among them, a subtitle array can be a row or a column of subtitles displayed in the playback interface of the video file. For example, please refer to Figure 2, which is a schematic diagram of the playback interface of a video file provided in an embodiment of the present application. In Figure 2, a row of subtitles can be used as a subtitle array, that is, "Hello everyone", "Thanks for coming", and "My live broadcast room" are each used as a subtitle array.

[0035] When identifying the subtitles of a video file, the playback interface of the video file can be scanned frame by frame to obtain multiple image frames. After identifying the subtitles in the image frame, the multiple subtitle arrays obtained are combined to obtain the subtitle recognition result. Based on optical character recognition technology, the subtitles in the image frame can be accurately identified, thereby preventing recognition errors of homophones, proper nouns, idioms, names, etc. However, there is a large amount of redundant information in the subtitle recognition result and there are no segmentation punctuation marks, resulting in no accurate causal relationship and logic between the characters. Therefore, the subtitle recognition result cannot be directly used as the video manuscript of the video file. For example, suppose that the playback interface of the video file is scanned once every 0.5 seconds (i.e., frame scanning). Refer to Figure 3 for a schematic diagram of frame scanning provided by an embodiment of the present application. Figure 3 exemplarily shows the image frames of a video file at 0.5 seconds, 1 second, 1.5 seconds and 2 seconds. From left to right, the 0.5-second image frame recognizes "Hello everyone"; the 1-second image frame recognizes "Hello everyone," "Thanks for coming," and "My live studio"; and the 1.5-second and 2-second image frames recognize "Thanks for coming," and "My live studio." Combining the recognition results of these four frames yields: "Hello everyone," "Hello everyone," "Thanks for coming," "My live studio," "Thanks for coming," "My live studio," "Thanks for coming," and "My live studio." Obviously, this recognition result cannot be directly used as a video transcript of a video file.

[0036] Through the above analysis of the advantages and disadvantages of audio recognition and subtitle recognition results, it can be understood that correcting the causal relationship and logic of the subtitle recognition results based on the causal relationship and logic of the audio recognition results, or correcting recognition errors such as homophones, proper nouns, idioms, and personal names in the audio recognition results based on the subtitle array in the subtitle recognition results, can improve the accuracy of the video transcript. The following describes this in sequence through steps S12 to S14.

[0037] Step S12: adding segmentation punctuation marks to the audio recognition result, and obtaining audio substrings segmented according to the target segmentation punctuation marks.

[0038] Segmentation marks can be symbols used to annotate video recognition results based on pause duration, intonation, and other factors. These marks include, but are not limited to, semicolons, commas, periods, question marks, and exclamation marks. Audio recognition results can be fed into a trained segmentation model, which then adds segmentation marks to the results.

[0039] In this embodiment, the target segmentation punctuation mark may be a comma, that is, in the audio recognition result, the audio substring is the substring between two adjacent commas.

[0040] Step S13 : for any audio substring, according to the timestamp of the audio substring, obtain the subtitle substring in the same time period as the audio substring in the subtitle recognition result, and correct the audio substring according to the subtitle substring.

[0041] The subtitle substring can be a string extracted from the subtitle recognition result and with redundant information removed. Referring to FIG4 , a flow chart of obtaining a subtitle substring is provided in accordance with an embodiment of the present application. In FIG4 , the process of identifying a subtitle substring includes:

[0042] Step S41 , extracting a target subtitle array that is in the same time period as the audio substring from the subtitle recognition result.

[0043] Specifically, each character in an audio substring can have a timestamp. The timestamp of a character indicates the time at which the character is played in the video file. For example, in the video file "Welcome to my Zhibo Room," if the character "Huan" is played at 1 second and the character "Ying" is played at 1.3 seconds, then the timestamp of the character "Huan" is 1 second, and the timestamp of "Ying" is 1.3 seconds.

[0044] Similarly, in the subtitle recognition results, each subtitle array can also have a timestamp. The timestamp of the subtitle array refers to the time point when the image frame where the subtitle array is located is played in the video file. For example, taking Figure 3 as an example, assuming that in the video file, the first image frame starting from the left is played at the 1st second, and the second image frame starting from the left is played at the 2nd second, then in the subtitle recognition results "Hello everyone", "Hello everyone", "Thank you for coming", and "My Live Room", the timestamp of the first subtitle array "Hello everyone" is 1, and the timestamps of the second subtitle array "Hello everyone", the third subtitle array "Thank you for coming", and the fourth subtitle array "My Live Room" are 2.

[0045] In the audio substring, each character may be sorted and stored in the order in which they are recognized, and in the subtitle recognition result, each subtitle array may be sorted in the order in which they are recognized.

[0046] In combination with the above-mentioned examples of the timestamps of the audio substring and the audio recognition results, it can be understood that since the timestamps of each subtitle array in the audio substring are determined based on the time point when the image frame is played, and the timestamps of each character in the audio substring are determined based on the time point when the character is played, the timestamp accuracy of the subtitle array is lower than the timestamp accuracy of the character. For example, the timestamp of the character can be accurate to milliseconds, but the timestamp of the subtitle array can only be accurate to seconds. In view of this, based on the timestamps of each character in the audio substring, the target subtitle array that is in the same time period as the audio substring can be extracted from the subtitle recognition results, thereby improving the accuracy of the target subtitle array found.

[0047] Specifically, the timestamps of the audio substring can include a minimum timestamp and a maximum timestamp. Based on the order in which the characters in the audio substring are played, the minimum timestamp is the timestamp of the starting character in the audio substring, and the maximum timestamp is the timestamp of the ending character in the audio substring. In the subtitle recognition results, the subtitle array with timestamps between the minimum and maximum timestamps is the target subtitle array.

[0048] In this embodiment, in order to ensure that the audio substring is completely located in the target subtitle array in the same time period, the above-mentioned extraction of the target subtitle array located in the same time period as the audio substring from the subtitle recognition result may include:

[0049] determining a first timestamp that is less than a minimum timestamp and a second timestamp that is greater than a maximum timestamp;

[0050] In the subtitle recognition result, a subtitle array whose timestamp is between the first timestamp and the second timestamp is extracted as a target subtitle array.

[0051] Simply put, the time range of the audio substring is expanded, and the target subtitle array is searched within the expanded time range. For example, assuming the timestamp accuracy of each character in the audio recognition result is milliseconds, the timestamp accuracy of each character in the subtitle recognition result is seconds, and the minimum timestamp of the audio substring is 2.225 seconds and the maximum timestamp is 4.450 seconds, if the target subtitle array is searched according to the minimum and maximum timestamps, the subtitle array with timestamps of 2 seconds, 4 seconds, and between 2 seconds and 4 seconds is searched. After the time range of the audio substring is expanded, the first timestamp less than the minimum timestamp can be determined to be 1.5 seconds, and the second timestamp greater than the maximum timestamp can be determined to be 5.5 seconds. Searching the target subtitle array according to the first and second timestamps means searching for subtitle arrays with timestamps of 1 second, 5 seconds, and between 1 second and 5 seconds. In this way, the audio substring can be ensured to be completely within the target subtitle array, thereby improving the accuracy of subsequent correction.

[0052] Step S42: Combining the target subtitle array into different array character strings, and taking the array character string with the greatest similarity to the audio substring as the subtitle substring.

[0053] Specifically, when combining the target subtitle arrays into array strings, the target subtitle arrays can be accumulated in an orderly manner according to the order in which each target subtitle array is recognized in the video file to obtain array strings of different combinations. The number of array strings obtained is n*(n+1) / 2, where n is the number of target subtitle arrays. For example, in the order in which they are recognized, the target subtitle arrays are: "Thanks", "Come", "My Live Room". The combined array strings may include:

[0054] "grateful"

[0055] “Thanks for coming”

[0056] “Thanks for coming to my live studio”

[0057] "come"

[0058] Come to my live studio

[0059] My Live Room

[0060] Suppose the audio substring is "Thank you for coming to my live broadcast room." The similarity between the audio substring and each of the array strings above can be calculated separately, and the array string with the greatest similarity to the audio substring is selected as the subtitle substring. For example, among the array strings obtained from the above combination, the array string "Thank you for coming to my live broadcast room" has the greatest similarity to the audio substring "Thank you for coming to my live broadcast room." Therefore, the array string "Come to my live broadcast room" is selected as the subtitle substring.

[0061] Specifically, in this embodiment, searching for a subtitle substring with the greatest similarity to an audio substring in an array string may include:

[0062] Calculate the similarity scores between the audio substring and each array string based on the length and character differences between the audio substring and each array string;

[0063] The array string with the highest similarity score to the audio substring is used as the subtitle substring.

[0064] For any array string, the smaller the length difference between the audio substring and the array string, and the smaller the character difference, the higher the similarity score. The array string with the highest similarity score to the audio substring is the subtitle substring with the greatest similarity to the audio substring.

[0065] Furthermore, for any array string, a similarity score between the audio substring and the array string is calculated, including:

[0066] Determining a length difference score based on a length difference between the audio substring and the array string, and determining a character similarity score based on a character difference between the audio substring and the array string;

[0067] The length difference score and the character similarity score are weighted and summed to obtain the similarity score.

[0068] The length difference score represents the length difference between the audio substring and the array string, and the character similarity score represents the character difference between the audio substring and the array string.

[0069] The length difference can be determined based on expression (1):

[0070] Wherein, distance(cand, target) represents the length difference between the audio substring and the array string, cand represents the length of the array string, and lenth(target) represents the length of the audio substring.

[0071] The character similarity score can be calculated based on the SW (Smith-Waterman algorithm). The character similarity score can be determined based on expression (2):

[0072] Where sw_string represents the optimal matching sequence between the audio substring and the array string, and target represents the audio substring.

[0073] The character similarity score and the length difference score can each have corresponding weights. The larger the weight, the greater the influence of the corresponding score on the similarity between the two strings. For example, if the weight of the character similarity score is 0.4 and the weight of the length difference score is 0.6, it means that the length difference score has a greater influence on the similarity between the two strings. The formula for the weighted sum of the length difference score and the character similarity score can be shown as expression (3).

[0074] Among them, score represents the similarity score between the audio substring and the array string, length_ratio represents the weight of the length difference score, and match_ratio represents the weight of the character similarity score.

[0075] Based on the similarity score, the array string with the greatest similarity to the audio substring can be determined and used as the subtitle substring.

[0076] After obtaining the subtitle substring, the audio substring can be corrected based on the subtitle substring.

[0077] Specifically, based on the relevant description of step S11, it can be seen that since the audio substring has an accurate causal relationship and logic, the text in the subtitle substring is relatively accurate. Therefore, based on the text of the subtitle substring, homophones, proper nouns, idioms, names, etc. that are incorrectly recognized in the audio substring can be corrected. While the corrected audio substring has an accurate causal relationship and logic, it reduces the recognition errors of homophones, proper nouns, idioms, and names, thereby improving the accuracy of the video manuscript.

[0078] The following describes in detail how to correct the audio substring based on the subtitle substring.

[0079] In some embodiments, before correcting the audio substring, the following pre-processing operations may be performed on the subtitle substring and the audio substring:

[0080] 1) If the subtitle substring and the audio substring include a separator, delete the separator. Specifically, taking the audio substring as an example, a separator refers to a punctuation mark included in the audio substring that has a smaller segmentation granularity than the separator. For example, if the separator is a comma, the separator can include a semicolon (i.e., a semicolon can be placed between two adjacent commas).

[0081] 2) If the subtitle substring and audio substring contain non-Chinese or English characters, delete the non-Chinese or English characters.

[0082] 3) Convert the numeric characters in the subtitle substring. Specifically, convert the numeric characters into text characters. For example, convert the Arabic numerals 123 into one, two, three.

[0083] After the preprocessing operation is completed, the following correction operation can be performed based on the subtitle substring and audio substring after the preprocessing operation is completed.

[0084] In some embodiments, correcting the audio substring based on the subtitle substring may include:

[0085] If the subtitle substring and the audio substring have the same length, and the first character of the subtitle substring and the first character of the audio substring have the same pronunciation, and the last character of the subtitle substring and the last character of the audio substring have the same pronunciation, the subtitle substring is used to replace the audio substring, where the same pronunciation means that the similarity of the character pronunciation exceeds the pronunciation threshold.

[0086] Specifically, the leading characters refer to the first number of characters found sequentially from the start of the substring toward the end of the substring; the trailing characters refer to the second number of characters found sequentially from the end of the substring toward the start of the substring. For example, taking the audio substring "Thanks for coming to my Zhibo room" as an example, assuming the first number is 2 and the second number is 3, then the leading characters are "Thanks" and the trailing characters are "Zhibo room".

[0087] When the subtitle substring and the audio substring have the same length and the same front-end and back-end characters, it can be considered that the two substrings only differ in characters. Since the characters in the subtitle substring are relatively accurate, the subtitle substring is directly used to replace the audio substring to correct the recognition errors of homophones, proper nouns, idioms, names, etc. in the audio substring. Compared to comparing the characters of the subtitle substring with the characters in the audio substring to find and correct the misrecognized characters in the audio substring, this application directly uses the subtitle substring to replace the audio substring, which can reduce the comparison workload.

[0088] In some embodiments, correcting the audio substring based on the subtitle substring may include:

[0089] If the length of the subtitle substring is greater than the length of the audio substring, and in the subtitle substring, there is a first character that has the same pronunciation as the first character of the audio substring, and there is a second character that has the same pronunciation as the last character of the audio substring, the audio substring is replaced with a string consisting of the first character, the second character, and the characters between the first and second characters.

[0090] Specifically, in these embodiments, the subtitle substring includes a string that only has character differences from the audio substring. In this case, it is only necessary to extract the corresponding string from the subtitle substring and replace the audio substring with the extracted string.

[0091] For example, assume that the subtitle substring is "Hello everyone. Thank you for coming to my live broadcast room" and the audio substring is "Thank you for coming to my live broadcast room". In the subtitle substring, there is a first character with the same pronunciation as the front-end character of the audio substring, such as "Thank you", and there is a second character with the same pronunciation as the back-end character of the audio substring, such as "broadcast room". The string formed by the first character, the second character, and the characters between the first character and the second character is "Thank you for coming to my live broadcast room". Just replace the audio substring "Thank you for coming to my live broadcast room" with the string "Thank you for coming to my live broadcast room" extracted from the subtitle substring.

[0092] In some embodiments, considering that when recognizing the subtitles of a video file based on optical character recognition technology, there may be problems with incorrect recognition of some artistic characters or special fonts in the subtitles. That is, there may also be incorrect characters in the subtitle substring or the string extracted from the subtitle substring, but these incorrect characters may be correct in the audio substring. In this case, if the subtitle substring or the string extracted from the subtitle substring is directly used to replace the audio substring, it may instead modify the audio substring incorrectly. For example, assume:

[0093] The subtitle substring is:以至干仅仅在这里呆上三天

[0094] The voice substring is:以至于仅仅在这里呆上三天

[0095] In the above two substrings, comparing "以至干" in the subtitle substring with "以至于" in the audio substring, obviously, "以至于" in the audio substring is correct. If the subtitle substring is directly used to replace the audio substring, it will instead modify the characters in the audio substring incorrectly.

[0096] To solve the above problems, it can be processed by combining the recognition confidence of each character in the subtitle substring. The recognition confidence represents the confidence of each character when the subtitle in the video file is recognized as each character in the subtitle substring. The recognition confidence can be an attribute自带的属性 of each character in the subtitle substring obtained based on optical character recognition technology. In other words, each character in the subtitle substring itself has a recognition confidence. The higher the recognition confidence of a character, the higher the confidence of that character. Based on the above description, the above use of the subtitle substring or the string extracted from the subtitle substring to replace the audio substring may include:

[0097] In the string used to replace the audio substring, if there are low-confidence characters with a recognition confidence lower than the confidence threshold, according to the position of the low-confidence characters in the string, find the alternative characters corresponding to the low-confidence characters in the audio substring;

[0098] If the shape similarity between the low-confidence character and the alternative character exceeds the shape threshold, and / or the low-confidence character and the alternative character have the same pronunciation, use the alternative character to replace the low-confidence character;

[0099] Use the string with the low-confidence characters replaced to replace the audio substring.

[0100] Among them, the alternative character and the low-confidence character can be characters corresponding in position. In other words, in the string used to replace the audio substring, assuming that the kth character from the starting position is the low-confidence character, then in the audio substring, the kth character from the starting position is the alternative character. Where k is an integer. For example, assume:

[0101] The subtitle substring is:以至干仅仅在这里呆上三天

[0102] The voice substring is:以至于仅仅在这里呆上三天

[0103] In the subtitle substring, assume that the 3rd character "干" from the starting position is the low-confidence character, then in the audio substring, the 3rd character "于" from the starting position is the alternative character corresponding to the low-confidence character.

[0104] After determining the low-confidence character and the alternative character, it can be determined whether to use the alternative character to replace the low-confidence character based on the shape similarity or pronunciation between the low-confidence character and the alternative character. Specifically, the shape similarity between the low-confidence character and the alternative character can be judged from aspects such as the number of strokes, the structure of Chinese characters, the four-corner code, and the radical. If the shape similarity between the low-confidence character and the alternative character is high or the pronunciation is similar, it can be said that the low-confidence character is an incorrectly recognized character. At this time, the alternative character can be used to replace the low-confidence character to correct the characters in the subtitle substring. After completing the character correction, the corrected subtitle substring can be used to replace the audio substring. In this way, the erroneous introduction of the audio substring into the subtitle substring can be avoided, and the reliability of the document generation method can be improved.

[0105] In some embodiments, correcting the audio substring based on the subtitle substring may include:

[0106] If the characters in the subtitle substring do not have the same pronunciation as the leading and trailing characters in the audio substring, searching for different character pairs between the subtitle substring and the audio substring based on the character correspondence between the subtitle substring and the audio substring;

[0107] In the difference character pair, if the characters in the subtitle substring and the characters in the audio substring meet a preset condition, the characters in the subtitle substring are used to replace the characters in the audio substring.

[0108] Specifically, in these embodiments, it is impossible to compare the two subtitle substrings and the audio substring based on their length and pronunciation, that is, the lengths of the two substrings are different, and the pronunciations of the front and back characters are also different. In this case, the edit distance between the subtitle substring and the audio substring can be calculated to find the difference character pairs in the subtitle substring and the audio substring, and then determine whether the characters in each difference character pair meet the preset conditions. The preset conditions may include the shape similarity, pronunciation similarity, etc. of the characters. If the characters in the difference character pair meet the preset conditions, the characters in the subtitle substring are used to replace the characters in the audio substring.

[0109] In other words, in these embodiments, rather than directly replacing the audio substring with the subtitle substring, the subtitle substring and the audio substring are searched for character pairs that differ. If the characters in the character pairs meet a pre-set condition, the subtitle substring is used to replace the audio substring. This allows for correction of the audio substrings, ensuring accuracy of the video transcript.

[0110] Furthermore, in some embodiments, after replacing characters in the audio substring with characters in the subtitle substring, the document generation method of the present application further includes:

[0111] Inputting the first audio substring without character replacement and the second audio substring with character replacement into the trained language model respectively, and having the language model output a first language score for the first audio substring and a second language score for the second audio substring;

[0112] If the second language score is higher than the first language score, the second audio substring is used as the corrected audio substring; if the second language score is not higher than the first language score, the first audio substring is used as the corrected audio substring.

[0113] Specifically, the language score is used to evaluate whether the audio substring conforms to the language norms, such as the fluency of the language. The higher the language score, the higher the probability that the audio substring conforms to the language norms. If the second language score is not higher than the first language score, it means that the replacement of the audio substring is incorrect, that is, the second audio substring that has completed the character replacement does not conform to the language norms as well as the first audio substring that has not undergone the character replacement. In this case, the first audio substring can be used as the corrected audio substring. Conversely, if the second language score is higher than the first language score, it means that the replacement of the audio substring is correct, that is, the second audio substring that has completed the character replacement conforms to the language norms better than the first audio substring that has not undergone the character replacement. In this case, the second audio substring can be used as the corrected audio substring. Based on the language model, the incorrect replacement of the audio substring can be corrected to ensure the accuracy of the correction.

[0114] Furthermore, in some embodiments, when using the second audio substring as the corrected audio substring, the method further includes:

[0115] Record the difference character pairs;

[0116] According to the character correspondence between the subtitle recognition result and the audio recognition result, if a character pair that meets the difference character pair is found, the characters in the subtitle recognition result are used to replace the characters in the audio recognition result.

[0117] Simply put, based on the subtitle and audio substrings, we identify character pairs that may differ between the subtitle and audio recognition results. Then, we perform a full-text replacement of the audio recognition results based on the recorded discrepancies. This eliminates the need to replace the same discrepancies in other audio substring corrections, reducing the correction workload.

[0118] In summary, the correction of the audio substring is completed. The following describes how to add punctuation to the corrected audio substring (as known from the previous preprocessing, the corrected audio substring does not include any punctuation).

[0119] In some embodiments, since the audio substring is a string located between two target segmentation punctuation marks, the target segmentation punctuation marks can be directly added after the corrected audio substring is completed. In addition, the segmentation punctuation marks (such as commas) in the audio substring deleted during preprocessing can be ignored and no longer added. This can avoid the problem of the segmentation punctuation marks being misplaced due to the different number of words in the second audio substring after correction and the first audio substring before correction. For example, suppose:

[0120] The first audio substring before correction is: I like to eat cake, cookies, ice cream

[0121] The corrected second audio substring is: I like to eat cheesecake, cookies, and ice cream

[0122] In the first audio substring, the comma is placed after the sixth and eighth characters. After correction, if commas are added after the sixth and eighth characters of the second audio substring, the second audio substring becomes: "I like cheese, cake, cookies, and ice cream." This is obviously incorrect. Therefore, by ignoring the segmentation punctuation, further correction issues can be avoided.

[0123] Furthermore, in some embodiments, the subtitle substring may contain a relatively rich punctuation mark. For example, a subtitle substring may read "Don't be fooled by this bizarre drinking method~~." Here, "~~" can add semantic information to the string. Given this, the punctuation marks in the subtitle substring can be added to the corrected audio substring to enrich the punctuation mark in the audio substring. However, considering that the punctuation marks in the subtitle substring are removed during the previous preprocessing process, to preserve these punctuation marks, the subtitle substring before the preprocessing operation can be saved as the original array string before performing the preprocessing operation. Furthermore, when using the subtitle substring to replace the audio substring, if the subtitle substring and the audio substring have the same length, the length of the subtitle substring is greater than the length of the corresponding original array string, and the recognition confidence of the subtitle substring's characters is above a confidence threshold, the original array string corresponding to the subtitle substring is used to replace the audio substring. In this way, the punctuation mark richness of the corrected audio sub-segmentation is improved.

[0124] Step S14: output the corrected audio recognition result as a video document of a video file.

[0125] Before outputting the corrected audio recognition result as a video transcript, the corrected audio recognition result can be checked to see if it includes the preset characters. If so, the preset characters are removed. In this way, by setting reasonable preset characters, the content of the video transcript can be modified to further improve the accuracy of the video transcript.

[0126] In summary, the entire description of the manuscript generation method of the present application has been completed. Based on the manuscript generation method of the present application, a video manuscript with high accuracy can be generated. In the technical solutions of some embodiments of the present application, segmentation punctuation is added to the audio recognition result of the video file, and the audio substring segmented according to the target segmentation punctuation is obtained, and based on the timestamp of the audio substring, the subtitle substring in the same time period as the audio substring is obtained in the subtitle recognition result of the video file, and the audio substring is corrected based on the subtitle substring. Since the text in the subtitle substring has a high accuracy, based on the subtitle substring, the homophones, proper nouns, idioms, names, etc. that are incorrectly recognized in the voice substring can be corrected, thereby improving the accuracy of the generated video manuscript.

[0127] Please refer to Figure 5, which is a module diagram of a document generation system provided by an embodiment of the present application. In Figure 5, the document generation system includes:

[0128] The recognition module 51 specifically includes an audio recognition module 511 and a video recognition module 512. The audio recognition module 511 is used to perform audio recognition on the video file to obtain an audio recognition result, and the video recognition module 512 is used to recognize the subtitles of the video file to obtain a subtitle recognition result;

[0129] A segmentation module 52 is configured to add segmentation punctuation marks to the audio recognition result and obtain audio substrings segmented according to the target segmentation punctuation marks;

[0130] A correction module 53 is configured to obtain, for any audio substring, a subtitle substring in the same time period as the audio substring in the subtitle recognition result according to the timestamp of the audio substring, and correct the audio substring based on the subtitle substring;

[0131] The document output module 56 is used to output the corrected audio recognition result as a video document of a video file.

[0132] Furthermore, the document generation system may further include a verification module 54 and a filtering module 55. The verification module 54 is configured to input the first audio substring without character replacement and the second audio substring with character replacement into a trained language model, respectively, and have the language model output a first language score for the first audio substring and a second language score for the second audio substring; if the second language score is higher than the first language score, the second audio substring is used as the corrected audio substring; if the second language score is not higher than the first language score, the first audio substring is used as the corrected audio substring.

[0133] The filtering module 55 is used to check whether the corrected audio recognition result includes the preset characters before the document output module 56 outputs the corrected audio recognition result as a video document. If so, the preset characters are removed.

[0134] Please refer to Figure 6, which is a schematic diagram of an electronic device provided in one embodiment of the present application. The electronic device includes a processor and a memory, wherein the memory is used to store a computer program. When the computer program is executed by the processor, the above method is implemented.

[0135] The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0136] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the methods described in the embodiments of the present invention. The processor executes the non-transitory software programs, instructions, and modules stored in the memory to perform various processor functions and data processing, thereby implementing the methods described in the aforementioned method embodiments.

[0137] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include a memory remotely located relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0138] One embodiment of the present application further provides a computer-readable storage medium, which is used to store a computer program. When the computer program is executed by a processor, the above method is implemented.

[0139] Although the embodiments of the present disclosure have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A manuscript generation method, wherein: The method comprises: Performing audio recognition on the video file to obtain an audio recognition result, and recognizing the subtitles of the video file to obtain a subtitle recognition result; Adding segmentation punctuation marks to the audio recognition result, and obtaining audio substrings segmented according to target segmentation punctuation marks; For any of the audio substrings, according to the timestamp of the audio substring, obtaining a subtitle substring in the same time period as the audio substring in the subtitle recognition result, and correcting the audio substring according to the subtitle substring; The corrected audio recognition result is output as a video document of the video file.

2. The method of claim 1, wherein: The subtitle recognition result includes an array of subtitles recognized from the video file at multiple time points; The obtaining of the subtitle substring in the subtitle recognition result that is in the same time period as the audio substring includes: Extracting a target subtitle array in the same time period as the audio substring from the subtitle recognition result; The target subtitle arrays are combined into different array strings, and the array string having the greatest similarity to the audio substring is used as the subtitle substring.

3. The method of claim 2, wherein: The audio substring and the subtitle array have timestamps, and the timestamp of the audio substring includes a minimum timestamp and a maximum timestamp; The step of extracting a target subtitle array that is in the same time period as the audio substring from the subtitle recognition result includes: Determining a first timestamp that is less than the minimum timestamp, and a second timestamp that is greater than the maximum timestamp; In the subtitle recognition result, the extraction timestamp is between the first timestamp and the second The subtitle array between the timestamps is used as the target subtitle array.

4. The method of claim 1, wherein: The correcting the audio substring according to the subtitle substring includes: If the subtitle substring and the audio substring have the same length, and the front character of the subtitle substring and the front character of the audio substring have the same pronunciation, and the back character of the subtitle substring and the back character of the audio substring have the same pronunciation, the subtitle substring is used to replace the audio substring, wherein the same pronunciation means that the similarity of the character pronunciation exceeds the pronunciation threshold.

5. The method of claim 4, wherein: The correcting the audio substring according to the subtitle substring includes: If the length of the subtitle substring is greater than the length of the audio substring, and in the subtitle substring, there is a first character that has the same pronunciation as the front character of the audio substring, and there is a second character that has the same pronunciation as the back character of the audio substring, the audio substring is replaced with a character string consisting of the first character, the second character, and the characters between the first character and the second character.

6. The method according to claim 4 or 5, wherein: The characters in the subtitle substring have recognition confidence; and the replacing the audio substring comprises: In the character string used to replace the audio substring, if there is a low-confidence character whose recognition confidence is lower than a confidence threshold, searching the audio substring for an alternative character corresponding to the low-confidence character according to the position of the low-confidence character in the character string; If the shape similarity between the low-confidence character and the candidate character exceeds a shape threshold, and / or the low-confidence character and the candidate character have the same pronunciation, the candidate character is used to replace the low-confidence character; The audio substring is replaced with a string in which low-confidence characters have been replaced.

7. The method according to claim 4 or 5, wherein: The correcting the audio substring according to the subtitle substring includes: If the characters in the subtitle substring do not have the same pronunciation as the front and rear characters of the audio substring, searching for a difference character pair between the subtitle substring and the audio substring according to the character correspondence between the subtitle substring and the audio substring; In the difference character pair, if the characters in the subtitle substring and the characters in the audio substring meet a preset condition, the characters in the subtitle substring are used to replace the characters in the audio substring.

8. The method of claim 7, wherein: After replacing the characters in the audio substring with the characters in the subtitle substring, the method further comprises: Inputting a first audio substring without character replacement and a second audio substring with character replacement into a trained language model respectively, and having the language model output a first language score for the first audio substring and a second language score for the second audio substring; If the second language score is higher than the first language score, the second audio substring is used as the corrected audio substring; if the second language score is not higher than the first language score, the first audio substring is used as the corrected audio substring.

9. The method of claim 8, wherein: When using the second audio substring as the corrected audio substring, the method further includes: Recording the difference character pairs; According to the character correspondence between the subtitle recognition result and the audio recognition result, if a character pair that matches the difference character pair is found, the characters in the subtitle recognition result are used to replace the characters in the audio recognition result.

10. The method according to claim 4 or 5, wherein: Before correcting the audio substring, the method further includes: The following preprocessing operations are performed on the subtitle substring and the audio substring: If the subtitle substring and the audio substring include a segmentation mark, delete the segmentation mark; and / or If the subtitle substring and the audio substring include non-Chinese or English characters, delete the non-Chinese or English characters; and / or Convert the numeric characters in the subtitle substring.

11. The method of claim 10, wherein: The characters in the subtitle substring have recognition confidence; before performing the preprocessing operation, the method further includes: Saving the subtitle substring before the preprocessing operation as an original array string; The step of replacing the audio substring with the subtitle substring comprises: If the subtitle substring has the same length as the audio substring, and the length of the subtitle substring is greater than the length of the corresponding original array string, and the recognition confidence of the characters of the subtitle substring is higher than the confidence threshold, the audio substring is replaced with the original array string corresponding to the subtitle substring.

12. The method of claim 1, wherein: After completing the correction of the audio substring, the method further includes: After the corrected audio substring is completed, the target segmentation punctuation is added.

13. The method of claim 1, wherein: Before outputting the corrected audio recognition result as a video manuscript, the method further includes: Check whether the corrected audio recognition result includes preset characters, and if so, remove the preset characters.

14. A document generation system, wherein: The system comprises: The recognition module is used to perform audio recognition on the video file to obtain an audio recognition result, and to recognize the subtitles of the video file to obtain a subtitle recognition result; The segmentation module is used to add segmentation punctuation to the audio recognition result and obtain Split punctuation-delimited audio substrings; a correction module, configured to obtain, for any of the audio substrings, a subtitle substring in the same time period as the audio substring in the subtitle recognition result according to the timestamp of the audio substring, and correct the audio substring according to the subtitle substring; The document output module is used to output the corrected audio recognition result as a video document of the video file.

15. A computer-readable storage medium, wherein: The computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.

16. An electronic device, wherein: The electronic device comprises a processor and a memory, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the method according to any one of claims 1 to 13 is implemented.

Images