Manuscript generation method, system and equipment and storage medium

Abstract

The invention relates to the technical field of computers, and discloses a manuscript generation method, system and device and a storage medium, and the manuscript generation method comprises the steps: carrying out the audio recognition of a video file to obtain an audio recognition result, and carrying out the recognition of subtitles of the video file to obtain a subtitle recognition result; adding a segmentation punctuation in the audio recognition result, and obtaining audio sub-strings segmented according to a target segmentation punctuation; for any audio sub-character string, according to the timestamp of the audio sub-character string, acquiring a subtitle sub-character string located in the same time period as the audio sub-character string from the subtitle recognition result, and correcting the audio sub-character string according to the subtitle sub-character string; and outputting the corrected audio recognition result as a video manuscript of the video file.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a manuscript generation method, system, device and storage medium. Background Art

[0002] In some application scenarios, it is necessary to output the video document corresponding to the video file. The video document is a text document corresponding to the audio of the video file. At present, the relevant technology uses automatic speech recognition technology to recognize the audio of the video file to obtain the video document. However, automatic speech recognition technology has its shortcomings, such as frequent recognition errors of some proprietary words.

[0003] In view of this, a method for improving the accuracy of video manuscripts is urgently needed. Summary of the invention

[0004] In view of this, the embodiments of the present disclosure provide a manuscript generation method, a manuscript generation system, an electronic device and a computer-readable storage medium, which can improve the accuracy of video manuscripts.

[0005] On the one hand, the present disclosure provides a manuscript generation method, the method comprising:

[0006] Performing audio recognition on the video file to obtain an audio recognition result, and recognizing the subtitles of the video file to obtain a subtitle recognition result;

[0007] Adding segmentation punctuation marks to the audio recognition result, and obtaining audio substrings segmented according to target segmentation punctuation marks;

[0008] For any of the audio substrings, according to the timestamp of the audio substring, obtaining a subtitle substring in the same time period as the audio substring in the subtitle recognition result, and correcting the audio substring according to the subtitle substring;

[0009] The corrected audio recognition result is output as a video document of the video file.

[0010] Another aspect of the present disclosure further provides a document generation system, the system comprising:

[0011] The recognition module is used to perform audio recognition on the video file to obtain an audio recognition result, and to recognize the subtitles of the video file to obtain a subtitle recognition result;

[0012] A segmentation module, used to add segmentation punctuation marks to the audio recognition result and obtain audio substrings segmented according to target segmentation punctuation marks;

[0013] a correction module, configured to obtain, for any of the audio substrings, a subtitle substring in the same time period as the audio substring in the subtitle recognition result according to the timestamp of the audio substring, and correct the audio substring according to the subtitle substring;

[0014] The document output module is used to output the corrected audio recognition result as a video document of the video file.

[0015] Another aspect of the present disclosure further provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0016] Another aspect of the present disclosure provides an electronic device, which includes a processor and a memory, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the method described above is implemented.

[0017] In the technical solutions of some embodiments of the present application, segmentation punctuation is added to the audio recognition result of the video file, and the audio substring segmented according to the target segmentation punctuation is obtained, and the subtitle substring in the same time period as the audio substring is obtained in the subtitle recognition result of the video file according to the timestamp of the audio substring, and the audio substring is corrected according to the subtitle substring. Since the text in the subtitle substring has a high accuracy, based on the subtitle substring, the homophones, proper nouns, idioms, names, etc. that are recognized incorrectly in the voice substring can be corrected, thereby improving the accuracy of the generated video manuscript. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The features and advantages of the present disclosure will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present disclosure in any way. In the accompanying drawings:

[0019] Figure 1 A schematic diagram showing a flow chart of a manuscript generation method provided by an embodiment of the present application;

[0020] Figure 2 A schematic diagram of a video file playback interface provided by an embodiment of the present application is shown;

[0021] Figure 3 A schematic diagram of frame extraction scanning provided by an embodiment of the present application is shown;

[0022] Figure 4 A schematic diagram of a process for obtaining a subtitle substring provided by an embodiment of the present application is shown;

[0023] Figure 5A module schematic diagram of a document generation system provided by an embodiment of the present application is shown;

[0024] Figure 6 A schematic diagram of an electronic device provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0026] Before describing the technical solution of the present application, the related concepts are first explained.

[0027] Automatic Speech Recognition (ASR), also known as speech recognition, computer speech recognition, and speech-to-text, refers to the technology that converts human speech into text through an application.

[0028] Optical Character Recognition (OCR) refers to the technology of recognizing and processing image files of text materials to obtain text and layout information, or recognizing text in image files and returning text.

[0029] Based on the above description, the present application provides a manuscript generation method that can improve the accuracy of video manuscripts. The manuscript generation method can be applied to electronic devices. The electronic devices include but are not limited to tablet computers, laptop computers, desktop computers, servers, etc. Figure 1 , which is a flow chart of a document generation method provided in one embodiment of the present application. Figure 1 In the method for generating a manuscript, the method comprises the following steps:

[0030] Step S11, performing audio recognition on the video file to obtain an audio recognition result, and performing recognition on the subtitles of the video file to obtain a subtitle recognition result.

[0031] Specifically, the audio recognition result refers to the text obtained by recognizing the audio of a video file based on automatic speech recognition technology. In the audio recognition result, the characters can have accurate causal relationships and logic, but it is easy to make recognition errors of homophones, proper nouns, idioms, names, etc. For example, taking an audio recognition result "Welcome to my direct doctoral study room" as an example, in this audio recognition result, the characters have accurate causal relationships and logic. However, the "direct doctoral study room" in the result should be "live broadcast room", that is, the audio recognition result has a recognition error of a proper noun.

[0032] Subtitle recognition results refer to text obtained by recognizing subtitles displayed in a video file based on optical character recognition technology. Subtitle recognition results may include subtitle arrays recognized from video files at multiple time points. A subtitle array may be a row or column of subtitles displayed in the playback interface of a video file. For example, see Figure 2 , which is a schematic diagram of a video file playback interface provided by an embodiment of the present application. Figure 2 In the example, a line of subtitles can be used as a subtitle array, that is, "Hello everyone", "Thanks for coming", and "My live broadcast room" are each used as a subtitle array.

[0033] When recognizing the subtitles of a video file, the playback interface of the video file can be scanned frame by frame to obtain multiple image frames. After recognizing the subtitles in the image frame, the multiple subtitle arrays obtained are combined to obtain the subtitle recognition result. Based on optical character recognition technology, the subtitles in the image frame can be accurately recognized, thereby preventing recognition errors of homophones, proper nouns, idioms, names, etc. However, there is a large amount of redundant information in the subtitle recognition result and there are no segmentation punctuation marks, resulting in no accurate causal relationship and logic between the characters. Therefore, the subtitle recognition result cannot be directly used as the video manuscript of the video file. For example, suppose that the playback interface of the video file is scanned once every 0.5 seconds (i.e., frame scanning). In conjunction with reference Figure 3 , which is a schematic diagram of frame extraction scanning provided in one embodiment of the present application. Figure 3In the figure, the image frames of a video file at 0.5 seconds, 1 second, 1.5 seconds and 2 seconds are shown as an example. From left to right, in the image frame at 0.5 seconds, "Hello everyone" can be recognized; in the image frame at 1 second, "Hello everyone", "Thanks for coming", and "My live studio" can be recognized; in the image frames at 1.5 seconds and 2 seconds, "Thanks for coming" and "My live studio" can be recognized. The recognition results of the four image frames are combined as: "Hello everyone", "Hello everyone", "Thanks for coming", "My live studio", "Thanks for coming", "My live studio", "Thanks for coming", and "My live studio". Obviously, this recognition result cannot be directly used as the video manuscript of the video file.

[0034] Through the above analysis of the advantages and disadvantages of the audio recognition results and the subtitle recognition results, it can be understood that the accuracy of the video manuscript can be improved by correcting the causal relationship and logic of the subtitle recognition results based on the causal relationship and logic of the audio recognition results, or correcting the recognition errors of homophones, proper nouns, idioms, names, etc. in the audio recognition results based on the subtitle array in the subtitle recognition results. The following is explained in sequence through steps S12 to S14.

[0035] Step S12, adding segmentation punctuation marks to the audio recognition result, and obtaining audio substrings segmented according to the target segmentation punctuation marks.

[0036] The segmentation punctuation may be a symbol used to mark the video recognition result according to the length of pause, intonation change, etc. The segmentation punctuation includes but is not limited to a semicolon, a comma, a period, a question mark, an exclamation mark, etc. The audio recognition result may be input into a trained segmentation model, and the segmentation model may add segmentation punctuation to the audio recognition result.

[0037] In this embodiment, the target segmentation punctuation may be a comma, that is, in the audio recognition result, the audio substring is the substring between two adjacent commas.

[0038] Step S13, for any audio substring, according to the timestamp of the audio substring, obtain the subtitle substring in the same time period as the audio substring in the subtitle recognition result, and correct the audio substring according to the subtitle substring.

[0039] The subtitle substring may be a string extracted from the subtitle recognition result and with redundant information removed. Figure 4 , is a flowchart of obtaining a subtitle substring provided by an embodiment of the present application. Figure 4 In , the process of identifying subtitle substrings includes:

[0040] Step S41, extracting a target subtitle array that is in the same time period as the audio substring from the subtitle recognition result.

[0041] Specifically, in the audio substring, each character may have a timestamp. The timestamp of the character indicates the time point when the character is played in the video file. For example, taking "Welcome to my Zhibo room" as an example, assuming that in the video file, the character "Huan" is played at the 1st second, and the character "Ying" is played at the 1.3th second, then the timestamp of the character "Huan" is 1 second, and the timestamp of "Ying" is 1.3 seconds.

[0042] Similarly, in the subtitle recognition result, each subtitle array can also have a timestamp. The timestamp of the subtitle array refers to the time point when the image frame where the subtitle array is located is played in the video file. Figure 3 For example, assuming that in the video file, the first image frame starting from the left is played at the 1st second, and the second image frame starting from the left is played at the 2nd second, then in the subtitle recognition results "Hello everyone", "Hello everyone", "Thanks for coming", and "My Live Studio", the timestamp of the first subtitle array "Hello everyone" is 1, and the timestamps of the second subtitle array "Hello everyone", the third subtitle array "Thanks for coming" and the fourth subtitle array "My Live Studio" are 2.

[0043] In the audio substring, each character may be sorted and stored in the order in which they are recognized, and in the subtitle recognition result, each subtitle array may be sorted in the order in which they are recognized.

[0044] In combination with the above-mentioned examples of timestamps of audio substrings and audio recognition results, it can be understood that since the timestamps of each subtitle array in the audio substring are determined based on the time point when the image frame is played, and the timestamps of each character in the audio substring are determined based on the time point when the character is played, the timestamp accuracy of the subtitle array is lower than the timestamp accuracy of the character. For example, the timestamp of the character can be accurate to milliseconds, but the timestamp of the subtitle array can only be accurate to seconds. In view of this, based on the timestamps of each character in the audio substring, the target subtitle array in the same time period as the audio substring can be extracted from the subtitle recognition results, so as to improve the accuracy of the target subtitle array found.

[0045] Specifically, the timestamp of the audio substring may include a minimum timestamp and a maximum timestamp. According to the order in which the characters in the audio substring are played, the minimum timestamp is the timestamp of the starting character in the audio substring, and the maximum timestamp is the timestamp of the ending character in the audio substring. In the subtitle recognition result, the subtitle array whose timestamp is between the minimum timestamp and the maximum timestamp is the target subtitle array.

[0046] In this embodiment, in order to make the audio substring completely located in the target subtitle array in the same time period, the above-mentioned extraction of the target subtitle array located in the same time period as the audio substring from the subtitle recognition result may include:

[0047] Determine a first timestamp that is less than a minimum timestamp and a second timestamp that is greater than a maximum timestamp;

[0048] In the subtitle recognition result, a subtitle array whose timestamp is between the first timestamp and the second timestamp is extracted as a target subtitle array.

[0049] In simple terms, the time range of the audio substring is extended, and the target subtitle array is searched within the extended time range. For example, assuming that the timestamp accuracy of each character in the audio recognition result is milliseconds, the timestamp accuracy of each character in the subtitle recognition result is seconds, and the minimum timestamp of the audio substring is 2.225 seconds, and the maximum timestamp is 4.450 seconds, if the target subtitle array is searched according to the minimum timestamp and the maximum timestamp, the subtitle array with the timestamp located at 2 seconds, 4 seconds, and between 2 seconds and 4 seconds is searched. After the time range of the audio substring is extended, the first timestamp less than the minimum timestamp can be determined as 1.5 seconds, and the second timestamp greater than the maximum timestamp can be determined as 5.5 seconds. The target subtitle array is searched according to the first timestamp and the second timestamp, that is, the subtitle array with the timestamp located at 1 second, 5 seconds, and between 1 second and 5 seconds is searched. In this way, it can be ensured that the audio substring is completely located in the target subtitle array to improve the subsequent correction accuracy.

[0050] Step S42: Combining the target subtitle array into different array character strings, and taking the array character string with the greatest similarity to the audio substring as the subtitle substring.

[0051] Specifically, when combining the target subtitle arrays into array strings, the target subtitle arrays can be accumulated in order according to the order in which the target subtitle arrays are identified in the video file to obtain array strings of different combinations. The number of array strings obtained is n*(n+1) / 2, where n is the number of target subtitle arrays. For example, according to the order in which they are identified, the target subtitle arrays are: "Thanks", "Come", "My Live Studio". The combined array strings may include:

[0052] "grateful"

[0053] “Thanks for coming”

[0054] “Thanks for coming to my live studio”

[0055] "come"

[0056] "Come to my live studio"

[0057] "My Live Room"

[0058] Assume that the audio substring is "Thank you for coming to my live broadcast room". The similarity between the audio substring and each of the above array strings can be calculated respectively, and then the array string with the greatest similarity to the audio substring is used as the subtitle substring. For example, in the array string obtained by the above combination, the array string "Thank you for coming to my live broadcast room" has the greatest similarity with the audio substring "Thank you for coming to my live broadcast room", so the array string "Come to my live broadcast room" is used as the subtitle substring.

[0059] Specifically, in this embodiment, searching the array string for a subtitle substring having the greatest similarity to the audio substring may include:

[0060] Based on the length difference and character difference between the audio substring and each array string, respectively calculate the similarity score between the audio substring and each array string;

[0061] The array string with the highest similarity score to the audio substring is taken as the subtitle substring.

[0062] For any array string, the smaller the length difference between the audio substring and the array string, and the smaller the character difference, the higher the similarity score. The array string with the highest similarity score to the audio substring is the subtitle substring with the greatest similarity to the audio substring.

[0063] Furthermore, for any array string, a similarity score between the audio substring and the array string is calculated, including:

[0064] Determine a length difference score based on a length difference between the audio substring and the array string, and determine a character similarity score based on a character difference between the audio substring and the array string;

[0065] The length difference score and the character similarity score are weightedly summed to obtain the similarity score.

[0066] The length difference score represents the length difference between the audio substring and the array string, and the character similarity score represents the character difference between the audio substring and the array string.

[0067] The length difference can be determined based on expression (1):

[0068]

[0069] Among them, distance(cand, target) represents the length difference between the audio substring and the array string, cand represents the length of the array string, and lenth(target) represents the length of the audio substring.

[0070] The character similarity score can be calculated based on the SW (Smith-Waterman algorithm). The character similarity score can be determined based on expression (2):

[0071]

[0072] Among them, sw_string represents the optimal matching sequence between the audio substring and the array string, and target represents the audio substring.

[0073] The character similarity score and the length difference score may have their own corresponding weights. The larger the weight, the greater the influence of the corresponding score on the similarity of the two character strings. For example, the weight of the character similarity score is 0.4, and the weight of the length difference score is 0.6, which means that the length difference score has a greater influence on the similarity of the two character strings. The formula for weighted summation of the length difference score and the character similarity score can be shown as expression (3).

[0074]

[0075] Among them, score represents the similarity score between the audio substring and the array string, length_ratio represents the weight of the length difference score, and match_ratio represents the weight of the character similarity score.

[0076] Based on the similarity score, an array string having the greatest similarity to the audio substring can be determined and used as the subtitle substring.

[0077] After obtaining the subtitle substring, the audio substring can be corrected according to the subtitle substring.

[0078] Specifically, based on the relevant description of step S11, it can be learned that since the audio substring has an accurate causal relationship and logic, the text in the subtitle substring is relatively accurate. Therefore, based on the text of the subtitle substring, the misrecognized homophones, proper nouns, idioms, names, etc. in the audio substring can be corrected, so that the corrected audio substring has an accurate causal relationship and logic while reducing the recognition errors of homophones, proper nouns, idioms, and names, thereby improving the accuracy of the video manuscript.

[0079] The following describes in detail how to correct the audio substring based on the subtitle substring.

[0080] In some embodiments, before correcting the audio substring, the following pre-processing operations may be performed on the subtitle substring and the audio substring:

[0081] 1) If the subtitle substring and the audio substring include a segmentation punctuation, delete the segmentation punctuation. Specifically, take the audio substring as an example. The segmentation punctuation refers to a punctuation included in the audio substring that has a smaller segmentation granularity than the segmentation punctuation. For example, when the segmentation punctuation is a comma, the segmentation punctuation may include a semicolon (i.e., a semicolon may be located between two adjacent commas).

[0082] 2) If the subtitle substring and the audio substring include non-Chinese or English characters, delete the non-Chinese or English characters.

[0083] 3) Convert the numeric characters in the subtitle substring. Specifically, convert the numeric characters into text characters. For example, convert the Arabic numerals 123 into one, two, three.

[0084] After the preprocessing operation is completed, the following correction operation may be performed based on the subtitle substring and the audio substring after the preprocessing operation is completed.

[0085] In some embodiments, correcting the audio substring according to the subtitle substring may include:

[0086] If the subtitle substring and the audio substring have the same length, and the front characters of the subtitle substring and the front characters of the audio substring have the same pronunciation, and the back characters of the subtitle substring and the back characters of the audio substring have the same pronunciation, the audio substring is replaced with the subtitle substring, where the same pronunciation means that the similarity of the character pronunciation exceeds the pronunciation threshold.

[0087] Specifically, the front characters refer to the first number of characters found in sequence from the starting position of the substring to the end position of the substring; the back characters refer to the second number of characters found in sequence from the end position of the substring to the starting position of the substring. For example, taking the audio substring "Thanks for coming to my Zhibo room" as an example, assuming that the first number is 2 and the second number is 3, then the front characters are "Thanks" and the back characters are "Zhibo room".

[0088] When the length of the subtitle substring is the same as that of the audio substring and the front and back end characters are the same, it can be considered that there are only character differences between the two substrings. Also, since the characters in the subtitle substring are relatively accurate, the subtitle substring is directly used to replace the audio substring to correct recognition errors such as homophones, proper nouns, idioms, and personal names in the audio substring. Compared with comparing the characters of the subtitle substring with those in the audio substring to find and correct the misrecognized characters in the audio substring, directly using the subtitle substring to replace the audio substring in this application can reduce the comparison workload.

[0089] In some embodiments, correcting the audio substring according to the subtitle substring may include:

[0090] If the length of the subtitle substring is greater than that of the audio substring, and in the subtitle substring, there is a first character with the same pronunciation as the front end character of the audio substring, and there is a second character with the same pronunciation as the back end character of the audio substring, use the string composed of the first character, the second character, and the characters between the first character and the second character to replace the audio substring.

[0091] Specifically, in these embodiments, the subtitle substring includes a string that only has character differences from the audio substring. In this case, only the corresponding string needs to be extracted from the subtitle substring and the extracted string is used to replace the audio substring.

[0092] For example, assume that the subtitle substring is "Hello everyone. Thank you for coming to my live PhD defense room", and the audio substring is "Thank you for coming to my live PhD defense room". In the subtitle substring, there is a first character with the same pronunciation as the front end character of the audio substring, such as "Thank you", and there is a second character with the same pronunciation as the back end character of the audio substring, such as "defense room". The string composed of the first character, the second character, and the characters between the first character and the second character is "Thank you for coming to my live PhD defense room". Use the string "Thank you for coming to my live PhD defense room" extracted from the subtitle substring to replace the audio substring "Thank you for coming to my live PhD defense room".

[0093] In some embodiments, considering that when recognizing the subtitles of a video file based on optical character recognition technology, there may be problems with misrecognition of some artistic characters or special fonts in the subtitles. That is, the subtitle substring or the string extracted from the subtitle substring may also have misrecognized characters, but these misrecognized characters may be correct in the audio substring. In this case, if the subtitle substring or the string extracted from the subtitle substring is directly used to replace the audio substring, it may instead modify the audio substring incorrectly. For example, assume:

[0094] The subtitle substring is:以至于仅仅在这里呆上三天

[0095] The speech substring is:以至于仅仅在这里呆上三天

[0096] In the above two substrings, comparing "以至于" in the subtitle substring with "以至于" in the audio substring, obviously, "以至于" in the audio substring is correct. If the subtitle substring is directly used to replace the audio substring, the characters in the audio substring will be modified incorrectly instead.

[0097] To solve the above problem, it can be processed by combining the recognition confidence of each character in the subtitle substring. Among them, the recognition confidence represents the confidence corresponding to each character when the subtitle in the video file is recognized as each character in the subtitle substring. The recognition confidence can be an attribute自带的属性 of each character in the subtitle substring obtained based on optical character recognition technology. In other words, each character in the subtitle substring itself has a recognition confidence. The higher the recognition confidence of a character, the higher the confidence of the character. Based on the above description, using the subtitle substring or the string extracted from the subtitle substring to replace the audio substring can include:

[0098] In the string used to replace the audio substring, if there are low-confidence characters with a recognition confidence lower than the confidence threshold, according to the position of the low-confidence characters in the string, find the alternative characters corresponding to the low-confidence characters in the audio substring;

[0099] If the shape similarity between the low-confidence character and the alternative character exceeds the shape threshold, and / or the low-confidence character and the alternative character have the same pronunciation, use the alternative character to replace the low-confidence character;

[0100] Use the string with the low-confidence characters replaced to replace the audio substring.

[0101] Among them, the alternative character and the low-confidence character can be characters corresponding in position. In other words, in the string used to replace the audio substring, assuming that the kth character from the starting position is the low-confidence character, then in the audio substring, the kth character from the starting position is the alternative character. Where k is an integer. For example, assume:

[0102] The subtitle substring is:以至于仅仅在这里呆上三天

[0103] The speech substring is:以至于仅仅在这里呆上三天

[0104] In the subtitle substring, assuming that the third character '干' starting from the starting position is a low-confidence character, then in the audio substring, the third character '于' starting from the starting position is the alternative character corresponding to the low-confidence character.

[0105] After determining the low-confidence character and the alternative character, it is possible to judge whether to use the alternative character to replace the low-confidence character based on the shape similarity or pronunciation between the low-confidence character and the alternative character. Specifically, the shape similarity between the low-confidence character and the alternative character can be judged from aspects such as the number of strokes, Chinese character structure, four-corner code, and radical. If the shape similarity between the low-confidence character and the alternative character is relatively high or the pronunciation is relatively similar, it can indicate that the low-confidence character is a misrecognized character. At this time, the alternative character can be used to replace the low-confidence character to correct the characters in the subtitle substring. After completing the character correction, the corrected subtitle substring can be used to replace the audio substring. In this way, it is possible to avoid the incorrect introduction of the audio substring into the subtitle substring and improve the reliability of the document generation method.

[0106] In some embodiments, correcting the audio substring based on the subtitle substring may include:

[0107] If the characters in the subtitle substring do not have the same pronunciation as the front and back end characters of the audio substring, according to the character correspondence between the subtitle substring and the audio substring, find the different character pairs where the subtitle substring and the audio substring differ;

[0108] Among the different character pairs, if the characters in the subtitle substring and the characters in the audio substring meet the preset conditions, use the characters in the subtitle substring to replace the characters in the audio substring.

[0109] Specifically, in these embodiments, it is impossible to compare the two substrings based on the length and pronunciation of the subtitle substring and the audio substring, that is, the lengths of the two substrings are different, and the pronunciations of the front and back end characters are also different. In this case, the edit distance between the subtitle substring and the audio substring can be calculated to find the different character pairs where the subtitle substring and the audio substring differ, and then judge whether the characters in each different character pair meet the preset conditions. The preset conditions may include shape similarity, pronunciation similarity, etc. If the characters in the different character pair meet the preset conditions, use the characters in the subtitle substring to replace the characters in the audio substring.

[0110] That is, in these embodiments, instead of using the subtitle substring to directly replace the audio substring, the difference character pair between the subtitle substring and the audio substring is found, and then if the characters in the difference character pair meet the preset conditions, the characters in the subtitle substring are used to replace the characters in the audio substring. In this way, the voice substring is corrected to ensure the accuracy of the video transcript.

[0111] Furthermore, in some embodiments, after replacing the characters in the audio substring with the characters in the subtitle substring, the manuscript generation method of the present application further includes:

[0112] Inputting the first audio substring without character replacement and the second audio substring with character replacement into the trained language model respectively, and the language model outputs the first language score of the first audio substring and the second language score of the second audio substring;

[0113] If the second language score is higher than the first language score, the second audio substring is used as the corrected audio substring; if the second language score is not higher than the first language score, the first audio substring is used as the corrected audio substring.

[0114] Specifically, the language score is used to evaluate whether the audio substring conforms to the language norms, such as the fluency of the language. The higher the language score, the higher the probability that the audio substring conforms to the language norms. If the second language score is not higher than the first language score, it means that the replacement of the audio substring is wrong, that is, the second audio substring that has completed the character replacement does not conform to the language norms as well as the first audio substring that has not undergone character replacement. In this case, the first audio substring can be used as the corrected audio substring. On the contrary, if the second language score is higher than the first language score, it means that the replacement of the audio substring is correct, that is, the second audio substring that has completed the character replacement conforms to the language norms better than the first audio substring that has not undergone character replacement. In this case, the second audio substring can be used as the corrected audio substring. Based on the language model, the incorrect replacement of the audio substring can be corrected to ensure the accuracy of the correction.

[0115] Furthermore, in some embodiments, when using the second audio substring as the corrected audio substring, the method further includes:

[0116] Record the difference character pairs;

[0117] According to the character correspondence between the subtitle recognition result and the audio recognition result, if a character pair that matches the difference character pair is found, the characters in the subtitle recognition result are used to replace the characters in the audio recognition result.

[0118] In simple terms, based on the subtitle substring and audio substring, after determining the character pairs that may be different in the subtitle recognition results and the audio recognition results, the audio recognition results are fully replaced based on the recorded different character pairs. In this way, there is no need to replace the same different character pairs in other audio substring corrections, reducing the correction workload.

[0119] In summary, the correction of the audio substring is completed. The following describes how to add punctuation to the corrected audio substring (it can be known from the previous preprocessing that the corrected audio substring does not include any punctuation).

[0120] In some embodiments, since the audio substring is a string located between two target segmentation punctuation marks, the target segmentation punctuation marks can be directly added after the corrected audio substring is completed. In addition, the segmentation punctuation marks (such as commas) in the audio substring deleted during preprocessing can be ignored and no longer added, so as to avoid the problem of misplacement of the segmentation punctuation marks due to the different number of words in the second audio substring after correction and the first audio substring before correction.

[0121] For example, suppose:

[0122] The first audio substring before correction is: I like to eat cake, biscuits, ice cream. The second audio substring after correction is: I like to eat cheesecake, biscuits, ice cream.

[0123] In the first audio substring, the comma is located after the sixth and eighth characters. After correction, if a comma is added after the sixth and eighth characters of the second audio substring, the second audio substring becomes: I like to eat cheese, cake, cookies and ice cream. Obviously, this is wrong. Therefore, by ignoring the split punctuation, more correction problems can be avoided.

[0124] Further, in some embodiments, it is considered that the punctuation in the subtitle substring may be relatively rich. For example, a subtitle substring may be expressed as "Don't be confused by this strange way of drinking~~", where "~~" can increase the semantic information of the string. In view of this, the punctuation of the subtitle substring can be added to the corrected audio substring to make the punctuation of the audio substring richer. However, considering that the punctuation in the subtitle substring will be deleted in the previous preprocessing process, in order to retain these punctuation, the subtitle substring before the preprocessing operation can be saved as the original array string before performing the preprocessing operation. Furthermore, in the case of using the subtitle substring to replace the audio substring, if the subtitle substring has the same length as the audio substring, and the length of the subtitle substring is greater than the length of the corresponding original array string, and the recognition confidence of the characters of the subtitle substring is higher than the confidence threshold, the original array string corresponding to the subtitle substring is used to replace the audio substring. In this way, the richness of the punctuation of the corrected audio sub-segmentation is improved.

[0125] Step S14, outputting the corrected audio recognition result as a video document of a video file.

[0126] Before outputting the corrected audio recognition result as a video manuscript, it is also possible to check whether the corrected audio recognition result includes the preset characters, and if so, remove the preset characters. In this way, by setting reasonable preset characters, the content of the video manuscript can be modified to further improve the accuracy of the video manuscript.

[0127] In summary, the entire description of the manuscript generation method of the present application has been completed. Based on the manuscript generation method of the present application, a video manuscript with high accuracy can be generated. In the technical solutions of some embodiments of the present application, segmentation punctuation is added to the audio recognition result of the video file, and the audio substring segmented according to the target segmentation punctuation is obtained, and according to the timestamp of the audio substring, the subtitle substring in the same time period as the audio substring is obtained in the subtitle recognition result of the video file, and the audio substring is corrected based on the subtitle substring. Since the text in the subtitle substring has a high accuracy, based on the subtitle substring, the homophones, proper nouns, idioms, names, etc. that are recognized incorrectly in the voice substring can be corrected, thereby improving the accuracy of the generated video manuscript.

[0128] See also Figure 5 , which is a module diagram of a document generation system provided in one embodiment of the present application. Figure 5 In the document generation system, the document generation system includes:

[0129] The recognition module 51 specifically includes an audio recognition module 511 and a video recognition module 512. The audio recognition module 511 is used to perform audio recognition on the video file to obtain an audio recognition result, and the video recognition module 512 is used to recognize the subtitles of the video file to obtain a subtitle recognition result;

[0130] The segmentation module 52 is used to add segmentation punctuation marks to the audio recognition result and obtain the audio substrings segmented according to the target segmentation punctuation marks;

[0131] The correction module 53 is used for obtaining, for any audio substring, a subtitle substring in the same time period as the audio substring in the subtitle recognition result according to the timestamp of the audio substring, and correcting the audio substring according to the subtitle substring;

[0132] The document output module 56 is used to output the corrected audio recognition result as a video document of a video file.

[0133] Furthermore, the document generation system may also include a verification module 54 and a filtering module 55. The verification module 54 is used to input the first audio substring without character replacement and the second audio substring with character replacement into the trained language model respectively, and the language model outputs the first language score of the first audio substring and the second language score of the second audio substring; if the second language score is higher than the first language score, the second audio substring is used as the corrected audio substring, and if the second language score is not higher than the first language score, the first audio substring is used as the corrected audio substring.

[0134] The filtering module 55 is used to check whether the corrected audio recognition result includes the preset characters before the document output module 56 outputs the corrected audio recognition result as a video document. If so, the preset characters are removed.

[0135] See also Figure 6 , is a schematic diagram of an electronic device provided by an embodiment of the present application. The electronic device includes a processor and a memory, the memory is used to store a computer program, and when the computer program is executed by the processor, the above method is implemented.

[0136] The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0137] As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the method in the embodiment of the present invention. The processor executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory, that is, implementing the method in the above method embodiment.

[0138] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0139] One embodiment of the present application further provides a computer-readable storage medium, which is used to store a computer program. When the computer program is executed by a processor, the above method is implemented.

[0140] Although the embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A manuscript generation method, characterized in that: The method comprises: Performing audio recognition on the video file to obtain an audio recognition result, and recognizing the subtitles of the video file to obtain a subtitle recognition result; Adding segmentation punctuation marks to the audio recognition result, and obtaining audio substrings segmented according to target segmentation punctuation marks; For any of the audio substrings, according to the timestamp of the audio substring, obtaining a subtitle substring in the same time period as the audio substring in the subtitle recognition result, and correcting the audio substring according to the subtitle substring; The corrected audio recognition result is output as a video document of the video file.

2. The method according to claim 1, characterized in that The subtitle recognition result includes an array of subtitles recognized from the video file at multiple time points; The obtaining of the subtitle substring in the subtitle recognition result that is in the same time period as the audio substring includes: Extracting a target subtitle array in the same time period as the audio substring from the subtitle recognition result; The target subtitle arrays are combined into different array strings, and the array string having the greatest similarity to the audio substring is used as the subtitle substring.

3. The method according to claim 2, characterized in that The audio substring and the subtitle array have timestamps, and the timestamp of the audio substring includes a minimum timestamp and a maximum timestamp; The step of extracting a target subtitle array that is in the same time period as the audio substring from the subtitle recognition result includes: Determining a first timestamp that is less than the minimum timestamp, and a second timestamp that is greater than the maximum timestamp; In the subtitle recognition result, a subtitle array whose timestamp is between the first timestamp and the second timestamp is extracted as the target subtitle array.

4. The method according to claim 1, characterized in that The correcting the audio substring according to the subtitle substring includes: If the subtitle substring and the audio substring have the same length, and the front character of the subtitle substring and the front character of the audio substring have the same pronunciation, and the back character of the subtitle substring and the back character of the audio substring have the same pronunciation, the subtitle substring is used to replace the audio substring, wherein the same pronunciation means that the similarity of the character pronunciation exceeds the pronunciation threshold.

5. The method according to claim 4, characterized in that The correcting the audio substring according to the subtitle substring includes: If the length of the subtitle substring is greater than the length of the audio substring, and in the subtitle substring, there is a first character that has the same pronunciation as the front character of the audio substring, and there is a second character that has the same pronunciation as the back character of the audio substring, the audio substring is replaced with a character string consisting of the first character, the second character, and the characters between the first character and the second character.

6. The method according to claim 4 or 5, characterized in that The characters in the subtitle substring have recognition confidence; and the replacing the audio substring comprises: In the character string used to replace the audio substring, if there is a low-confidence character whose recognition confidence is lower than a confidence threshold, searching the audio substring for an alternative character corresponding to the low-confidence character according to the position of the low-confidence character in the character string; If the shape similarity between the low-confidence character and the candidate character exceeds a shape threshold, and / or the low-confidence character and the candidate character have the same pronunciation, the candidate character is used to replace the low-confidence character; The audio substring is replaced with a string in which low-confidence characters have been replaced.

7. The method according to claim 4 or 5, characterized in that The correcting the audio substring according to the subtitle substring includes: If the characters in the subtitle substring do not have the same pronunciation as the front and rear characters of the audio substring, searching for a difference character pair between the subtitle substring and the audio substring according to the character correspondence between the subtitle substring and the audio substring; In the difference character pair, if the characters in the subtitle substring and the characters in the audio substring meet a preset condition, the characters in the subtitle substring are used to replace the characters in the audio substring.

8. The method according to claim 7, characterized in that After replacing the characters in the audio substring with the characters in the subtitle substring, the method further comprises: Inputting a first audio substring without character replacement and a second audio substring with character replacement into a trained language model respectively, and having the language model output a first language score for the first audio substring and a second language score for the second audio substring; If the second language score is higher than the first language score, the second audio substring is used as the corrected audio substring; if the second language score is not higher than the first language score, the first audio substring is used as the corrected audio substring.

9. The method according to claim 8, characterized in that When using the second audio substring as the corrected audio substring, the method further includes: Recording the difference character pairs; According to the character correspondence between the subtitle recognition result and the audio recognition result, if a character pair that matches the difference character pair is found, the characters in the subtitle recognition result are used to replace the characters in the audio recognition result.

10. The method according to claim 4 or 5, characterized in that Before correcting the audio substring, the method further includes: The following preprocessing operations are performed on the subtitle substring and the audio substring: If the subtitle substring and the audio substring include a segmentation mark, delete the segmentation mark; and / or If the subtitle substring and the audio substring include non-Chinese or English characters, delete the non-Chinese or English characters; and / or Convert the numeric characters in the subtitle substring.

11. The method according to claim 10, characterized in that The characters in the subtitle substring have recognition confidence; before performing the preprocessing operation, the method further includes: Saving the subtitle substring before the preprocessing operation as an original array string; The step of replacing the audio substring with the subtitle substring comprises: If the subtitle substring has the same length as the audio substring, and the length of the subtitle substring is greater than the length of the corresponding original array string, and the recognition confidence of the characters of the subtitle substring is higher than the confidence threshold, the audio substring is replaced with the original array string corresponding to the subtitle substring.

12. The method according to claim 1, characterized in that After completing the correction of the audio substring, the method further includes: After the corrected audio substring is completed, the target segmentation punctuation is added.

13. The method according to claim 1, characterized in that Before outputting the corrected audio recognition result as a video manuscript, the method further includes: Check whether the corrected audio recognition result includes preset characters, and if so, remove the preset characters.

14. A document generation system, characterized in that: The system comprises: The recognition module is used to perform audio recognition on the video file to obtain an audio recognition result, and to recognize the subtitles of the video file to obtain a subtitle recognition result; A segmentation module, used to add segmentation punctuation marks to the audio recognition result and obtain audio substrings segmented according to target segmentation punctuation marks; a correction module, configured to obtain, for any of the audio substrings, a subtitle substring in the same time period as the audio substring in the subtitle recognition result according to the timestamp of the audio substring, and correct the audio substring according to the subtitle substring; The document output module is used to output the corrected audio recognition result as a video document of the video file.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.

16. An electronic device, characterized in that: The electronic device comprises a processor and a memory, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the method according to any one of claims 1 to 13 is implemented.

Images