Audio and text combination method, device, electronic device and storage medium

Abstract

The embodiments of the present disclosure provide a method, device, electronic device and storage medium for combining audio and text. The method for combining audio and text includes: performing text recognition on key frames in a video file to obtain a first subtitle set; performing subtitle switching detection on the first subtitle set to obtain a target subtitle and a timestamp of a subtitle switch between the target subtitles; based on the timestamp, extracting audio from an audio file of the video file to obtain a target audio; and constructing a combination pair of text and audio based on the target subtitle and the target audio. The embodiments of the present disclosure can improve the matching degree between audio and text.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to an audio and text combination method, device, electronic device and storage medium. Background Art

[0002] Speech recognition technology is a type of artificial intelligence technology and is currently widely used in many fields such as voice input, voice interaction, voice search, voice control, and voice subtitles. Especially under the current impact of the epidemic, contactless voice interaction and control highlight the value of speech recognition technology. Since speech recognition technology is based on deep learning algorithms, it relies on massive and high-quality training data to improve the accuracy of the algorithm, so that the quality of speech recognition can achieve ideal results.

[0003] The general method for generating training data for existing speech recognition algorithms is to obtain video resources with subtitles, such as audiobooks, live commentary, documentaries, dramas, interviews, news, variety shows, self-media and many other fields; use OCR (optical character recognition) text recognition technology to extract subtitles and subtitle timestamps from the video; match the audio clip corresponding to the subtitle from the video through the timestamp, and combine the subtitles and the audio clip to form a <text, audio> pair training sample.

[0004] However, existing subtitle extraction solutions all use OCR text recognition technology to extract subtitles from videos. This method only relies on unilateral information from the video content, making the accuracy of subtitle extraction dependent on the accuracy of OCR recognition technology. When there is a complex background in the video or the subtitle position is not fixed, the OCR recognition effect is poor, so when matching <text, audio> pairs, it is easy for the subtitle and audio time information to be misaligned, and the subtitle content to be misaligned with the audio content. In turn, the recognition accuracy of the speech recognition algorithm is affected. Summary of the invention

[0005] The embodiments of the present disclosure provide an audio and text combination method, device, electronic device and storage medium to solve or alleviate one or more technical problems in the prior art.

[0006] As a first aspect of an embodiment of the present disclosure, an embodiment of the present disclosure provides an audio and text combination method, comprising:

[0007] Performing text recognition on key frames in the video file to obtain a first subtitle set;

[0008] Performing subtitle switching detection on the first subtitle set to obtain a timestamp of a subtitle switching between a target subtitle and the target subtitle;

[0009] Based on the timestamp, extracting audio from the audio file of the video file to obtain target audio;

[0010] Based on the target subtitles and the target audio, a text-audio combination pair is constructed.

[0011] As a second aspect of an embodiment of the present disclosure, an embodiment of the present disclosure provides an audio and text combination device, including:

[0012] A text recognition module, used for performing text recognition on key frames in a video file to obtain a first subtitle set;

[0013] A subtitle switching detection module, used to perform subtitle switching detection on the first subtitle set, and obtain a timestamp of a subtitle switching between a target subtitle and the target subtitle;

[0014] A target audio interception module, used to intercept audio from the audio file of the video file based on the timestamp to obtain target audio;

[0015] The audio-text combination module is used to construct a combination pair of text and audio based on the target subtitles and the target audio.

[0016] As a third aspect of an embodiment of the present disclosure, an embodiment of the present disclosure provides an electronic device, including:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the audio and text combination method provided in the embodiment of the present disclosure.

[0020] As a fourth aspect of an embodiment of the present disclosure, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the audio and text combination method provided by the embodiment of the present disclosure.

[0021] As a fourth aspect of an embodiment of the present disclosure, an embodiment of the present disclosure provides a computer program product, including a computer program, which implements the audio and text combination method provided by the embodiment of the present disclosure when executed by a processor.

[0022] The technical solution provided by the embodiment of the present disclosure performs text recognition on key frames in a video file to obtain a first subtitle set, and performs subtitle switching detection on the first subtitle set to obtain a target subtitle and a timestamp of a subtitle switching between the target subtitles. Thus, based on the timestamp, the audio is intercepted from the audio file of the video file to obtain an accurate target audio, so that the combination pair of text and audio constructed based on the target subtitle and the target audio has a high degree of matching, avoiding the situation where the subtitle and audio time information are misaligned, and the subtitle content and the audio content are not aligned.

[0023] The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present disclosure will be readily apparent by reference to the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments according to the present disclosure and should not be regarded as limiting the scope of the present disclosure.

[0025] Figure 1 is a flow chart of a method for combining audio and text according to an embodiment of the present disclosure;

[0026] Figure 2 is a flowchart of a subtitle processing method according to an embodiment of the present disclosure;

[0027] Figure 3A is a schematic diagram of a key frame of an embodiment of the present disclosure;

[0028] Figure 3B is a schematic diagram of a key frame of another embodiment of the present disclosure;

[0029] Figure 4 is a schematic diagram of a key frame of another embodiment of the present disclosure;

[0030] Figure 5 is a flowchart of a process for constructing training samples of a speech recognition model according to an embodiment of the present disclosure;

[0031] Figure 6 is a flowchart of a subtitle file generation process according to an embodiment of the present disclosure;

[0032] Figure 7 is a flowchart of subtitle verification, switching detection and error correction according to an embodiment of the present disclosure;

[0033] Figure 8 is a schematic diagram of the anatomy of a pictographic character according to an embodiment of the present disclosure;

[0034] Fig. 9 is a schematic diagram of a subtitle file according to an embodiment of the present disclosure;

[0035] Fig.10 is a schematic diagram of an audio and text combination device according to an embodiment of the present disclosure;

[0036] Fig.11 is a schematic diagram of an electronic device according to an embodiment of the present disclosure; DETAILED DESCRIPTION

[0037] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the spirit or scope of the present disclosure. Therefore, the drawings and descriptions are considered to be exemplary and non-restrictive in nature.

[0038] Figure 1 FIG. 1 is a flow chart of a method for combining audio and text provided by an embodiment of the present disclosure. Figure 1 It is shown that the audio and text combination method may include the following steps:

[0039] S110, performing text recognition on key frames in the video file to obtain a first subtitle set;

[0040] S120, performing subtitle switching detection on the first subtitle set to obtain a timestamp of a subtitle switching between a target subtitle and a target subtitle;

[0041] S130, based on the timestamp of the subtitle switching of the target subtitle, intercepting the audio from the audio file of the video file to obtain the target audio;

[0042] S140, constructing a text-audio combination pair based on the target subtitle and the target audio.

[0043] In this example, text recognition is performed on key frames in a video file to obtain a first subtitle set, and subtitle switching detection is performed on the first subtitle set to obtain a timestamp when a subtitle switch occurs between a target subtitle and the target subtitle. Thus, based on the timestamp, audio is intercepted from an audio file of the video file to obtain an accurate target audio, so that the text and audio combination constructed based on the target subtitle and the target audio has a high matching degree, avoiding the situation where the subtitle and audio time information are misaligned, and the subtitle content and the audio content are not aligned.

[0044] Exemplarily, a video file includes multiple image frames played in time sequence and corresponding audio. Video files can include audio books, live commentary, documentaries, dramas, interviews, news, variety shows, self-media and other video resources. Video files can be short videos or longer videos.

[0045] For example, in the above step S110, text recognition may be performed on all key frames in the video file, or on some key frames. A key frame may be any image frame in the video file, or an image frame with text.

[0046] Exemplarily, text recognition may adopt OCR recognition technology.

[0047] Exemplarily, the first subtitle set includes a plurality of subtitles, each of which may be from a different key frame. The subtitles may have the same content or different content.

[0048] Exemplarily, for a subtitle, it may appear in a plurality of consecutive key frames. When it is different from the subtitle of the previous frame or the subtitle of the next frame, it may be considered that a subtitle switch has occurred. Therefore, subtitle switch detection may be to detect whether a subtitle switch has occurred by using the similarity of the subtitles of two adjacent frames. If the similarity of the subtitles between the two adjacent frames is greater than a set subtitle similarity threshold, it is determined that no subtitle switch has occurred. If the similarity of the subtitles between the two adjacent frames is less than a set subtitle similarity threshold, it is determined that a subtitle switch has occurred. If a subtitle switch has occurred, the timestamp of the subtitle switch may be determined based on the timestamps of the two adjacent frames. For example, the timestamp of the first frame of the two adjacent frames is used as the timestamp of the subtitle switch, or the timestamp of the second frame of the two adjacent frames is used as the timestamp of the subtitle switch.

[0049] Exemplarily, the timestamp of subtitle switching includes a start timestamp and an end timestamp. The subtitles in each key frame between the start timestamp and the end timestamp have the same subtitle content, that is, the similarity of subtitles in two adjacent frames is greater than a set subtitle similarity threshold.

[0050] Exemplarily, the constructed text-audio combination pair can be applied to the training of a speech recognition model.

[0051] In some embodiments, in the above step S110, text recognition is performed on key frames in the video file to obtain the first subtitle set, including operations such as verifying whether the text is a subtitle and filtering non-subtitle text.

[0052] For example, Figure 2 FIG. 1 is a flowchart of a subtitle processing method provided by an embodiment of the present disclosure. Figure 2 As shown, the subtitle processing method may include the following steps:

[0053] S210, performing text recognition on key frames in the video file to obtain candidate subtitle texts;

[0054] S220, determining, in the video file, verification audio corresponding to the candidate subtitle text;

[0055] S230, if the relationship between the candidate subtitle text and the verification audio meets the set condition, determining the candidate subtitle text as the subtitle text of the video file;

[0056] S240: Acquire subtitle text of the video file to obtain a first subtitle set.

[0057] In this embodiment, the verification text of speech recognition is used to verify the candidate subtitle text obtained by text recognition to determine whether the candidate subtitle text is a real subtitle, thereby improving the accuracy of subtitle recognition, so as to avoid the misalignment of text and audio time information and the misalignment of text content and audio content when subsequently constructing a combination of text and audio.

[0058] For example, in the above step S210, text recognition may be performed on all key frames in the video file, or on some key frames. A key frame may be any image frame in the video file, or an image frame with text.

[0059] Exemplarily, the above step S210 may be to perform text recognition on the key frame using OCR text recognition technology. When performing text recognition on the key frame, one or more texts in the key frame may be obtained, such as text blocks and text lines. In this case, some texts may not be subtitles, and these texts may be used as candidate subtitles. Then, audio verification is performed on the candidate subtitles to accurately determine whether the candidate subtitles are real subtitles.

[0060] Exemplarily, text recognition is performed on key frames in a video file to obtain at least one text, and candidate subtitle texts are determined from multiple texts in the key frames based on the subtitle area of ​​the video file and the position information of each text in the key frames to further determine whether the candidate subtitle texts are subtitles. For example, if the position information of the text matches the subtitle area, it is considered that these texts are subtitles. If the position information of the text does not match the subtitle area, it is considered that these texts may not be subtitles and further verification is required to determine whether they are subtitles. The position information may include the position area of ​​the text in the key frame, for example, the text is framed by a rectangular frame, and the coordinates of the four corners of the rectangular frame are the position area of ​​the text in the key frame.

[0061] Exemplarily, the candidate subtitle text is text for which it is not determined whether it is a subtitle, for example, it may be a subtitle, or it may be background text or watermark text in a key frame.

[0062] For example, Figure 3A and Figure 3BAs shown, this is a scene where teacher Actor 1 is teaching. The text in the text area pointed to by teacher Actor 1 in the figure is the background text, and the two text blocks below the text area, "This is a poem by Li Bai, a poet of the Tang Dynasty" and "The poet used the metaphor of the depth of the pool to describe the friendship between Wang Lun and him, using an exaggerated technique" are subtitles. If OCR recognition is used, both the background text and the subtitles are recognized at the same time, and it is impossible to determine which ones are subtitle texts and which ones are background texts. Therefore, the method provided by the embodiment of the present disclosure can accurately recognize subtitles and filter out background text.

[0063] For example, in the above step 220, based on the timestamp corresponding to the candidate subtitle text, the corresponding audio segment can be intercepted from the audio in the video file to obtain the verification audio. The timestamp can be the timestamp of the text switching of the candidate subtitle text in the video file, that is, the first appearance time and the last appearance time of the candidate subtitle text in the time-sequential continuous image frames. Thus, the target audio corresponding to the candidate subtitle text can be accurately intercepted, thereby improving the accuracy of the subsequent verification of the subtitles.

[0064] In some embodiments, the overlap or confidence level between the candidate subtitle text and its corresponding verification text may be used to determine whether it is a subtitle.

[0065] Exemplarily, the above step S230 may include: performing speech recognition on the verification audio to obtain the verification text; when the overlap between the candidate subtitle text and the verification text is greater than a set overlap threshold, determining that the candidate subtitle text is the subtitle text of the video file.

[0066] Exemplarily, the above-mentioned speech recognition may adopt ASR (Automatic Speech Recognition) technology.

[0067] For example, the overlap degree refers to the ratio between the same characters and all characters in two character strings, or the ratio between the same characters and one of the character strings. The ratio can be the ratio between the number of characters or the ratio between the lengths of the characters.

[0068] Exemplarily, the degree of overlap m between the candidate subtitle text and its corresponding verification text is m=len(string(ocr)&string(asr)) / len(string(ocr), wherein string(ocr) is the candidate subtitle text, string(asr) is the verification text corresponding to the candidate subtitle text, len(string(ocr)&string(asr)) is the length of the character intersection between the candidate subtitle text and the verification text, and len(string(ocr) is the length of the candidate subtitle text. The character intersection includes the set of identical characters between the candidate subtitle text and the verification text.

[0069] In this example, the overlap between the candidate subtitle text and the verification text recognized by the corresponding audio can be used to determine whether the candidate subtitle text is a subtitle, thereby improving the accuracy of subtitle recognition.

[0070] In some embodiments, when a large amount of subtitle recognition is performed on a video file, some relatively obvious non-subtitle texts such as watermarks may be pre-filtered.

[0071] For example, in the above step S210, text recognition is performed on key frames in the video file to obtain candidate subtitle texts, including:

[0072] Performing text recognition on key frames in the video file to obtain a first text set;

[0073] Based on the first text set, determining non-subtitle text;

[0074] Filtering non-subtitle texts in the first text set to obtain a second text set;

[0075] Based on the second text set, candidate subtitle texts are determined.

[0076] In this example, pre-filtering non-subtitle text is beneficial to improving the efficiency of subsequent subtitle recognition.

[0077] Exemplarily, before performing text recognition on each key frame in the video file, extract the key frame from the video file. Perform text detection on each image frame in the video file, and compare whether the two frames are repeated by the frame difference method. If they are repeated, delete the redundant frames, and finally obtain multiple key frames, and each key frame contains text. It is possible to determine whether the two frames are repeated frames by comparing the similarity between the texts in the two frames, such as the edit distance. The model for performing text detection on the image frame can be implemented by using the RCNN deep learning model. In addition, in order to increase the speed of extracting key frames, key frames can be obtained according to the set interval time, for example, 3 frames are extracted as key frames per second, and one frame is extracted as a key frame every 8 seconds.

[0078] For example, by performing text recognition on a key frame, the text information and position information of each text in the key frame can be obtained. Each text can be obtained in the form of a text block or a text line. For example, the text recognition result of a key frame can be marked as:

[0079] [(text1,(xmin,xmax,ymin,ymax)),(text2,(xmin,xmax,ymin,ymax)),…];

[0080] Where text1 is a text, text2 is another text; xmin, xmax represent the minimum horizontal coordinate and maximum horizontal coordinate of the text block in the current key frame; ymin, ymax represent the minimum vertical coordinate and maximum vertical coordinate of the text block in the current key frame. Figure 4 As shown, it shows the position information of the text block "Study hard and make progress every day" in the current frame.

[0081] In some embodiments, non-subtitle text includes background text such as watermarks, station logos, or advertisements. For example, in some TV dramas and variety shows, there may be background information such as station logos or advertisements. Figure 4 As shown, Figure 4 The "XXTV TV Series Channel" in the video is a station logo. These background information often appear at a fixed position in the video and exist throughout the entire playback time of the video, that is, they exist in all key frames of the video. Therefore, based on the position information of each text in the first text set, the frequency of occurrence of texts with the same position can be counted, and texts with a frequency of occurrence higher than a set threshold can be determined as non-subtitle texts.

[0082] Exemplarily, the above-mentioned determination of non-subtitle text based on the position information of each text in the first text set may include: determining the frequency of occurrence of texts in the same position based on the position information of each text in the first text block set; when the frequency of occurrence is greater than a set frequency threshold, determining the texts in the same position as non-subtitle text.

[0083] It should be noted that the position coordinate information of the same text in the previous and next frames or between different frames may have slight differences. For example, Figure 4 The position coordinate information of "Study hard and make progress every day" in the previous and next two frames can be (125, 512, 678, 780) and (126, 513, 679, 781). Therefore, the two position coordinate information can be normalized and regarded as the same position.

[0084] Exemplarily, when it is determined that the positional distance between two texts belonging to different key frames is less than a predetermined distance threshold based on the positional information of the two texts, the two texts are determined to be texts at the same position.

[0085] Exemplarily, the position of text block 1 at key frame a is (x1, y1); the position of text block 2 at key frame b is (x2, y2). If ||x1 - x2|| < a and ||y1 - y2|| < b, then text block 1 and text block 2 are texts at the same position.

[0086] Exemplarily, the occurrence frequency of each text at the same position is counted to obtain the occurrence frequency of each text at the same position. The text with the highest occurrence frequency is determined to be non-caption text, or a text with an occurrence frequency greater than 80% or 70% of the total number of key frames of the video file is determined to be non-caption text.

[0087] In some embodiments, after filtering non-caption text from the first text set, a second text set is obtained. At this time, all texts in the second text set can be used as candidate caption texts for audio verification to obtain the caption texts of the video file. Or, the texts in the second text set are screened, and the screened texts are used as candidate caption texts.

[0088] In some embodiments, the candidate caption texts can be screened from the second text set by using the caption area.

[0089] In some embodiments, the method for determining the caption area can be: after filtering non-caption text, the remaining texts can be sorted according to the occurrence frequency of their positions, and the position area where the text with an occurrence frequency greater than a set threshold, such as the text with the highest frequency, is located is determined as the caption area of the video text.

[0090] In some embodiments, the caption area can be used to determine whether a text in the second text set after filtering non-caption text is a caption. For example, if the position of the text matches the caption area, the text can be determined to be a caption. If the position of the text does not match the caption area, it cannot be determined whether the text is a caption and further verification is required.

[0091] Exemplarily, the determination of candidate caption texts based on the second text set includes:

[0092] Determining a first caption area based on the position information of each text in the second text set;

[0093] For each text in the second text set, when the degree of matching between the position information of the text and the first caption area is less than a set matching threshold, the text is determined to be a candidate caption text.

[0094] In this example, the candidate subtitle text is the text whose status as a subtitle is uncertain and needs to be further verified using the verification text obtained from speech recognition to improve the accuracy of subtitle recognition.

[0095] Exemplarily, determining the first subtitle area based on the position information of each text in the second text set may include: sorting the occurrence frequencies of texts with the same position in the second text set, and determining the position area where the text with the highest occurrence frequency is located as the first subtitle area.

[0096] Exemplarily, as Figure 3A and Figure 3B shown, they are two key frames in a video of a teacher giving a lecture. There are background subtitles in the key frames and they are multi-line subtitles. The text in the text area pointed to by teacher Actor1 is background text. Verifying the text area pointed to by teacher Actor1 using the determined subtitle area, it can be found that the subtitle area does not match the text area pointed to by teacher Actor1. Assuming the subtitle area is denoted as (Xmin, Xmax, Ymin, Ymax), and the text area pointed to by teacher Actor1 is (Xminc, Xmaxc, Yminc, Ymaxc), if Yminc < Ymin or Ymaxc > Ymax, the text corresponding to this text area is text outside the subtitle area and cannot be determined as a subtitle. However, if this text area is directly considered not to be a subtitle, subtitles may be lost. Therefore, this kind of text is determined as candidate subtitle text, and the verification text obtained from the corresponding audio of speech recognition is used to verify this candidate subtitle text to determine whether this text is a subtitle. Thus, on the one hand, subtitle loss can be avoided, and on the other hand, the accuracy of subtitle recognition can be improved.

[0097] Since subtitles can be recognized using the subtitle area, therefore, recognizing the first text set or the second text set using the subtitle area can quickly obtain the first subtitle set.

[0098] Exemplarily, determining the candidate subtitle text based on the second text set further includes:

[0099] For each text in the second text set, when the degree of matching between the position information of the text and the first subtitle area is greater than a set matching threshold, determining the text as the subtitle text of the video file.

[0100] Exemplarily, in the above step S210, performing text recognition on the key frames of the video file to obtain the first subtitle set may include:

[0101] Performing text recognition on each key frame in the video file to obtain a third text set;

[0102] determining a second subtitle region based on position information of each text in the third text set;

[0103] For each text in the third text set, when the matching degree between the position information of the text and the second subtitle area is greater than a set matching threshold, the text is determined to be a subtitle text of the video file.

[0104] In this example, the subtitle area is used to determine the subtitle, so that the subtitle can be determined quickly and accurately, thereby improving the efficiency of subtitle recognition.

[0105] Exemplarily, text recognition is performed on each key frame in the video file to obtain a third text set, and the above-mentioned method for filtering non-subtitle texts can be used to filter the non-subtitle texts in the third text set to obtain a filtered third text set. For details, reference can be made to the filtering process of the non-subtitle texts in the second text set and the recognition process of the non-subtitle texts, which will not be described in detail here.

[0106] Exemplarily, the process of determining the second subtitle area may be the same as or similar to the aforementioned process of determining the first subtitle area. For details, please refer to the aforementioned content and will not be described in detail here.

[0107] Exemplarily, the first subtitle area and the second subtitle area may be the same area or different areas.

[0108] In some embodiments, after obtaining the first subtitle set, operations such as subtitle switching detection and deduplication can be performed on the first subtitle set to avoid repeated text and audio combination pairs.

[0109] Exemplarily, in the above step S120, it may include:

[0110] Based on the similarity between any two subtitle texts belonging to adjacent key frames in the first subtitle set, determining a timestamp of subtitle switching, and dividing the first subtitle set by using the timestamp to obtain a plurality of second subtitle sets;

[0111] De-duplicate the second subtitle set to obtain target subtitles;

[0112] Based on the timestamp of the second subtitle set, a timestamp at which subtitle switching occurs in the target subtitle is determined.

[0113] In this example, subtitle switching detection is performed on the first subtitle set to obtain multiple second subtitle sets, each of which includes multiple subtitles with the same content. Therefore, each second subtitle set can be deduplicated to obtain target subtitles for combining text and audio pairs, thereby avoiding repeated text and audio combination pairs.

[0114] Exemplarily, before performing subtitle switching detection on the first subtitle set, subtitles belonging to the same key frame are merged. For example, a key frame may have two lines of subtitles, and the two lines of subtitles can be merged into one subtitle. In this way, subsequent subtitle switching detection can be facilitated.

[0115] Exemplarily, in the first subtitle set, the similarity of any two subtitle texts belonging to adjacent key frames is calculated. When the similarity is greater than a set first similarity threshold, the subtitle texts of the adjacent key frames are determined to be subtitles with the same content; when the similarity is less than the set similarity threshold, the two subtitle texts of the adjacent key frames are determined to be subtitles with different contents, that is, subtitle switching occurs, and the timestamp corresponding to one of the two key frames is determined as the timestamp of the subtitle switching.

[0116] After obtaining all timestamps of subtitle switching in the first subtitle set, the timestamps are used as dividing lines, and subtitles corresponding to key frames whose timestamps belong to between two dividing lines are divided into the same subtitle set, thereby obtaining multiple second subtitle sets.

[0117] Exemplarily, the second subtitle set includes a plurality of subtitles, and the plurality of subtitles are subtitles having the same content.

[0118] Exemplarily, the start timestamp and the end timestamp corresponding to the second subtitle set are the timestamps when the target subtitle is switched.

[0119] In some embodiments, although the similarities between the subtitle texts belonging to adjacent key frames in the second subtitle set are greater than the first similarity threshold, they are not necessarily completely the same subtitles, which indicates that there may be text recognition errors in the two subtitles, that is, there is a wrong word. Therefore, before deduplication of the second subtitle set, the subtitles with wrong words in the second subtitle set can be corrected.

[0120] In order to improve the efficiency of subtitle error correction, the similarity between any two subtitles in the second subtitle set may be used to determine the subtitles with typos, and the subtitles with typos may be corrected.

[0121] Exemplarily, before deduplication of the second subtitle set, the method further includes:

[0122] When the similarity between the first subtitle text and the second subtitle text in the second subtitle set is greater than the first similarity threshold and less than the second similarity threshold, text error correction is performed on the first subtitle text and the second subtitle text.

[0123] In this example, the timestamp of subtitle switching is used to divide subtitle sets belonging to the same subtitle content, and subtitles that need text error correction are determined based on the similarity between subtitles in each subtitle set, improving the efficiency of subtitle error correction.

[0124] Exemplarily, the above similarity can be calculated using the edit distance to measure the similarity between two texts. The edit distance, also known as the Levenshtein distance, is a quantitative measurement of the difference between two strings (such as English words). The measurement method is to determine how many operations are required at least to transform one string into another string.

[0125] Exemplarily, the first similarity threshold can be 90%, and the second similarity threshold can be 100%.

[0126] Exemplarily, the first similarity threshold can be 95%, and the second similarity threshold can be 99.9%.

[0127] In some embodiments, misspelled words in the subtitle text for which text error correction is performed can be masked, and based on the masked subtitle text and the masked words, target predicted words are determined, and then the masked words in the subtitle text are corrected using the target predicted words.

[0128] Exemplarily, the above text error correction for the first subtitle text and the second subtitle text includes:

[0129] In the first subtitle text and the second subtitle text, words with the same position but different words are determined as masked words;

[0130] In the first subtitle text and the second subtitle text, the masked words are masked to obtain a third subtitle text;

[0131] Based on the third subtitle text and the masked words, target predicted words are determined;

[0132] Based on the target predicted words, the masked words in the first subtitle text and the second subtitle text are corrected.

[0133] In this example, by using the subtitle text and the masked words obtained after masking misspelled words to determine the target predicted words, the correct words can be accurately obtained.

[0134] Exemplarily, the above first subtitle text and second subtitle text can be subtitle texts of adjacent two frames.

[0135] Exemplarily, assume the first subtitle text is "The pedestrian flow at the current intersection is very large. Still need to be a little careful.", and the second subtitle text is "The incoming flow at the current intersection is very large. Still need to be a little careful.", then "人" (person) and "入" (incoming) are two words with the same position but different pronunciations, which are the masked words.

[0136] Exemplarily, the target prediction word(s) may include one or more.

[0137] Exemplarily, semantic analysis may be performed on the third subtitle text to obtain a set of prediction words and the confidence of each prediction word in each set of prediction words, and then the target prediction word is determined by using the confidence of each prediction word and the relationship between each prediction word and the masking word.

[0138] Exemplarily, the pre-trained language model BERT (Bidirectional Encoder Representation from Transformers) in natural language processing may be used to perform semantic analysis on the third subtitle text to obtain a set of prediction words and the confidence of each prediction word in each set of prediction words.

[0139] In practical applications, the BERT model is called a MASK (masking) language model, that is, its main task in the training stage is to predict the masked elements. For example, before inputting the text into the BERT model, the suspected incorrect characters are masked. For example, masking the above "人" and "入", the third subtitle text becomes "The current [MASK] flow is very large, still be careful". Then, the BERT model is used to predict what the masked word is. The set of candidate words at the MASK positions predicted by BERT and the confidence of each word.

[0140] In some embodiments, the target prediction word may be determined by combining the confidence of each prediction word and the edit distance between each prediction word and the masking word.

[0141] Exemplarily, determining the target prediction word based on the third subtitle text and the masking word includes:

[0142] Performing semantic analysis on the third subtitle text to obtain a set of prediction words and the confidence of each prediction word in the set of prediction words;

[0143] Determining the quality score of each prediction word based on the confidence of each prediction word and the edit distance between each prediction word and the masking word;

[0144] Determining the target prediction word in the set of prediction words based on the quality score of each prediction word.

[0145] In this example, by combining the confidence of each prediction word and the edit distance between each prediction word and the masking word to determine the target prediction word among multiple prediction words, the prediction accuracy can be further improved.

[0146] For example, the predicted word and the confidence of the predicted word can be predicted by the model. The confidence refers to the credibility of the predicted word being the real text of the masked position in the third subtitle text.

[0147] Exemplarily, the semantic analysis of the third subtitle text may adopt a BERT model.

[0148] For example, assume that the characters predicted by the BERT module at the MASK position are: "car" and "person", where the confidence p of "car" is 0.8123, and the edit distance s between it and the masked word "entry" is 0.012, the confidence p of "person" is 0.2313, and the edit distance s between it and the IDS of "entry" is 0.9342. Then, the confidence p of "car" and the edit distance are summed, and the quality score of "car" is 0.8243, and the quality score of "person" is 1.1655. Therefore, the predicted word "person" can be selected as the target predicted word.

[0149] In some embodiments, whether to modify the target predicted word may be determined based on the quality score of the target predicted word and the edit distance between the target predicted word and the blocked word to avoid erroneous modification.

[0150] Exemplarily, based on the target predicted word, the blocked words in the first subtitle text and the second subtitle text are corrected, including:

[0151] When the quality score of the target prediction word is greater than a set quality score threshold and the edit distance between the target prediction word and the blocked word is greater than a set edit distance threshold, the blocked word in the first subtitle text and the second subtitle text is modified into the target prediction word.

[0152] Exemplarily, when the quality score of the target prediction word is less than a set quality score threshold or the edit distance between the target prediction word and the blocked word is less than a set edit distance threshold, the blocked word in the first subtitle text and the second subtitle text is not modified.

[0153] In this example, whether to modify is determined based on the quality score of the target prediction word and the edit distance between the target prediction word and the blocked word, thereby avoiding erroneous modification.

[0154] Exemplarily, when the quality score of the target prediction word is greater than 1 and the edit distance between the target prediction word and the blocked word is greater than 0.5, the blocked word is modified, otherwise no modification is made and the subtitle text remains unchanged.

[0155] In some embodiments, after performing text error correction on the second subtitle set, the second subtitle set may be deduplicated to obtain target subtitles.

[0156] Exemplarily, the above-mentioned deduplication of the second subtitle set to obtain the target subtitles may include:

[0157] When the similarity between any two subtitle texts in the second subtitle set is greater than the third similarity threshold, the second subtitle set is deduplicated to obtain target subtitles.

[0158] In this example, if the similarity between any two subtitle texts in the second subtitle set is greater than the third similarity threshold, it means that the subtitles in the second subtitle set are all the same subtitles and there are no typos. At this time, the second subtitle set can be deduplicated to improve the accuracy of obtaining the target subtitles.

[0159] Exemplarily, the third similarity threshold may be greater than or equal to the aforementioned second similarity threshold.

[0160] Exemplarily, the third similarity threshold may be 99% or 100%, etc.

[0161] In some embodiments, after obtaining a combination pair of text and audio through the above steps, the text in the combination pair can be verified using the audio in the combination pair to determine whether the combination pair can be used as a training sample for the speech recognition model, thereby improving the recognition accuracy of the speech recognition model trained using the training sample.

[0162] Exemplarily, after obtaining the text-audio combination pair, the method further includes:

[0163] Perform speech recognition on the target audio based on the speech recognition model to obtain the verification text corresponding to the target subtitle;

[0164] Determining the confidence of the combined pair based on the target subtitle and its corresponding verification text;

[0165] When the confidence of the combination pair meets the set confidence threshold, the combination pair is determined as a training sample of the speech recognition model.

[0166] In this example, a pre-trained speech recognition model is used to perform speech recognition on the target audio, and the recognized verification text is used to verify the target subtitles to obtain the confidence of the combined pair consisting of the target audio and the target subtitles. Based on the confidence, it is determined whether to determine the combined pair as a training sample, which can improve the training accuracy of the speech recognition model.

[0167] Exemplarily, the confidence of the combination pair may be determined using the edit distance between the target subtitle and its verification text, the character length of the target subtitle, and the character length of the verification text.

[0168] For example, assuming that the result recognized by the speech recognition model ASR is string1, and the subtitle result obtained by the text recognition OCR is string2, the confidence C of the training sample can be expressed as:

[0169] C=1-EditDistance(string1,string2) / max(len(string1),len(string2))

[0170] Among them, EditDistance(string1, string2) represents the edit distance between string1 and string2, len(string1) is the length of string1, len(string2) is the length of string2, and the max() function is the maximum value function.

[0171] When the ASR recognition result is highly consistent with the OCR recognition result, that is, string1 and string2 are highly consistent, the confidence C value of the training sample is high. Conversely, when the consistency between the two is low, the confidence C value is low. Therefore, the confidence C can be used to filter out samples with low annotation quality.

[0172] Exemplarily, the above training samples can be applied to the training of any speech recognition model, wherein the target audio is used as the input of the speech recognition model, and the target subtitles are used as the annotations of the target audio, which are compared with the output of the speech recognition model, and the parameters of the speech recognition model are adjusted according to the comparison results.

[0173] In some embodiments, the training samples generated above cannot be directly applied to model training. Some numbers or function symbols may appear in the text of the training samples, which need to be regularized.

[0174] Exemplarily, after obtaining the training sample, the method further includes:

[0175] When the target characters exist in the target subtitles, the target subtitles are regularized.

[0176] In this example, the target subtitles containing the target characters are regularized, and the obtained training samples can be applied to model training.

[0177] For example, the target characters may include numbers, function symbols, formulas, time dates, units, etc. The target characters are converted into corresponding Chinese characters. This can generally be achieved through methods such as dictionary mapping and regular matching.

[0178] For example, this piece of gold weighs 324.75 grams, and after regularization it becomes: This piece of gold weighs three hundred and twenty-four point seven five grams; she was born on August 18, 1986, and her brother was born on March 1, 1995, and after regularization it becomes: She was born on August 18, 1986, and her brother was born on March 1, 1995; There is a 62% probability of rain tomorrow, and after regularization it becomes: There is a 62% probability of rain tomorrow; This is the mobile phone number 8618544139121, and after regularization it becomes: This is the mobile phone number 8618544139121.

[0179] The following describes an application example of building a training sample set for a speech recognition model based on a combination of OCR recognition and ASR recognition, as follows:

[0180] like Figure 5 As shown in the figure, the process of constructing the training sample set of the speech recognition model mainly includes: video key frame extraction, OCR text recognition, watermark and scene non-subtitle text filtering, subtitle switching detection, text error correction and generation, text regularization and sample annotation verification.

[0181] (1) Video key frame extraction

[0182] Traverse each frame in the video to detect text, and compare the previous and next frames by frame difference method to see if they are repeated. If they are repeated, delete the redundant frames, and finally get the key frame containing subtitles. The method to determine whether the previous and next frames are repeated is to compare the similarity of the text strings extracted from the two frames. The similarity can be calculated by editing distance, etc. The text detection and recognition model for image frames can be implemented using the RCNN deep learning model. In addition, key frames can be quickly extracted based on FPS (frames per second). For example, when the FPS is 24, 3 frames can be extracted as key frames per second, and a key frame can be extracted every 8 frames to obtain a key frame set. This extraction method is faster. The specific method of extracting key frames can be determined according to the characteristics of the video data or resource conditions.

[0183] (2)OCR text recognition

[0184] After obtaining the key frames of the video file, OCR technology can be used to identify the text information of the text block and the position information of each text block in the key frame from the key frame. In general, the result obtained by OCR is [(text1, (xmin, xmax, ymin, ymax)), (text2, (xmin, xmax, ymin, ymax)), ...]. Among them, text1 and text2 are the recognized text information, xmin, xmax represent the minimum horizontal coordinate and maximum horizontal coordinate of the text block in the key frame, ymin, ymax represent the minimum vertical coordinate and maximum vertical coordinate of the text block in the key frame, as shown in the figure. Figure 4as shown

[0185] (3) Non-subtitle text filtering of watermarks and backgrounds

[0186] In some TV dramas and variety shows, there may be background information such as logos or advertisements. For example Figure 4 "XXTV Drama Channel" in [] is a logo. These background information often appear in fixed positions in the video and exist throughout the entire playback period of the video file. Therefore, by counting the frequency of the coordinate information of all texts in the OCR recognition results, these background contents can be filtered out through a frequency threshold to avoid misidentifying this information as subtitle information. It should be noted that there will be slight differences in the position coordinate information of the same text recognized by OCR in different frame images. For example, in Figure 3, the position coordinates of "Study hard and make progress every day" recognized in the front and back two frames may be (125, 512, 678, 780) and (126, 513, 679, 781). Therefore, their coordinates need to be "normalized". That is, when the coordinate difference between two text blocks in two different frames is less than a predetermined difference threshold, they can be regarded as the same coordinate. For example, when the coordinate difference between text block (x1, y1) and text block (x2, y2) is ||x1 - x2|| < a, ||y1 - y2|| < b, the positions of the two text blocks are considered the same. Where a and b are predetermined thresholds and can be set according to the actual situation. For example, a is 50 pixel values and b is 100 pixel values. After filtering out the watermark and background contents, the remaining texts are sorted according to the appearance frequency, and the coordinate area of the text with the highest appearance frequency is determined as the subtitle area, which can be denoted as (Xmin, Xmax, Ymin, Ymax). The text located in this area, or the area where the text is located matches the subtitle area, then the text is determined as a subtitle.

[0187] (4) Subtitle switching detection, text error correction and generation

[0188] As Figure 6 shown, after obtaining the subtitles of the video, we need to merge the text blocks in the same frame, such as in the case of double-line subtitles. At the same time, in the case of key frame redundancy, there will be a problem of repeated subtitles, and the repeated subtitles need to be deleted.

[0189] As Figure 7 shown, in the subtitle position verification module, in the subtitles obtained by text recognition, there may be problems of matching and adaptation due to multi-line subtitles or insufficient background filtering. This situation often exists in videos produced by self-media. As Figure 3A and Figure 3BAs shown in the figure, it is a scene when a teacher is teaching. There are background subtitles in the video, and there are multiple lines of subtitles. If the subtitle position obtained above is used to verify the current text position, there will be a situation where the subtitle position does not match the text position, that is, the text position area (Xminc, Xmaxc, Yminc, Ymaxc) is larger than the subtitle position, for example, Yminc<Ymin,Ymaxc> Ymax, in this case, if the subtitles are determined only based on the subtitle position, the subtitle content will be lost, but the text information outside the subtitle area cannot be directly regarded as subtitles by default. Therefore, the location area of ​​the current text can be marked, and the corresponding audio can be obtained using the timestamp of the text switching of the current text, and then the audio can be recognized by ASR to obtain the verification text, and the verification text can be used to recall the text currently recognized by OCR. The specific recall method is:

[0190] Calculate the overlap between the OCR recognized text line string(ocr) and the ASR recognition result string(asr): m = len(string(ocr)&string(asr)) / len(string(ocr);

[0191] Among them, len() refers to the function of obtaining the character length, and string(ocr)&string(asr) refers to the intersection of the characters string(ocr) and string(asr);

[0192] When the overlap degree is greater than a set overlap degree threshold, the OCR recognized text line is determined as subtitle content.

[0193] like Figure 6 As shown in FIG. 1 , after obtaining all subtitles of a video file, all subtitles of the same frame are merged. Then, Figure 7 Subtitle switching detection shown.

[0194] This application example uses the similarity of subtitle information between different frames to detect subtitle switching. Figure 7 As shown, the subtitle lines are traversed in sequence, the starting frame frame_s of the current frame number is recorded, and the subtitle similarity between the subtitle of the current frame and the subtitle of the next frame is compared. When the subtitle similarity of adjacent frames is greater than a preset threshold (such as 0.9), it is assumed that the subtitle content of the two frames is the same, and the next frame is skipped, and the comparison is continued with the next frame until different subtitle frames are compared, and the current frame number is recorded as the end frame number frame_e of the subtitle, and so on, until all subtitle lines are traversed. Thus, the timestamps of the subtitle switching of the current subtitle are obtained: frame_s and frame_e.

[0195] The above-mentioned similarity scheme can be calculated using the edit distance of strings. For example, the similarity can be compared based on the edit distance of IDS (Ideographic Description Sequence). IDS only represents ideograms in the form of a sequence, similar to representing English words with a sequence of letters. Chinese characters can be represented by strokes and the structure of Chinese characters. As shown in Figure 8 the IDS representation of the character "pín" (poor). This representation method is more powerful and robust than simply representing similar-looking characters.

[0196] If, during subtitle switching detection, the similarity of subtitles in adjacent frames is greater than a predetermined threshold but they are not exactly the same, it indicates that there may be an OCR recognition error in the current subtitle. For example, the recognition results of two adjacent frames are "The pedestrian flow at the current intersection is very large. Still, be careful." and "The incoming flow at the current intersection is very large. Still, be careful." In this example, OCR recognized "人" (person) as "入" (incoming). When performing text recognition, OCR will give the recognition result of the text, the position information of the text, and the confidence of the text. Therefore, the confidence in the OCR recognition result can also be used to determine whether the current character is incorrect and needs to be modified.

[0197] Therefore, this application example considers correcting the text of the subtitle content with errors. The pre-trained language model BERT in natural language processing can be used for correction. BERT is called a MASK (masking) language model, that is, its main task during the training phase is to predict the masked elements. For example, when the text is input into the BERT model, the suspected incorrect character is masked: "The current [MASK] flow is very large. Still, be careful.", and then the BERT model is used to predict what the masked word is. Based on the set of candidate words at the masked position predicted by BERT, the confidence p of each word, and the IDS edit distance s between the candidate word and the masked character, determine what the correct word at the final masked position is. For example, the words predicted by BERT at the masked position are:

[0198] 1) "车" (car), the confidence p is 0.8123, and the IDS edit distance s from "入" (incoming) is 0.012;

[0199] 2) "人" (person), the confidence p is 0.2313, and the IDS edit distance s from "入" (incoming) is 0.9342;

[0200] The final scoring rule is that the candidate word with the highest value of Score = p + s is the final candidate word.

[0201] In some embodiments, additional constraints may be added to improve accuracy and avoid erroneous modification. For example, the original character is modified only when the final similarity s is greater than 0.5 and the Score value is greater than 1, otherwise it remains unchanged.

[0202] (5) Text regularization and sample annotation verification

[0203] The subtitle information of the final generated video file is a subtitle file in srt format, which contains time code + subtitle information, such as Fig. 9 shown.

[0204] The original audio file can be extracted from the original video through audio and video processing tools such as ffmpeg. The audio can be split out by combining the subtitle file, and the audio and subtitles can be matched and composed.<text,audio> Text-audio pairs are the training samples of text and speech required for speech recognition training.

[0205] However, the data generated by the above process cannot be directly used for ASR model training, and the text needs to be regularized. Text regularization refers to expressing the Elieber numbers, time dates, units, etc. that appear in the text with Chinese characters. It can generally be achieved through dictionary mapping and regular matching, as shown below:

[0206] 1) This piece of gold weighs 324.75 grams -> Regularization -> This piece of gold weighs 324.75 grams;

[0207] 2) She was born on August 18, 1986, and her brother was born on March 1, 1995 --> Regularization --> She was born on August 18, 1986, and her brother was born on March 1, 1995;

[0208] 3) There is a 62% chance of rain tomorrow -> Regularization -> There is a 62% chance of rain tomorrow;

[0209] 4) This is the mobile phone 8618544139121——>Regularization——> This is the mobile phone 8618544139121.

[0210] In addition, to ensure the quality of the training samples, the pre-trained ASR model can be used to verify the OCR-annotated text in the training samples. Assuming that the audio recognition result obtained by ASR is string1, and the subtitle result recognized by OCR is string2, the confidence of the annotation of this data sample can be expressed as:

[0211] C=1-EditDistance(string1,string2) / max(len(string1),len(string2))

[0212] When the ASR recognition result and the OCR recognition result are highly consistent, string1 and string2 are highly consistent, and the confidence C is high. Conversely, when the ASR recognition result and the OCR recognition result are not consistent, the confidence C is low. Therefore, the confidence C value can be used to filter out training samples with low annotation quality.

[0213] Fig.10 4 is a block diagram of an audio and text combination device according to an embodiment of the present disclosure.

[0214] like Fig.10 As shown, the embodiment of the present disclosure provides an audio and text combination device, including:

[0215] A text recognition module 110 is used to perform text recognition on key frames in a video file to obtain a first subtitle set;

[0216] A subtitle switching detection module 120, configured to perform subtitle switching detection on the first subtitle set, and obtain a timestamp of a subtitle switching between a target subtitle and the target subtitle;

[0217] A target audio interception module 130, configured to intercept audio from the audio file of the video file based on the timestamp to obtain target audio;

[0218] The audio-text combination module 140 is used to construct a combination pair of text and audio based on the target subtitles and the target audio.

[0219] In some possible implementations, the text recognition module 110 may include:

[0220] A candidate subtitle recognition unit is used to perform text recognition on key frames in a video file to obtain candidate subtitle texts;

[0221] A verification audio acquisition unit, used to determine the verification audio corresponding to the candidate subtitle text in the video file;

[0222] A subtitle verification unit, configured to determine that the candidate subtitle text is the subtitle text of the video file if the relationship between the candidate subtitle text and the verification audio meets a set condition;

[0223] The first subtitle set acquisition unit is used to acquire the subtitle text of the video file to obtain a first subtitle set.

[0224] Exemplarily, the subtitle verification unit is specifically used for:

[0225] Performing speech recognition on the verification audio to obtain a verification text;

[0226] When the degree of overlap between the candidate subtitle text and the verification text is greater than a set threshold of overlap, the candidate subtitle text is determined to be the subtitle text of the video file.

[0227] In some possible implementations, the candidate subtitle recognition unit is specifically used to:

[0228] Performing text recognition on each key frame in the video file to obtain a first text set;

[0229] Based on the first text set, determining non-subtitle text;

[0230] Filtering the non-subtitle text in the first text set to obtain a second text set;

[0231] Based on the second text set, candidate subtitle texts are determined.

[0232] In some possible implementations, determining the non-subtitle text based on the first text set includes:

[0233] Based on the position information of each text in the first text block set, determining the occurrence frequency of texts in the same position;

[0234] When the occurrence frequency is greater than a set frequency threshold, the text at the same position is determined as non-subtitle text.

[0235] In some possible implementations, determining candidate subtitle texts based on the second text set includes:

[0236] Determining a first subtitle area based on position information of each text in the second text set;

[0237] For each text in the second text set, when the matching degree between the position information of the text and the first subtitle area is less than a set matching threshold, the text is determined to be a candidate subtitle text.

[0238] In some possible implementations, the determining of candidate subtitle texts based on the second text set further includes:

[0239] For each text in the second text set, when the matching degree between the position information of the text and the first subtitle area is greater than a set matching threshold, the text is determined to be a subtitle text of the video file.

[0240] In some possible implementations, the candidate subtitle recognition unit is specifically used to:

[0241] Performing text recognition on each key frame in the video file to obtain a third text set;

[0242] Determining a second subtitle area based on the position information of each text in the third text set;

[0243] For each text in the third text set, if the matching degree between the position information of the text and the second subtitle area is greater than a set matching threshold, determining that the text is a subtitle text of the video file;

[0244] The subtitle text of the video text is obtained to obtain a first subtitle set.

[0245] In some possible implementations, the subtitle switching detection module 120 includes:

[0246] a subtitle division unit, configured to determine a timestamp of subtitle switching based on a similarity between any two subtitle texts belonging to adjacent key frames in the first subtitle set, and divide the first subtitle set using the timestamp to obtain a plurality of second subtitle sets;

[0247] a subtitle deduplication unit, configured to dedupe the second subtitle set to obtain target subtitles;

[0248] The timestamp determining unit is used to determine the timestamp of the target subtitle when the subtitle is switched based on the timestamp of the second subtitle set.

[0249] In some possible implementations, the subtitle switching detection module 120 further includes:

[0250] The subtitle correction unit is used to perform text correction on the first subtitle text and the second subtitle text in the second subtitle set before deduplication is performed on the second subtitle set, when the similarity between the first subtitle text and the second subtitle text in the second subtitle set is greater than a first similarity threshold and less than a second similarity threshold.

[0251] In some possible implementations, the subtitle error correction unit is specifically used to:

[0252] In the first subtitle text and the second subtitle text, words with the same position and different words are determined as blocked words;

[0253] In the first subtitle text and the second subtitle text, the blocked words are blocked to obtain a third subtitle text;

[0254] Determining a target predicted word based on the third subtitle text and the blocked word;

[0255] Based on the target predicted words, blocked words in the first subtitle text and the second subtitle text are modified.

[0256] In some possible implementations, the subtitle deduplication unit is specifically used to:

[0257] When the similarity between any two subtitle texts in the second subtitle set is greater than a third similarity threshold, the second subtitle set is deduplicated to obtain target subtitles.

[0258] In some possible implementations, the above device further includes:

[0259] A verification text acquisition module, used to perform speech recognition on the target audio based on the speech recognition model to obtain a verification text corresponding to the target subtitle;

[0260] A confidence determination module, configured to determine the confidence of the combination pair based on the target subtitle and its corresponding verification text;

[0261] The training sample determination module is used to determine that the combination pair is a training sample of the speech recognition model when the confidence of the combination pair meets a set confidence threshold.

[0262] In some possible implementations, the above device further includes:

[0263] The regularization processing module is used to perform regularization processing on the target subtitle when the target subtitle has target characters.

[0264] The functions of each unit, module or sub-module in each device of the embodiments of the present disclosure can be found in the corresponding description in the above method embodiments, and will not be repeated here.

[0265] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0266] Fig.11 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0267] like Fig.11As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0268] Multiple components in the electronic device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the electronic device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0269] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as the audio and text combination method. For example, in some embodiments, the audio and text combination method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 800 via ROM 102 and / or a communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the audio and text combination method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the audio and text combination method in any other appropriate manner (e.g., by means of firmware).

[0270] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0271] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable atmosphere light adjustment device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0272] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0273] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0274] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0275] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0276] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0277] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for combining audio and text, characterized in that: include: Performing text recognition on key frames in the video file to obtain a first subtitle set; Performing subtitle switching detection on the first subtitle set to obtain a timestamp of a subtitle switching between a target subtitle and the target subtitle; Based on the timestamp, extracting audio from the audio file of the video file to obtain target audio; Based on the target subtitles and the target audio, construct a combined pair of text and audio; Determining a timestamp of subtitle switching based on the similarity between any two subtitle texts belonging to adjacent key frames in the first subtitle set, and dividing the first subtitle set by using the timestamp to obtain a plurality of second subtitle sets; Deduplication and error correction are performed on the second subtitle set to obtain target subtitles; Before deduplication of the second subtitle set, the method further includes, when the similarity between the first subtitle text and the second subtitle text in the second subtitle set is greater than a first similarity threshold and less than a second similarity threshold, performing text error correction on the first subtitle text and the second subtitle text; The text error correction is performed using the pre-trained language model BERT in natural language processing, including: In the first subtitle text and the second subtitle text, words with the same position and different words are determined as blocked words; In the first subtitle text and the second subtitle text, the blocked words are blocked to obtain a third subtitle text; Determining a target predicted word based on the third subtitle text and the blocked word; When the quality score of the target predicted word is greater than a set quality score threshold, and the edit distance between the target predicted word and the blocked word is greater than a set edit distance threshold, the blocked word in the first subtitle text and the second subtitle text is modified to the target predicted word; and based on the timestamp of the second subtitle set, the timestamp of the subtitle switching of the target subtitle is determined.

2. The method according to claim 1, characterized in that The step of performing text recognition on key frames in the video file to obtain a first subtitle set includes: Perform text recognition on key frames in the video file to obtain candidate subtitle texts; In the video file, determining the verification audio corresponding to the candidate subtitle text; In the case where the relationship between the candidate subtitle text and the verification audio meets a set condition, determining that the candidate subtitle text is the subtitle text of the video file; The subtitle text of the video file is obtained to obtain a first subtitle set.

3. The method according to claim 2, characterized in that The determining that the candidate subtitle text is the subtitle text of the video file when the relationship between the candidate subtitle text and the verification audio meets a set condition comprises: Performing speech recognition on the verification audio to obtain a verification text; When the degree of overlap between the candidate subtitle text and the verification text is greater than a set threshold of overlap, the candidate subtitle text is determined to be the subtitle text of the video file.

4. The method according to claim 2, characterized in that: The step of performing text recognition on key frames in the video file to obtain candidate subtitle texts includes: Performing text recognition on key frames in the video file to obtain a first text set; Based on the first text set, determining non-subtitle text; Filtering the non-subtitle text in the first text set to obtain a second text set; Based on the second text set, candidate subtitle texts are determined.

5. The method according to claim 4, characterized in that The step of determining the non-subtitle text based on the first text set includes: Based on the position information of each text in the first text set, determining the occurrence frequency of texts with the same position; When the occurrence frequency is greater than a set frequency threshold, the text at the same position is determined as non-subtitle text.

6. The method according to claim 4, characterized in that The step of determining candidate subtitle texts based on the second text set includes: Determining a first subtitle area based on position information of text in the second text set; For a text in the second text set, when the matching degree between the position information of the text and the first subtitle area is less than a set matching threshold, the text is determined to be a candidate subtitle text.

7. The method according to claim 6, characterized in that Also includes: For the text in the second text set, when the matching degree between the position information of the text and the first subtitle area is greater than a set matching threshold, the text is determined to be the subtitle text of the video file.

8. The method according to claim 1, characterized in that The performing text recognition on the key frames of the video file to obtain the first subtitle set comprises: Performing text recognition on key frames in the video file to obtain a third text set; Determining a second subtitle area based on position information of text in the third text set; For the text in the third text set, if the matching degree between the position information of the text and the second subtitle area is greater than a set matching threshold, determining that the text is the subtitle text of the video file; The subtitle text of the video file is obtained to obtain a first subtitle set.

9. The method according to claim 1, characterized in that: The step of removing duplicates from the second subtitle set to obtain target subtitles includes: When the similarity between any two subtitle texts in the second subtitle set is greater than a third similarity threshold, the second subtitle set is deduplicated to obtain target subtitles.

10. The method according to claim 1, characterized in that Also includes: Performing speech recognition on the target audio based on a speech recognition model to obtain a verification text corresponding to the target subtitle; Determining the confidence of the combined pair based on the target subtitle and its corresponding verification text; When the confidence of the combination pair meets a set confidence threshold, the combination pair is determined to be a training sample of a speech recognition model.

11. The method according to claim 1, characterized in that: Also includes: In the case that the target subtitle has target characters, regularization processing is performed on the target subtitle.

12. An audio and text combination device, characterized in that: include: A text recognition module, used for performing text recognition on key frames in a video file to obtain a first subtitle set; A subtitle switching detection module, used to perform subtitle switching detection on the first subtitle set, and obtain a timestamp of a subtitle switching between a target subtitle and the target subtitle; A target audio interception module, used to intercept audio from the audio file of the video file based on the timestamp to obtain target audio; An audio-text combination module, used to construct a combination pair of text and audio based on the target subtitle and the target audio; Determining a timestamp of subtitle switching based on the similarity between any two subtitle texts belonging to adjacent key frames in the first subtitle set, and dividing the first subtitle set by using the timestamp to obtain a plurality of second subtitle sets; Deduplication and error correction are performed on the second subtitle set to obtain target subtitles; Before deduplication of the second subtitle set, the method further includes, when the similarity between the first subtitle text and the second subtitle text in the second subtitle set is greater than a first similarity threshold and less than a second similarity threshold, performing text error correction on the first subtitle text and the second subtitle text; The text error correction is performed using the pre-trained language model BERT in natural language processing, including: In the first subtitle text and the second subtitle text, words with the same position and different words are determined as blocked words; In the first subtitle text and the second subtitle text, the blocked words are blocked to obtain a third subtitle text; Determining a target predicted word based on the third subtitle text and the blocked word; When the quality score of the target predicted word is greater than a set quality score threshold, and the edit distance between the target predicted word and the blocked word is greater than a set edit distance threshold, the blocked word in the first subtitle text and the second subtitle text is modified to the target predicted word; and based on the timestamp of the second subtitle set, the timestamp of the subtitle switching of the target subtitle is determined.

13. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 11.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method of any one of claims 1 to 11.

15. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 11.

Images