Method, device and computer equipment for obtaining speech recognition training data

By recognizing video subtitles and performing speech boundary detection and segmentation, and matching subtitles and audio segments, the problem of low efficiency in acquiring training data in traditional speech recognition is solved, and efficient and accurate training data acquisition is achieved.

CN117012179BActive Publication Date: 2026-11-10TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202210760992.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-06-30
Publication Date
2026-11-10
Estimated Expiration
2042-06-30

AI Technical Summary

Technical Problem

Traditional speech recognition training data acquisition is inefficient, requiring a large amount of manual annotation, which leads to low efficiency.

Method used

By identifying subtitle content and time intervals in the video, speech boundary detection is performed to segment the video, and subtitles and audio segments are matched to perform speech recognition to obtain training samples.

Benefits of technology

It improves the accuracy and efficiency of acquiring speech training samples, eliminates the audio annotation process, and directly utilizes video subtitles and audio for processing, quickly obtaining a large amount of training data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117012179B_ABST
    Figure CN117012179B_ABST
Patent Text Reader

Abstract

The application relates to a method and device for obtaining speech recognition training data, a computer device, a storage medium and a computer program product. The method comprises the following steps: recognizing subtitles of each video frame in a target video to obtain subtitle content of multiple effective subtitles of the target video and a time interval of each piece of subtitle content; performing speech boundary detection on the target video, splitting the target video according to the detected speech boundary to obtain multiple audio segments of the target video and a time interval of each audio segment; determining matched target subtitle content and a target audio segment according to the time interval of the subtitle content and the time interval of the audio segment; performing speech recognition on the target audio segment to obtain a speech recognition result; and if the speech recognition result matches the target subtitle content, obtaining a speech training sample according to the target subtitle content and the target audio segment. The method improves the accuracy and efficiency of speech training sample extraction.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of speech recognition technology, and in particular to a method, apparatus, computer device, storage medium, and computer program product for acquiring speech recognition training data. Background Technology

[0002] With the development of speech recognition technology, it has been widely applied in various fields such as smart homes, automotive electronics, and consumer electronics. As the application scope of speech recognition technology expands, the demand for speech recognition accuracy also increases. Among these, the key to improving speech recognition accuracy is to improve the recognition performance of the speech recognition model, which requires a large amount of accurate training data to be provided for training the speech recognition model.

[0003] Traditionally, language experts are required to manually listen to and annotate each sentence to generate text-to-speech matching data for training the speech recognition model. This method of obtaining training samples is inefficient. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, apparatus, computer device, computer-readable storage medium, and computer program product for acquiring speech recognition training data that can improve the acquisition efficiency in response to the above-mentioned technical problems.

[0005] Firstly, this application provides a method for acquiring speech recognition training data. The method includes:

[0006] Identify the subtitles of each video frame in the target video to obtain the subtitle content of multiple valid subtitles in the target video and the time interval of each subtitle content;

[0007] The target video is subjected to speech boundary detection, and the target video is segmented according to the detected speech boundaries to obtain multiple audio segments of the target video and the time interval of each audio segment;

[0008] Based on the time interval of the subtitle content and the time interval of the audio segment, determine the matching target subtitle content and target audio segment;

[0009] The target audio segment is subjected to speech recognition to obtain the speech recognition result;

[0010] If the speech recognition result matches the target subtitle content, then a speech training sample is obtained based on the target subtitle content and the target audio segment.

[0011] Secondly, this application also provides an apparatus for acquiring speech recognition training data, the apparatus comprising:

[0012] The subtitle processing module is used to identify the subtitles of each video frame in the target video, and obtain the subtitle content of multiple valid subtitles in the target video and the time interval of each subtitle content;

[0013] The speech processing module is used to perform speech boundary detection on the target video, segment the target video according to the detected speech boundaries, and obtain multiple audio segments of the target video and the time interval of each audio segment.

[0014] The matching module is used to determine the target subtitle content and the target audio segment to be matched based on the time interval of the subtitle content and the time interval of the audio segment.

[0015] The recognition module is used to perform speech recognition on the target audio segment and obtain the speech recognition result;

[0016] The processing module is configured to obtain a speech training sample based on the target subtitle content and the target audio segment if the speech recognition result matches the target subtitle content.

[0017] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0018] Identify the subtitles of each video frame in the target video to obtain the subtitle content of multiple valid subtitles in the target video and the time interval of each subtitle content;

[0019] The target video is subjected to speech boundary detection, and the target video is segmented according to the detected speech boundaries to obtain multiple audio segments of the target video and the time interval of each audio segment;

[0020] Based on the time interval of the subtitle content and the time interval of the audio segment, determine the matching target subtitle content and target audio segment;

[0021] The target audio segment is subjected to speech recognition to obtain the speech recognition result;

[0022] If the speech recognition result matches the target subtitle content, then a speech training sample is obtained based on the target subtitle content and the target audio segment.

[0023] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0024] Identify the subtitles of each video frame in the target video to obtain the subtitle content of multiple valid subtitles in the target video and the time interval of each subtitle content;

[0025] The target video is subjected to speech boundary detection, and the target video is segmented according to the detected speech boundaries to obtain multiple audio segments of the target video and the time interval of each audio segment;

[0026] Based on the time interval of the subtitle content and the time interval of the audio segment, determine the matching target subtitle content and target audio segment;

[0027] The target audio segment is subjected to speech recognition to obtain the speech recognition result;

[0028] If the speech recognition result matches the target subtitle content, then a speech training sample is obtained based on the target subtitle content and the target audio segment.

[0029] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0030] Identify the subtitles of each video frame in the target video to obtain the subtitle content of multiple valid subtitles in the target video and the time interval of each subtitle content;

[0031] The target video is subjected to speech boundary detection, and the target video is segmented according to the detected speech boundaries to obtain multiple audio segments of the target video and the time interval of each audio segment;

[0032] Based on the time interval of the subtitle content and the time interval of the audio segment, determine the matching target subtitle content and target audio segment;

[0033] The target audio segment is subjected to speech recognition to obtain the speech recognition result;

[0034] If the speech recognition result matches the target subtitle content, then a speech training sample is obtained based on the target subtitle content and the target audio segment.

[0035] The aforementioned methods, apparatus, computer devices, storage media, and computer program products for acquiring speech recognition training data utilize video subtitles to extract subtitle content, perform boundary detection on the video's speech, segment it into multiple speech segments, and match target subtitle content and target audio segments based on the time intervals of the subtitle content and the audio segments. This ensures the consistency of the target subtitle content and target audio segments along the time axis, and then performs speech recognition on the target audio segments. By verifying the consistency between the target subtitle content and target audio segments from the content perspective, the final determined speech training samples are matched in both the time and content dimensions, improving the accuracy of speech training sample extraction. Furthermore, this method directly utilizes subtitles and audio from the video, and by processing the subtitles and audio, it eliminates the need for audio annotation, enabling the rapid acquisition of large amounts of speech training data and improving the efficiency of speech training data acquisition. Attached Figure Description

[0036] Figure 1 This is an application environment diagram of a method for obtaining speech recognition training data in one embodiment;

[0037] Figure 2 This is a flowchart illustrating a method for obtaining speech recognition training data in one embodiment;

[0038] Figure 3 This is a flowchart illustrating a method for obtaining speech recognition training data in another embodiment;

[0039] Figure 4 This is a schematic diagram illustrating the principle of processing speech segments in one embodiment;

[0040] Figure 5 This is a schematic diagram illustrating the results of training sample processing in one embodiment;

[0041] Figure 6 This is a schematic diagram illustrating the steps for extracting the time interval of subtitle content in one embodiment;

[0042] Figure 7 This is a schematic diagram illustrating the processing of valid subtitles in a valid subtitle area in one embodiment;

[0043] Figure 8 This is a schematic diagram illustrating the relationship between the coordinates of the subtitle area in one embodiment;

[0044] Figure 9 This is a flowchart illustrating a method for obtaining speech recognition training data in one embodiment;

[0045] Figure 10 This is a flowchart illustrating a method for obtaining speech recognition training data in another embodiment;

[0046] Figure 11 This is a structural block diagram of a device for acquiring speech recognition training data in one embodiment;

[0047] Figure 12 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0049] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0050] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0051] The solutions provided in this application relate to artificial intelligence speech technology. Key technologies in speech technology include automatic speech recognition (ASR), text-to-speech (TTS), and voiceprint recognition. Enabling computers to hear, see, speak, and feel is the future direction of human-computer interaction, with speech emerging as one of the most promising methods for future human-computer interaction.

[0052] The method for obtaining speech recognition training data provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, annotation platform 101 communicates with crawling server 102 via a network, and also communicates with training platform 103 via a network. Crawling server 102 crawls target videos from a specified website. Annotation platform 101 identifies subtitles in each video frame of the target video, obtaining the subtitle content and time intervals of multiple valid subtitles. It performs speech boundary detection on the target video, segments the video based on the detected speech boundaries, obtaining multiple audio segments and time intervals of each audio segment. Based on the time intervals of the subtitle content and the audio segments, it determines the matching target subtitle content and target audio segment, performs speech recognition on the target audio segment, and obtains the speech recognition result. If the speech recognition result matches the target subtitle content, then speech training samples are obtained based on the target subtitle content and target audio segment. Training platform 103 uses the speech training samples from the annotation platform to train a speech recognition model.

[0053] The annotation platform 101 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0054] In one embodiment, such as Figure 2 and Figure 3 As shown, a method for obtaining speech recognition training data is provided, which can be applied to... Figure 1 Taking the annotation platform in [the document] as an example, the following steps are included:

[0055] Step 202: Identify the subtitles of each video frame in the target video to obtain the subtitle content of multiple valid subtitles in the target video and the time interval of each subtitle content.

[0056] Among these methods, the target video file can be obtained by crawling a server based on the training objective, or by crawling the video file and then preprocessing it.

[0057] The crawled video files included subtitle files. A large number of video files exist on the internet, such as those in films and television shows with subtitles. The subtitles in these films and television shows are mostly manually proofread and therefore have a certain degree of accuracy. Therefore, by using this type of video file as a basis, subtitle content and audio segments can be extracted, providing a large number of audio training samples without manual annotation.

[0058] In this context, the training target can be the target language, and preprocessing can involve verifying whether the crawled video file is in the target language. Specifically, obtaining the target video includes: crawling the video file from a specified website, extracting at least one frame from the video file, verifying whether the subtitles of the video file are in the target language, extracting audio of a preset duration from the video file, and verifying whether the audio of the video file uses the target language. Specifically, an optical character recognition (OCR) model for the target language is used to verify whether the subtitles of the video file are in the target language. A speech recognition model for the target language is used to verify whether the audio of the video file uses the target language. For example, if the target language is Chinese, then the target video file should be a video file with Chinese pronunciation and Chinese subtitles. Another example: if the target language is language A, then the target video file should be a video file with language A pronunciation and language A subtitles. If the subtitles of the verified video do not use the target language, or the audio of the video does not use the target language, then the video is filtered and no further processing is performed. Therefore, by using the method for obtaining speech recognition training data of this application, the training data required for training the target language model can be obtained.

[0059] After obtaining the target video, the subtitles are extracted. In this embodiment, the subtitles in the target video are external subtitles (i.e., no embedded subtitle file). For each video frame of the target video, an Optical Character Recognition (OCR) method can be used to recognize the subtitles in each video frame. The Optical Character Recognition (OCR) method can convert text and numerical information in an image into text information, that is, achieve image-to-text conversion. It can be understood that for each video frame of the target video, optical character recognition corresponding to the target language is used to recognize the subtitles in each video frame.

[0060] Valid subtitles refer to subtitles that accurately reflect the audio content of the video. Generally, the text content in a target video includes not only the audio but also subtitles such as station logos and advertisements. Furthermore, optical character recognition (OCR) methods may mistakenly extract text from the video scene as subtitles. Therefore, identifying valid subtitles avoids interference from station logos, advertisements, and scene text, accurately extracting subtitle content relevant to the audio content.

[0061] After extracting the valid subtitles, the time range of each subtitle can be determined based on its appearance and disappearance times.

[0062] Step 204: Perform speech boundary detection on the target video, and segment the target video according to the detected speech boundaries to obtain multiple audio segments of the target video and the time interval of each audio segment.

[0063] Speech boundary detection, also known as speech activity detection or speech endpoint detection, aims to identify and eliminate long periods of silence in the audio signal stream. Speech boundary detection can employ Voice Activity Detection (VAD), which uses silence segmentation recognition to segment speech segments from the entire video audio. For example... Figure 4 As shown, VAD is used to detect silence pauses, and the start and end times of each sentence are retained to obtain multiple audio segments and the time interval of each audio segment.

[0064] Step 206: Determine the matching target subtitle content and target audio segment based on the time interval of the subtitle content and the time interval of the audio segment.

[0065] Typical speech recognition needs primarily involve recognizing normal speech conversations, such as wake words for smart home appliances and voice control commands for smart devices. Therefore, from the perspective of speech recognition requirements, speech conversations are preferable as training data. Since videos are usually dialogue-based, extracting speech recognition training data from videos with subtitles yields better training results.

[0066] When creating subtitles, to ensure they match the video content, each subtitle typically represents a single sentence. Depending on the length of the spoken content, a subtitle lasts for several seconds, covering several consecutive frames of video. Therefore, the duration of a subtitle usually matches the duration of the corresponding audio segment. For example, in a film or television drama, a scene depicts a kindergarten enrollment where a parent hands their child to the teacher, and the child says, "Good morning, teacher," to which the teacher responds, "Good morning, Zhuangzhuang." When performing speech boundary detection on this video segment, the pauses in the child's and teacher's dialogue are segmented into two audio segments. From the subtitle content perspective, the child's speech and the teacher's speech are each created as separate subtitles. By matching the time intervals of the subtitle content with the time intervals of the audio segments, it is possible to match the audio segment of the child's speech with the subtitle, and the audio segment of the teacher's speech with the subtitle, respectively.

[0067] The target subtitle content and target audio segment refer to the subtitle content and audio segment that match the time interval. Alternatively, only the subtitle content and audio segment that match the time interval can be extracted as the target subtitle content and target audio segment. Or, the time interval of the audio segment can be used to correct the time interval of the subtitle content to obtain the target subtitle content and target audio segment.

[0068] In this embodiment, by determining the target subtitle content and target audio segment to match, each subtitle content can be matched with an audio segment, and the resulting target subtitle content and target audio segment are subtitles and audio corresponding to each other.

[0069] Step 208: Perform speech recognition on the target audio segment to obtain the speech recognition result.

[0070] To further verify the degree of matching between the target audio segment and the target subtitle content for time interval matching, speech recognition is performed on the target audio segment to obtain the speech recognition result, which is then matched with the subtitle content.

[0071] This can be achieved by using a speech recognition model to perform speech recognition on the target audio segment, thus obtaining the speech recognition result. It can be understood that, for different languages, the corresponding language recognition model is used to perform speech recognition on the target audio segment.

[0072] Step 210: If the speech recognition result matches the target subtitle content, then a speech training sample is obtained based on the target subtitle content and the target audio segment.

[0073] Specifically, the similarity between the speech recognition result and the target subtitle content is compared. If the similarity is within a threshold range, the target subtitle content and the target audio segment are used as speech training samples, with the target subtitle content being the labeled result of the target audio segment. For example, considering that the speech recognition model is still in the improvement stage, the similarity threshold can be set to 20%. If the similarity between the speech recognition result and the target subtitle content is less than 20%, the target subtitle content and the target audio segment are deleted. If the similarity between the speech recognition result and the target subtitle content is greater than 20%, and the speech recognition result matches the target subtitle content, then the subtitle content and the audio content can be considered to match in content, and the target subtitle content and the target audio segment are used as speech training samples. Figure 5 As shown, for each speech training sample, the processing results of the subtitle content and audio segment can be processed into a standard table format of the annotation platform. This standard table records the storage address of the audio segment, the mapping relationship between the subtitle content and the subtitle content, as well as information such as the video frame number, subtitle time interval, and audio segment time interval.

[0074] Using this method, the type of target language can be set, and the corresponding text recognition and speech recognition methods can be used to obtain the training samples required for the target language.

[0075] The aforementioned method for obtaining speech recognition training data utilizes video subtitles to extract subtitle content, performs boundary detection on the video speech to segment it into multiple speech segments, and matches the target subtitle content and target audio segment based on the time intervals of the subtitle content and the audio segment. This ensures the consistency of the target subtitle content and target audio segment on the time axis, and then performs speech recognition on the target audio segment. The consistency between the target subtitle content and target audio segment is verified from the content perspective, ensuring that the final determined speech training samples are matched in both the time and content dimensions, thus improving the accuracy of speech training sample extraction. At the same time, this method directly utilizes the subtitles and audio in the video, and by processing the subtitles and audio, it eliminates the need for audio annotation, enabling the rapid acquisition of a large amount of speech training data and improving the efficiency of speech training data acquisition.

[0076] One approach to matching target subtitle content with target audio segments is to determine the target subtitle content and target audio segment with a time interval overlap within the first range, based on the time intervals of each subtitle content and each audio segment.

[0077] In one embodiment, the subtitle content and audio segments can be sorted by time, and the matching degree of the time intervals of the subtitle content and audio segments can be confirmed sequentially. For example, the time interval of the first subtitle content can be matched with the time interval of the first audio segment, the time interval of the third subtitle content can be matched with the time interval of the third audio segment, and so on, until the time interval of the Nth subtitle content is matched with the time interval of the Nth audio segment. If the overlap between their time intervals is within a first range, then the subtitle content and audio segment can be considered to match. For example, if the time interval of the subtitle content is 1-5 seconds and the time interval of the audio segment is 0.3-4.8 seconds, and the overlap between their time intervals reaches the first range, it means that they match in the time axis dimension. The first time range is set based on experience, such as 90%. If the overlap between the time intervals of the subtitle content and the time intervals of the audio segment is greater than 90%, then they are considered to match.

[0078] In another embodiment, target subtitle content and target audio segments with a time interval overlap within a first range can also be directly searched. Specifically, based on the time interval of the subtitle content, audio segments that intersect with that time interval can be found in the audio segment set. Then, the overlap between the time interval of the subtitle content and the time interval of the audio segments is calculated. If the overlap between their time intervals is within a first range, the time intervals of the subtitle content and the audio segments can be considered to match. Alternatively, based on the time interval of the audio segments, audio-visual subtitle content that intersects with that time interval can be found in the subtitle content set. Then, the overlap between the time interval of the subtitle content and the time interval of the audio segments is calculated. If the overlap between their time intervals is within a first range, the time intervals of the subtitle content and the audio segments can be considered to match.

[0079] For example, if a subtitle's time range is 1-5 seconds, then the audio clip set is searched for audio clips whose time range overlaps with 1-5 seconds. For instance, if an audio clip's time range is 1.2-4.8 seconds, which overlaps with 1-5 seconds, then this audio clip overlaps with the subtitle's time range. Further calculations are made to determine the degree of overlap between the subtitle's time range and the audio clip's time range. If the overlap is within the first range, then the subtitle and audio clip are considered a match.

[0080] If multiple subtitles or audio clips are found, the one with the widest intersection is selected as the search result. For example, in the audio clip set, searching for audio clips whose time interval intersects with 1-5 seconds includes audio clip A (0-1.2 seconds) and audio clip B (1.2-4.8 seconds). Audio clip B has the widest intersection with the subtitle content, so audio clip B is selected as the subtitle content search result.

[0081] This implementation determines the target subtitle content and target audio segment whose time interval deviation is within the allowable range based on the time interval of each subtitle content and the time interval of each audio segment, and can quickly determine the target subtitle content and target audio segment that match on the timeline.

[0082] However, this method only considers target subtitle content with small time interval deviations, which to some extent pre-selects some material, potentially leading to a shortage of training data. In practical applications, the low time interval matching between subtitle content and audio segments is due to subtitle recognition accuracy. Specifically, subtitle recognition accuracy is easily affected by interference from video content and the text recognition model. When an error is detected in the subtitle, it can easily lead to errors in the time interval of the subtitle content.

[0083] In response to this situation, one implementation involves determining the matching target subtitle content and target audio segment time interval based on the time interval of the subtitle content and the time interval of the audio segment. Alternatively, it can be done by: determining the initial subtitle content and target audio segment with a time interval overlap within a second range based on the time interval of each subtitle content and the time interval of each audio segment; correcting the time interval of the initial subtitle content; and obtaining the target subtitle content based on the subtitles within the corrected time interval.

[0084] Specifically, regardless of whether the target subtitle content and target audio segment are matched sequentially, or the target subtitle content and target audio segment are directly searched by time interval, the initial subtitle content and target audio segment with the overlap of time intervals within the second range are determined based on the time intervals of each subtitle content and the time intervals of each audio segment. Then, the time intervals of the initial subtitle content are corrected to obtain the target subtitle content.

[0085] The boundary value of the second range should be smaller than that of the first range. If the overlap between the subtitle content and the audio segment is not within the first range, it can be further determined whether the overlap is within the second range. The second range is determined empirically based on the range of deviation between the subtitle content's time interval and the audio segment's time interval caused by subtitle recognition error; for example, the first range can be set to 90%-100%, and the second range can be set to 70%-90%.

[0086] Taking the method of confirming the matching degree of the time intervals of the subtitle content and the audio segment in sequence as an example, if the overlap of the time intervals of the subtitle content and the audio segment is not in the first range, then it is confirmed whether the overlap of the time intervals of the subtitle content and the audio segment is in the second range. If the overlap of the time intervals of the subtitle content and the audio segment is not in the first range, but in the second range, then the time interval of the initial subtitle content is corrected, and the target subtitle content is obtained based on the subtitles within the corrected time interval.

[0087] In practical applications, segmenting target video using audio boundaries can effectively reflect the characteristics of dialogue and has high accuracy. Therefore, it can be considered that the accuracy of audio segmentation is higher than that of subtitle content segmentation. One implementation method involves using the time interval of the target audio segment as a reference to correct the time interval of the initially selected subtitle content.

[0088] The corrected time of the initial selected subtitle content can be either expanded from the original time range (e.g., the original subtitle content time range is 1-3 seconds, and the corrected subtitle content time range is 0.8-3.8 seconds) or shortened from the original time range (e.g., the original subtitle content time range is 0.8-3.8 seconds, and the corrected subtitle content time range is 1-3 seconds). Correspondingly, subtitles in each video frame within this time range are identified to obtain the target subtitle content.

[0089] In this embodiment, if the segmented audio clips and subtitle content do not match in time interval, the time interval of the subtitle content can be corrected, which can reduce the reduction in the number of samples caused by subtitle recognition errors to a certain extent.

[0090] In this embodiment, the time range of the initially selected subtitle content is corrected using the time range of the target audio segment as a reference. Specifically, the time range of the initially selected subtitle content is corrected based on the time range of the target audio segment to obtain the target subtitle content. The corrected time range of the target subtitle content is consistent with the time range of the target audio segment. For example, if the time range of the initially selected audio segment is 0.8 seconds to 3.8 seconds, and the time range of the initially selected subtitle content is 1 second to 3 seconds, then after correction, the time range of the initially selected subtitle content is corrected to 0.8 seconds to 3.8 seconds.

[0091] In another approach, the time range of the initial subtitle content is corrected using the time range of the target audio segment as a reference. If the time range of the target audio segment is longer than the time range of the initial subtitle content, the time range of the target subtitle content is extended by a preset duration to obtain the corrected time range of the target subtitle. If the time range of the target audio segment is shorter than the time range of the initial subtitle content, the time range of the initial subtitle content is shortened by a preset duration to obtain the corrected time range of the initial subtitle. The corrected time range of the target subtitle content is consistent with or similar to the time range of the target audio segment. For example, if the time range of the target subtitle content is 1-5 seconds and the time range of the initial audio segment is 0.3-8 seconds, extending the time range of the initial subtitle content by 1 second in both directions will result in a corrected time range of 0-6 seconds for the target subtitle content.

[0092] In this embodiment, by using the time interval of the target audio segment as a reference to correct the time interval of the initially selected subtitle content, the time interval of the audio segment and the time interval of the subtitle content can be made comparable.

[0093] Improving the accuracy of subtitle recognition through technical means can further enhance the accuracy of training samples. In one embodiment, subtitles in each video frame of a target video are identified, yielding the content of multiple valid subtitles and the time interval of each subtitle, such as... Figure 6 As shown, it includes:

[0094] Step 602: Identify the subtitle content of each video frame in the target video and determine the subtitle region containing subtitles.

[0095] Specifically, for each video frame in the target video, an optical character recognition (OCR) model for the target language is used to obtain the subtitles in each video frame, as well as the subtitle region where each subtitle is located. The subtitle region specifically refers to the position of the subtitle within the video frame image. For example... Figure 7 As shown, for this video frame, the subtitle area is identified in the lower center of the video frame.

[0096] Step 604: Determine the valid subtitle area. Determine the subtitles in the valid subtitle area of ​​each video frame as valid subtitles.

[0097] This process involves setting rules based on experience to eliminate invalid subtitle areas from each subtitle region, thus obtaining valid subtitle areas. Invalid subtitle areas include those containing the station logo or advertisements. Compared to normal subtitles, these interfering subtitles are usually unchanging; for example, the station logo is typically in the upper left corner of the video frame, and advertisement subtitles are usually displayed for a longer period with less variation to achieve better advertising effectiveness. Based on this, the validity of a subtitle region can be determined by observing changes in the text within that region.

[0098] Other inspection rules can also be set according to actual applications, and there are no restrictions here.

[0099] Once the valid subtitle area is determined, the subtitles within that area are considered valid subtitles. For example... Figure 7 As shown, the subtitle area in the middle of the bottom of the video frame is determined to be the valid subtitle area, and the subtitles in this valid subtitle area are valid subtitles.

[0100] Step 606: Based on the appearance time and disappearance time of the valid subtitles, obtain the subtitle content of multiple valid subtitles and the time interval of each subtitle content.

[0101] Specifically, when a change is detected in the valid subtitle area of ​​a video frame between consecutive frames, such as a change in the valid subtitle area of ​​a later video frame relative to the previous frame, the time of the later frame is determined as the appearance time of the valid subtitle, and the content of the valid subtitle is identified. Then, the subtitles in the valid subtitle areas of each video frame are compared. When a valid subtitle disappears, the disappearance time of the valid subtitle in that frame is determined. By identifying the appearance and disappearance times of valid subtitles in the video, the corresponding time interval is determined as the time interval for valid subtitles.

[0102] In this embodiment, by setting corresponding recognition rules to determine the effective subtitle area, the influence of interfering subtitles on subtitle recognition can be eliminated, and the effective subtitles can be accurately extracted.

[0103] Specifically, the effective subtitle area is determined by identifying the subtitles in the effective subtitle area of ​​each video frame as effective subtitles. This includes: if the subtitle content in the subtitle area does not meet the requirements for effective subtitles, then the subtitle area is removed; in the remaining subtitle areas, the subtitle areas whose horizontal or vertical coordinate positions remain unchanged are taken as effective position subtitle areas; and the subtitles in the effective subtitle areas are determined as effective subtitles.

[0104] The specific requirements for effective subtitles can be determined based on experience and business needs. Typical requirements may include the degree of repetition in the subtitles and the character type of the subtitle content.

[0105] If the subtitle content in a subtitle area does not meet the requirements for valid subtitles, that subtitle area will be removed. For example, if the subtitles in the upper left subtitle area have a high degree of repetition, then the upper left subtitle area will be identified as an invalid subtitle area and removed.

[0106] Within the remaining subtitle area, subtitle regions whose horizontal or vertical coordinates remain unchanged are considered valid subtitle regions. In practical applications, video subtitles typically come in two forms: horizontally placed at the bottom of the screen, or vertically placed on the left or right side. The subtitle position within a video is usually fixed. Therefore, the horizontal or vertical coordinate of the valid subtitle region should remain constant. Thus, defining valid subtitle regions with fixed horizontal or vertical coordinates within the remaining subtitle area allows us to determine the fixed-position valid subtitle regions based on the subtitle's location. It's understandable that a target video should only have one scenario: either the horizontal coordinate remains unchanged, or the vertical coordinate remains unchanged. A fixed vertical coordinate indicates the subtitle is displayed vertically, while a fixed horizontal coordinate indicates the subtitle is displayed horizontally. For example... Figure 8 As shown, the vertical axis of the subtitle area remains unchanged.

[0107] In this embodiment, the rules for valid subtitle content and the rules for displaying subtitle areas are utilized to accurately extract valid subtitles.

[0108] Valid subtitle requirements may include at least one of the following: the subtitle content is not repetitive, and the subtitle content includes characters of the target language. Accordingly, if the subtitle content in a subtitle area does not meet the valid subtitle requirements, the subtitle area is removed. This can be achieved by removing the subtitle area if the subtitle content is repetitive, or by removing the subtitle area if the subtitle content does not include the character encoding of the target language.

[0109] Specifically, if the content of a subtitle area is repeated twice, or if the similarity between two subtitles is more than 80%, the content of the subtitle area is considered not to meet the requirements for a valid subtitle. The subtitle area is then designated as an invalid subtitle area, and the repeated subtitles in the subtitle area are deleted. In this case, the repeated subtitles are usually station logos, drama titles, or advertisements.

[0110] If the subtitle content in a subtitle area is repeated, the subtitle area will be removed. Alternatively, if the subtitle content in a subtitle area does not include the character encoding of the target language, the subtitle area will be removed.

[0111] Specifically, if the subtitle content of a subtitle area does not contain the target language character encoding, then the subtitle area is considered not to be displaying subtitles in the target language, and the subtitle area is deleted. Taking English as an example, if the subtitle content of a subtitle area does not contain any English character Unicode encoding, then the subtitle area is deleted.

[0112] In this embodiment, interfering subtitles are removed from the text content dimension by considering both text repetition and target language text, thereby improving the accuracy of subtitle extraction.

[0113] In another embodiment, after identifying the subtitle content of each video frame in the target video and determining the subtitle region with subtitles, the method further includes: identifying the video frames where the subtitle content changes as keyframes. If the number of keyframes in the target video is not less than a preset number, then the step of determining the valid subtitle region and identifying the subtitles in the valid subtitle region of each video frame as valid subtitles is performed.

[0114] Specifically, after identifying the subtitle content of each video frame, frames where the subtitle content changes are identified as keyframes. From the video content, subtitles such as station logos remain unchanged; usually, only normal subtitles change. However, considering the inherent errors in text recognition (such as misidentifying text present in the scene as subtitles), and taking into account the presence of interfering subtitles, if a video has fewer keyframes than a preset number, it can be inferred that the video itself does not contain normal subtitles, or that the subtitles are too few, and the video is then filtered out. If the number of keyframes in a video meets the requirements, the video is then processed further.

[0115] In this embodiment, by detecting the number of keyframes in the video, videos without subtitles or with too few subtitles can be filtered out.

[0116] A method for obtaining speech recognition training data, such as Figure 9 As shown, it includes:

[0117] Step 801: Crawl videos from the specified website.

[0118] Step 802: Preprocess the video to obtain the target video.

[0119] Step 803: Identify the subtitle content of each video frame in the target video and determine the subtitle region containing subtitles.

[0120] Specifically, such as Figure 10 As shown, OCR services are used to identify subtitle content in each frame of the target video. Subtitle regions are defined based on their location within the video frames. Frames where the subtitle content changes are designated as keyframes. If the number of keyframes is less than a preset limit, the target video is filtered out.

[0121] Step 804: Determine the valid subtitle area. Determine the subtitles in the valid subtitle area of ​​each video frame as valid subtitles.

[0122] Specifically, if the subtitle content in the subtitle area does not meet the requirements for a valid subtitle, the subtitle area is removed. In the remaining subtitle areas, the subtitle areas whose horizontal or vertical coordinate positions remain unchanged are taken as valid subtitle areas, and the subtitles in the valid subtitle areas are determined as valid subtitles.

[0123] Step 805: Based on the appearance time and disappearance time of the valid subtitles, obtain the subtitle content of multiple valid subtitles and the time interval of each subtitle content.

[0124] Step 806: Perform speech boundary detection on the target video, and segment the target video according to the detected speech boundaries to obtain multiple audio segments of the target video and the time interval of each audio segment.

[0125] like Figure 10 As shown, VAD detection is applied to the target video, and audio is segmented based on the pause points detected by VAD to obtain multiple audio segments and the time interval of each audio segment.

[0126] Step 807: Based on the time intervals of each subtitle content and the time intervals of each audio segment, determine the initial subtitle content and target audio segment with a time interval overlap within the second range.

[0127] Step 808: Correct the time range of the initially selected subtitle content.

[0128] like Figure 10 As shown, 0.5 seconds can be added to the start time of the initial subtitle content to correct the time range of the initial subtitle content.

[0129] Step 809: Obtain the target subtitle content based on the subtitles within the correction time interval, and obtain the target audio segment based on the initially selected audio segment.

[0130] Step 810: Perform speech recognition on the target audio segment to obtain the speech recognition result.

[0131] Specifically, the ASR service is invoked to identify the target audio segment.

[0132] Step 811: If the speech recognition result matches the target subtitle content, then a speech training sample is obtained based on the target subtitle content and the target audio segment.

[0133] like Figure 10 The similarity between the speech recognition result of the target audio segment and the subtitle content is calculated. If the similarity is greater than the similarity threshold, the speech recognition result is considered to match the target subtitle content.

[0134] Step 812: If the speech recognition result does not match the target subtitle content, then remove the target subtitle content and the target audio segment.

[0135] like Figure 10 If the similarity is less than the similarity threshold, the target subtitle content and the target audio segment will be deleted.

[0136] The method for obtaining speech recognition training data in this embodiment addresses the problem of requiring a large amount of manpower to annotate training data during the text-to-speech matching speech recognition training sample annotation process. By extracting subtitle results from video resources and then matching them with subtitle intervals and speech timestamps, training samples are automatically produced in large quantities, saving manpower for annotation and improving the effectiveness of the annotated data.

[0137] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0138] Based on the same inventive concept, this application also provides an apparatus for acquiring speech recognition training data to implement the method for acquiring speech recognition training data described above. The solution provided by this apparatus is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the apparatus for acquiring speech recognition training data provided below can be found in the limitations of the method for acquiring speech recognition training data described above, and will not be repeated here.

[0139] In one embodiment, such as Figure 11 As shown, an apparatus for acquiring speech recognition training data is provided, comprising:

[0140] The subtitle processing module 1102 is used to identify the subtitles of each video frame in the target video, and obtain the subtitle content of multiple valid subtitles in the target video as well as the time interval of each subtitle content.

[0141] The speech processing module 1104 is used to perform speech boundary detection on the target video, and to segment the target video according to the detected speech boundaries to obtain multiple audio segments of the target video and the time interval of each audio segment.

[0142] The matching module 1106 is used to determine the target subtitle content and target audio segment to be matched based on the time interval of the subtitle content and the time interval of the audio segment.

[0143] The recognition module 1108 is used to perform speech recognition on the target audio segment and obtain the speech recognition result.

[0144] The processing module 1110 is used to obtain speech training samples based on the target subtitle content and the target audio segment if the speech recognition result matches the target subtitle content.

[0145] The aforementioned device for acquiring speech recognition training data utilizes video subtitles to extract subtitle content, performs boundary detection on the video's speech, and segments it into multiple speech segments. Based on the time intervals of the subtitle content and the audio segments, it matches the target subtitle content and the target audio segment, ensuring consistency between them along the time axis. Then, it performs speech recognition on the target audio segment, verifying the consistency between the target subtitle content and the target audio segment from a content perspective. This ensures that the final determined speech training samples are matched in both the time and content dimensions, improving the accuracy of speech training sample extraction. Furthermore, this method directly utilizes the subtitles and audio in the video, eliminating the need for audio annotation and enabling the rapid acquisition of large amounts of speech training data, thus improving the efficiency of speech training data acquisition.

[0146] In another embodiment, the matching module is used to determine the target subtitle content and the target audio segment with a time interval overlap within a first range, based on the time interval of each subtitle content and the time interval of each audio segment.

[0147] In another embodiment, the matching module includes:

[0148] The matching and judgment module is used to determine the initial subtitle content and target audio segment with a time interval overlap within the second range based on the time interval of each subtitle content and the time interval of each audio segment.

[0149] The correction module is used to correct the time range of the initially selected subtitle content; based on the subtitles within the corrected time range, the target subtitle content is obtained.

[0150] In another embodiment, the correction module is used to correct the time interval of the initially selected subtitle content with reference to the time interval of the target audio segment.

[0151] In another embodiment, the subtitle processing module is used for:

[0152] The text recognition module is used to identify the subtitle content of each video frame in the target video and determine the subtitle area containing the subtitles.

[0153] The effective subtitle region processing module is used to determine the effective subtitle region and identify the subtitles in the effective subtitle region of each video frame as effective subtitles.

[0154] The subtitle interval processing module is used to obtain the subtitle content of multiple valid subtitles and the time interval of each subtitle content based on the appearance time and disappearance time of the valid subtitles.

[0155] In another embodiment, the effective subtitle region processing module is used to remove the subtitle region if the subtitle content in the subtitle region does not meet the requirements for effective subtitles; in the remaining subtitle regions, the subtitle regions whose horizontal or vertical coordinate positions remain unchanged are taken as effective position subtitle regions; and the subtitles in the effective subtitle regions are determined as effective subtitles.

[0156] In another embodiment, the effective subtitle region processing module is used to remove the subtitle region if the subtitle content of the subtitle region is repeated, or to remove the subtitle region if the subtitle content of the subtitle region does not include the character encoding of the target language.

[0157] In another embodiment, the text recognition module is further configured to identify video frames in which the subtitle content changes as keyframes, and the effective subtitle region processing module, with a number of keyframes in the target video not less than a preset number, is configured to identify effective subtitle regions and determine the subtitles in the effective subtitle regions of each video frame as effective subtitles.

[0158] Each module in the aforementioned device for acquiring speech recognition training data can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0159] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 12 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores video data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When executed by the processor, the computer program implements a method for acquiring speech recognition training data.

[0160] Those skilled in the art will understand that Figure 12The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0161] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method for acquiring speech recognition training data as described in the above embodiments.

[0162] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method for acquiring speech recognition training data as described in the above embodiments.

[0163] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the method for acquiring speech recognition training data as described in the above embodiments.

[0164] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0165] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0166] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0167] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for acquiring speech recognition training data, characterized in that, The method includes: Identify the subtitles of each video frame in the target video to obtain the subtitle content of multiple valid subtitles in the target video and the time interval of each subtitle content; The target video is subjected to speech boundary detection, and the target video is segmented according to the detected speech boundaries to obtain multiple audio segments of the target video and the time interval of each audio segment; Based on the time intervals of each subtitle and the time intervals of each audio segment, determine the target subtitle content and target audio segment with the highest overlap in time intervals within the first range; If there are subtitle content and audio segments whose time interval overlap is not within the first range, then the initial subtitle content and target audio segment whose time interval overlap is within the second range are determined, the time interval of the initial subtitle content is corrected, and the target subtitle content is obtained based on the subtitles within the corrected time interval; wherein, the boundary value of the second range is smaller than the first range; speech recognition is performed on the target audio segment to obtain the speech recognition result; If the speech recognition result matches the target subtitle content, then a speech training sample is obtained based on the target subtitle content and the target audio segment.

2. The method according to claim 1, characterized in that, The correction of the time interval of the initially selected subtitle content includes: The time interval of the initially selected subtitle content is corrected using the time interval of the target audio segment as a reference.

3. The method according to claim 1, characterized in that, The process of identifying subtitles for each video frame in the target video, obtaining the subtitle content of multiple valid subtitles in the target video, and the time interval of each subtitle content, includes: Identify the subtitle content of each video frame in the target video and determine the subtitle region containing subtitles; Determine the valid subtitle region, and identify the subtitles in the valid subtitle region of each video frame as valid subtitles; Based on the appearance and disappearance times of the valid subtitles, the subtitle content of multiple valid subtitles is obtained, along with the time interval of each subtitle content.

4. The method according to claim 3, characterized in that, The step of determining the valid subtitle region, which involves identifying the subtitles within the valid subtitle region of each video frame as valid subtitles, includes: If the subtitle content in the subtitle area does not meet the requirements for valid subtitles, then the subtitle area will be removed. In the remaining subtitle areas, the subtitle areas whose horizontal or vertical coordinate positions remain unchanged are considered as valid position subtitle areas; The subtitles in the effective subtitle area are determined as effective subtitles.

5. The method according to claim 4, characterized in that, If the subtitle content in the subtitle area does not meet the requirements for valid subtitles, then the subtitle area shall be removed, including at least one of the following methods: The first method is to remove the subtitle area if the subtitle content is repeated. The second method: If the subtitle content in the subtitle area does not include the character encoding of the target language, then the subtitle area is removed.

6. The method according to claim 3, characterized in that, After identifying the subtitle content of each video frame in the target video and determining the subtitle region with subtitles, the method further includes: The video frames in which the subtitle content changes are identified as keyframes; If the number of keyframes in the target video is not less than a preset number, then the step of determining the effective subtitle region is performed, and the subtitles in the effective subtitle region of each video frame are determined as effective subtitles.

7. A device for acquiring speech recognition training data, characterized in that, The device includes: The subtitle processing module is used to identify the subtitles of each video frame in the target video, and obtain the subtitle content of multiple valid subtitles in the target video and the time interval of each subtitle content; The speech processing module is used to perform speech boundary detection on the target video, segment the target video according to the detected speech boundaries, and obtain multiple audio segments of the target video and the time interval of each audio segment. The matching module is used to determine the target subtitle content and target audio segment with a time interval overlap within a first range based on the time intervals of each subtitle content and each audio segment; if there are subtitle content and audio segments with a time interval overlap outside the first range, then the module determines the initial subtitle content and target audio segment with a time interval overlap within a second range, corrects the time interval of the initial subtitle content, and obtains the target subtitle content based on the subtitles within the corrected time interval; wherein the boundary value of the second range is smaller than that of the first range; The recognition module is used to perform speech recognition on the target audio segment and obtain the speech recognition result; The processing module is configured to obtain a speech training sample based on the target subtitle content and the target audio segment if the speech recognition result matches the target subtitle content.

8. The apparatus according to claim 7, characterized in that, The matching module includes a correction module, which is used to correct the time interval of the initially selected subtitle content with reference to the time interval of the target audio segment.

9. The apparatus according to claim 7, characterized in that, The subtitle processing module is further configured to: identify the subtitle content of each video frame in the target video and determine the subtitle region with subtitles; determine the effective subtitle region and identify the subtitles in the effective subtitle region of each video frame as effective subtitles; and obtain the subtitle content of multiple effective subtitles and the time interval of each subtitle content according to the appearance time and disappearance time of the effective subtitles.

10. The apparatus according to claim 9, characterized in that, The subtitle processing module is further configured to: if the subtitle content in the subtitle area does not meet the requirements for a valid subtitle, then remove the subtitle area; in the remaining subtitle areas, take the subtitle areas whose horizontal or vertical coordinate positions remain unchanged as valid position subtitle areas; and determine the subtitles in the valid subtitle areas as valid subtitles.

11. The apparatus according to claim 10, characterized in that, The subtitle processing module is further configured to: remove the subtitle region if the subtitle content of the subtitle region is repeated; and remove the subtitle region if the subtitle content of the subtitle region does not include the character encoding of the target language.

12. The apparatus according to claim 9, characterized in that, The subtitle processing module is further configured to: identify video frames in which subtitle content changes as keyframes; and if the number of keyframes in the target video is not less than a preset number, execute the step of determining the effective subtitle region and identifying the subtitles in the effective subtitle region of each video frame as effective subtitles.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Citation Information

Patent Citations

  • Training data acquisition method and device

    CN113450774A

  • Audio-to-text training corpus obtaining method and device, equipment and storage medium

    CN113705300A

  • Language-agnostic subtitle drift detection and localization

    US10945041B1