Intelligent system for speculating and repairing subtitles based on voice content

Abstract

The invention discloses an intelligent system for speculating and repairing subtitles based on voice content, and the system comprises the following modules: a voice recognition module which converts an input voice signal into a text, employs a deep learning technology, can recognize voices of different languages and accent, and converts the voices into an accurate text; the subtitle association module is used for associating the preprocessed voice text with the original subtitles; the subtitle speculation module is used for speculating subtitle contents by utilizing a natural language processing technology according to the voice contents; according to the invention, the voice content can be automatically identified, the voice content can be matched with the existing subtitles, inconsistent parts can be identified, automatic or semi-automatic repair suggestions can be provided, the subtitle accuracy and watching experience can be greatly improved by the application of the system, the manual auditing cost can be reduced, and the system has a wide market application prospect.

Description

Technical Field

[0001] The invention belongs to the technical field of subtitle restoration, and in particular relates to an intelligent system for inferring and repairing subtitles based on speech content. Background Art

[0002] With the rapid development of the Internet, video content has become an important way for people to obtain information. However, in practical applications, due to various reasons, video subtitles have problems such as errors and omissions, which bring inconvenience to viewers. Existing subtitle generation technologies mainly rely on manual dictation or automatic speech recognition technology, but these technologies often cannot fully and accurately capture the speech content, especially in noisy backgrounds or when the speaker has a heavy accent. In addition, the post-editing and review of subtitles is also quite cumbersome and requires a lot of manpower and time investment. Summary of the invention

[0003] The technical problem to be solved by the present invention is to overcome the existing defects and provide an intelligent system for inferring and repairing subtitles based on speech content to solve the problems raised in the above background technology.

[0004] To achieve the above object, the present invention provides the following technical solution: an intelligent system for inferring and repairing subtitles based on speech content, comprising the following modules:

[0005] Speech recognition module: converts the input speech signal into text. The speech recognition module uses deep learning technology to recognize speech in different languages ​​and accents and convert it into accurate text;

[0006] Subtitle association module: associates the preprocessed speech text with the original subtitles;

[0007] Subtitle inference module: Infer the subtitle content based on the speech content using natural language processing technology;

[0008] Subtitle matching module: matches the converted text with the existing subtitles and identifies inconsistencies. The subtitle matching module is responsible for matching the text output by the speech recognition module with the existing subtitles. The subtitle matching module uses a text similarity algorithm to quickly identify inconsistencies between subtitles and speech content. It uses edit distance, Jaccard similarity coefficient, and cosine similarity algorithm to evaluate the similarity between subtitle text and speech recognition output, quantify the difference between the two text sequences, and quickly identify inconsistencies. It integrates machine learning models, support vector machines, or neural networks to perform more complex matching analysis on subtitles and speech recognition results. The subtitle matching module needs to process data in real time to ensure the synchronization of subtitles during video playback. In the video editing process, it automatically matches and proofreads subtitles to reduce the workload of manual proofreading. In a live broadcast environment, it generates and proofreads subtitles in real time to improve the accessibility of live broadcast content.

[0009] Repair suggestion module: provides repair suggestions based on the matching results. The repair suggestion module provides automatic or semi-automatic repair suggestions based on the output of the subtitle matching module. The repair suggestion module uses natural language processing technology to analyze context information and provide the most appropriate repair solution. It uses NLP technology, dependency syntax analysis, entity recognition and semantic role labeling to understand context information and make repair suggestions. The module can provide more accurate repair suggestions based on context information, the theme and emotional color of the dialogue. Combined with the preset rule library, it automatically identifies common error types and proposes repair solutions. In the process of automatic subtitle generation, it automatically identifies and corrects common spelling and grammatical errors, and provides repair suggestions for subtitles in different languages ​​to adapt to different cultures and language habits.

[0010] Subtitle repair module: proofread and repair the inferred subtitles, generate correct subtitles, use machine learning algorithms, convolutional neural networks or recurrent neural networks to detect errors in subtitles, automatically or semi-automatically correct errors in subtitles based on the results of error detection, optimize the layout and format of subtitles to improve readability and aesthetics, automatically correct errors in the subtitle production process of movies and TV series, improve the quality of subtitles, provide accurate subtitles for educational videos, and help learners better understand the content.

[0011] Preferably, the speech recognition module includes a speech text preprocessing module, a speech feature extraction module, a speech model training module and a speech recognition output module.

[0012] Preferably, the speech-to-text preprocessing module includes a speech denoising module, a speech endpoint detection module, a speech word segmentation module, a part-of-speech tagging module and a named entity recognition module.

[0013] Preferably, the speech denoising module uses a noise suppression algorithm to denoise the speech signal;

[0014] The speech endpoint detection module realizes endpoint detection of speech signals through energy threshold and zero-crossing rate parameters;

[0015] The speech segmentation module uses a hidden Markov model and a conditional random field algorithm to segment the speech text;

[0016] The part-of-speech tagging module performs part-of-speech tagging on each word according to the word segmentation result;

[0017] The named entity recognition module recognizes named entities in speech text, such as names of people, places, and organizations.

[0018] Preferably, the speech feature extraction module selects a large amount of speech data to perform feature extraction;

[0019] The speech model training module uses deep learning algorithms, convolutional neural networks, and recurrent neural networks to train acoustic models;

[0020] The speech recognition output module decodes the preprocessed speech signal according to the acoustic model and outputs the recognition result, namely the speech text.

[0021] Preferably, the subtitle association module includes a subtitle time axis alignment module and a subtitle content matching module;

[0022] The subtitle timeline alignment module aligns the speech text with the original subtitles to ensure the correspondence between the subtitles and the speech content;

[0023] The subtitle content matching module calculates the matching degree between the speech text and the subtitle content through cosine similarity and edit distance algorithms.

[0024] Preferably, the subtitle inference module includes a context analysis module and a semantic understanding module;

[0025] The context analysis module infers the subtitle content based on the context information of the speech text;

[0026] The semantic understanding module uses natural language processing techniques, such as word vectors and semantic role annotations, to understand the semantics of spoken text.

[0027] Preferably, the subtitle inference module generates inferred subtitles based on the results of context analysis and semantic understanding.

[0028] Preferably, the subtitle repair module includes an error detection module, an error correction module and a subtitle optimization module;

[0029] The error detection module detects errors in the inferred subtitles through rule matching and machine learning methods;

[0030] The error correction module corrects the inferred subtitles according to the error detection result;

[0031] The subtitle optimization module performs typesetting and format adjustment optimization operations on the corrected subtitles.

[0032] Compared with the prior art, the present invention provides an intelligent system for inferring and repairing subtitles based on speech content, which has the following beneficial effects:

[0033] The present invention can automatically recognize speech content and match it with existing subtitles, identify inconsistencies, and provide automatic or semi-automatic repair suggestions. The application of this system will greatly improve the accuracy of subtitles and viewing experience, reduce manual review costs, and has broad market application prospects.

[0034] illustrate:

[0035] 1. Speech recognition module;

[0036] 11. Voice and text preprocessing module;

[0037] 101. Speech denoising module; 102. Speech endpoint detection module; 103. Speech word segmentation module; 104. Part-of-speech tagging module; 105. Named entity recognition module;

[0038] 12. Speech feature extraction module; 13. Speech model training module; 14. Speech recognition output module;

[0039] 2. Subtitle association module;

[0040] 21. Subtitle timeline alignment module; 22. Subtitle content matching module;

[0041] 3. Subtitle inference module;

[0042] 31. Context analysis module; 32. Semantic understanding module;

[0043] 4. Subtitle matching module; 5. Repair suggestion module;

[0044] 6. Subtitle repair module;

[0045] 61. Error detection module; 62. Error correction module; 63. Subtitle optimization module. DETAILED DESCRIPTION

[0046] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0047] The present invention provides a technical solution: an intelligent system for inferring and repairing subtitles based on speech content, comprising the following modules:

[0048] Speech recognition module 1: converts the input speech signal into text. Speech recognition module 1 uses deep learning technology to recognize speech in different languages ​​and accents and convert it into accurate text;

[0049] Subtitle association module 2: associate the preprocessed speech text with the original subtitles;

[0050] Subtitle inference module 3: infer the subtitle content based on the speech content using natural language processing technology;

[0051] Subtitle matching module 4: matches the converted text with the existing subtitles and identifies inconsistencies. Subtitle matching module 4 is responsible for matching the text output by speech recognition module 1 with the existing subtitles. Subtitle matching module 4 uses a text similarity algorithm to quickly identify inconsistencies between subtitles and speech content. Edit distance, Jaccard similarity coefficient, and cosine similarity algorithm are used to evaluate the similarity between subtitle text and speech recognition output, quantify the difference between the two text sequences, and quickly identify inconsistencies. Machine learning models, support vector machines, or neural networks are integrated to perform more complex matching analysis on subtitles and speech recognition results. The subtitle matching module needs to process data in real time to ensure the synchronization of subtitles during video playback. In the video editing process, subtitles are automatically matched and proofread to reduce the workload of manual proofreading. In a live broadcast environment, subtitles are generated and proofread in real time to improve the accessibility of live broadcast content.

[0052] Repair suggestion module 5: provides repair suggestions based on the matching results. Repair suggestion module 5 provides automatic or semi-automatic repair suggestions based on the output of the subtitle matching module. Repair suggestion module 5 uses natural language processing technology to analyze context information and provide the most appropriate repair solution. It uses NLP technology, dependency syntax analysis, entity recognition and semantic role labeling to understand context information and make repair suggestions. The module can provide more accurate repair suggestions based on context information, the theme and emotional color of the dialogue. Combined with the preset rule library, it automatically identifies common error types and proposes repair solutions. In the process of automatic subtitle generation, it automatically identifies and corrects common spelling and grammatical errors, and provides repair suggestions for subtitles in different languages ​​to adapt to different cultures and language habits.

[0053] Subtitle Repair Module 6: Proofread and repair the inferred subtitles to generate correct subtitles. Use machine learning algorithms, convolutional neural networks or recurrent neural networks to detect errors in subtitles. According to the results of error detection, automatically or semi-automatically correct errors in subtitles. Optimize the layout and format of subtitles to improve readability and aesthetics. In the process of subtitle production for movies and TV series, automatically correct errors and improve the quality of subtitles. Provide accurate subtitles for educational videos to help learners better understand the content.

[0054] The speech recognition module 1 includes a speech text preprocessing module 11 , a speech feature extraction module 12 , a speech model training module 13 and a speech recognition output module 14 .

[0055] The speech text preprocessing module 11 includes a speech denoising module 101 , a speech endpoint detection module 102 , a speech word segmentation module 103 , a part-of-speech tagging module 104 and a named entity recognition module 105 .

[0056] The speech denoising module 101 uses a noise suppression algorithm to denoise the speech signal;

[0057] The speech endpoint detection module 102 realizes endpoint detection of speech signals through energy threshold and zero-crossing rate parameters;

[0058] The speech segmentation module 103 uses a hidden Markov model and a conditional random field algorithm to segment the speech text;

[0059] The part-of-speech tagging module 104 tags each word according to the word segmentation result;

[0060] The named entity recognition module 105 recognizes named entities in the speech text, such as names of people, places, and organizations.

[0061] The speech feature extraction module 12 selects a large amount of speech data and performs feature extraction;

[0062] The speech model training module 13 uses deep learning algorithms, convolutional neural networks, and recurrent neural networks to train acoustic models;

[0063] The speech recognition output module 14 decodes the preprocessed speech signal according to the acoustic model and outputs the recognition result, namely the speech text.

[0064] The subtitle association module 2 includes a subtitle time axis alignment module 21 and a subtitle content matching module 22;

[0065] The subtitle time axis alignment module 21 aligns the speech text with the original subtitles to ensure the correspondence between the subtitles and the speech content;

[0066] The subtitle content matching module 22 calculates the matching degree between the speech text and the subtitle content through the cosine similarity and edit distance algorithms.

[0067] The subtitle inference module 3 includes a context analysis module 31 and a semantic understanding module 32;

[0068] The context analysis module 31 infers the subtitle content based on the context information of the speech text;

[0069] The semantic understanding module 32 uses natural language processing techniques, such as word vectors and semantic role labeling, to understand the semantics of the spoken text.

[0070] The subtitle inference module 3 generates inferred subtitles based on the context analysis and semantic understanding results.

[0071] The subtitle repair module 6 includes an error detection module 61, an error correction module 62 and a subtitle optimization module 63;

[0072] The error detection module 61 detects errors in the inferred subtitles through rule matching and machine learning methods;

[0073] The error correction module 62 corrects the inferred subtitles according to the error detection result;

[0074] The subtitle optimization module 63 performs typesetting and format adjustment optimization operations on the corrected subtitles.

[0075] The working principle and use process of the present invention are as follows: When in use, the system first converts the input voice signal into text through the voice recognition module 1, which can handle multiple languages ​​and accents to ensure high-precision conversion. The voice-to-text preprocessing module 11 performs denoising, endpoint detection, word segmentation, part-of-speech tagging and named entity recognition on the voice signal to improve the accuracy of subsequent processing. At the same time, the voice feature extraction module 12 extracts key features from a large amount of voice data. The voice model training module 13 uses deep learning algorithms (such as convolutional neural networks and recurrent neural networks) to train acoustic models, and the voice recognition output module 14 uses these models to decode the preprocessed voice signal and output text. The subtitle association module 2 The recognized speech text is aligned with the original subtitles in time axis and content to ensure the consistency of the subtitles with the speech content. The subtitle inference module 3 uses the natural language processing technology through the context analysis module 31 and the semantic understanding module 32 to infer the subtitle content and generate preliminary subtitles. The subtitle matching module 4 identifies the inconsistency between the inferred subtitles and the existing subtitles and provides them to the repair suggestion module 5. The repair suggestion module 5 analyzes the context information and proposes automatic or semi-automatic repair suggestions. The subtitle repair module 6 includes an error detection module 61, an error correction module 62 and a subtitle optimization module 63, which are respectively responsible for detecting errors, correcting subtitles, and performing typesetting and format adjustments to finally generate correct subtitles.

[0076] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An intelligent system for inferring and repairing subtitles based on speech content, characterized in that: Includes the following modules: Speech recognition module (1): converts the input speech signal into text. The speech recognition module (1) uses deep learning technology to recognize speech in different languages ​​and accents and convert it into accurate text; Subtitle association module (2): associates the preprocessed speech text with the original subtitles; Subtitle inference module (3): infers subtitle content based on speech content using natural language processing technology; Subtitle matching module (4): matches the converted text with the existing subtitles and identifies inconsistencies. The subtitle matching module (4) is responsible for matching the text output by the speech recognition module (1) with the existing subtitles. The subtitle matching module (4) uses a text similarity algorithm to quickly identify inconsistencies between subtitles and speech content. It uses edit distance, Jaccard similarity coefficient, and cosine similarity algorithm to evaluate the similarity between subtitle text and speech recognition output, quantify the difference between the two text sequences, and thus quickly identify inconsistencies. It integrates machine learning models, support vector machines, or neural networks to perform more complex matching analysis on subtitles and speech recognition results. The subtitle matching module needs to process data in real time to ensure the synchronization of subtitles during video playback. In the video editing process, it automatically matches and proofreads subtitles to reduce the workload of manual proofreading. In a live broadcast environment, it generates and proofreads subtitles in real time to improve the accessibility of live broadcast content. Repair suggestion module (5): provides repair suggestions based on the matching results. The repair suggestion module (5) provides automatic or semi-automatic repair suggestions based on the output of the subtitle matching module. The repair suggestion module (5) uses natural language processing technology to analyze context information and provide the most appropriate repair solution. It uses NLP technology, dependency syntax analysis, entity recognition and semantic role labeling to understand context information and make repair suggestions. The module can provide more accurate repair suggestions based on context information, the theme and emotional color of the dialogue. Combined with the preset rule library, it can automatically identify common error types and propose repair solutions. In the process of automatic subtitle generation, it can automatically identify and correct common spelling and grammatical errors, and provide repair suggestions for subtitles in different languages ​​to adapt to different cultures and language habits. Subtitle repair module (6): Proofread and repair the inferred subtitles to generate correct subtitles. Use machine learning algorithms, convolutional neural networks or recurrent neural networks to detect errors in subtitles. According to the results of error detection, automatically or semi-automatically correct the errors in subtitles. Optimize the layout and format of subtitles to improve readability and aesthetics. In the process of subtitle production for movies and TV series, automatically correct errors and improve the quality of subtitles. Provide accurate subtitles for educational videos to help learners better understand the content.

2. The intelligent system for inferring and repairing subtitles based on speech content according to claim 1, characterized in that: The speech recognition module (1) comprises a speech text preprocessing module (11), a speech feature extraction module (12), a speech model training module (13) and a speech recognition output module (14).

3. The intelligent system for inferring and repairing subtitles based on speech content according to claim 2, characterized in that: The speech text preprocessing module (11) comprises a speech denoising module (101), a speech endpoint detection module (102), a speech word segmentation module (103), a part-of-speech tagging module (104) and a named entity recognition module (105).

4. The intelligent system for inferring and repairing subtitles based on speech content according to claim 3, characterized in that: The speech denoising module (101) uses a noise suppression algorithm to perform denoising on the speech signal; The speech endpoint detection module (102) realizes endpoint detection of speech signals through energy threshold and zero-crossing rate parameters; The speech segmentation module (103) uses a hidden Markov model and a conditional random field algorithm to segment the speech text; The part-of-speech tagging module (104) performs part-of-speech tagging on each word according to the word segmentation result; The named entity recognition module (105) recognizes named entities in speech text, such as names of people, places, and organizations.

5. The intelligent system for inferring and repairing subtitles based on speech content according to claim 2, characterized in that: The speech feature extraction module (12) selects a large amount of speech data and performs feature extraction; The speech model training module (13) uses a deep learning algorithm, a convolutional neural network, and a recurrent neural network to train an acoustic model; The speech recognition output module (14) decodes the preprocessed speech signal according to the acoustic model and outputs the recognition result, i.e., speech text.

6. The intelligent system for inferring and repairing subtitles based on speech content according to claim 1, characterized in that: The subtitle association module (2) comprises a subtitle time axis alignment module (21) and a subtitle content matching module (22); The subtitle time axis alignment module (21) performs time axis alignment on the speech text and the original subtitles to ensure the correspondence between the subtitles and the speech content; The subtitle content matching module (22) calculates the matching degree between the speech text and the subtitle content through cosine similarity and edit distance algorithms.

7. The intelligent system for inferring and repairing subtitles based on speech content according to claim 1, characterized in that: The subtitle inference module (3) comprises a context analysis module (31) and a semantic understanding module (32); The context analysis module (31) infers the subtitle content based on the context information of the speech text; The semantic understanding module (32) uses natural language processing technology, such as word vectors and semantic role annotations, to understand the semantics of the spoken text.

8. The intelligent system for inferring and repairing subtitles based on speech content according to claim 1, characterized in that: The subtitle inference module (3) generates inferred subtitles based on the results of context analysis and semantic understanding.

9. The intelligent system for inferring and repairing subtitles based on speech content according to claim 1, characterized in that: The subtitle repair module (6) comprises an error detection module (61), an error correction module (62) and a subtitle optimization module (63); The error detection module (61) detects errors in the inferred subtitles through rule matching and machine learning methods; The error correction module (62) corrects the inferred subtitles according to the error detection result; The subtitle optimization module (63) performs typesetting and format adjustment optimization operations on the corrected subtitles.