A real-time generation system for subtitles and their display graphics

Abstract

The invention discloses a subtitle and a display graphic real-time generation system thereof, which relates to the technical field of subtitle generation, and comprises an audio and video import module, an audio extraction module, a voice conversion module, a role recognition module, an emotion recognition module, a font graphic module, a subtitle display control module, and a subtitle graphic control module; the subtitle and the display graphic real-time generation system thereof can recognize the emotion of a role by setting the audio and video import module, the audio extraction module, the voice conversion module, the role recognition module, the emotion recognition module, etc., and use different set fonts and decorative graphics to display and decorate subtitles with different emotions, so that the audience can understand the emotion of the role through the subtitles, which is more interesting; the number of words in each subtitle is controlled, and the position of the barrage display is controlled based on a time interval, so that each subtitle information is not too long, and the display time of each subtitle information is guaranteed, so that the audience has time to watch.

Description

Technical Field

[0001] The present invention relates to the technical field of subtitle generation, and in particular to a real-time generation system of subtitles and display graphics thereof. Background Art

[0002] Subtitles refer to the non-visual content such as dialogues in television, film, and stage works displayed in text form, and also generally refer to the text processed in the post-production of film and television works. The commentary text and various texts that appear at the bottom of the movie screen or TV screen.

[0003] The prior art with the publication number CN113343675A discloses a subtitle generation method, device and device for generating subtitles. The method comprises: performing speech recognition processing on the audio and video signals to be processed, obtaining a text sequence corresponding to the audio and video signals and a timestamp mapping table of the text sequence, wherein the timestamp mapping table includes the timestamps corresponding to each word segment in the text sequence; determining the boundary word segment in the text sequence; splitting the text sequence according to the boundary word segment and the timestamp corresponding to the boundary word segment, and generating a subtitle line file corresponding to the audio and video signals. The embodiment of the present invention can improve the accuracy and fluency of the generated subtitle file.

[0004] The above-mentioned existing technologies are mainly aimed at optimizing the accuracy and fluency of subtitle display, and are more suitable for real-time recognition and conversion of voice information in audio and video files. However, the display graphics and display time of subtitles are not optimized. The voice speed of the audio and video may be too fast, resulting in the subtitle display time being too short, and the viewer may not have time to watch it. In addition, the existing subtitles are usually displayed directly, which lacks interest. Summary of the invention

[0005] The purpose of the present invention is to provide a real-time generation system of subtitles and display graphics thereof to solve the above-mentioned deficiencies in the prior art.

[0006] In order to achieve the above-mentioned object, the present invention provides the following technical solutions: a real-time generation system of subtitles and display graphics thereof, comprising an audio and video import module, an audio extraction module, a voice conversion module, a role recognition module, an emotion recognition module, a font graphics module, a subtitle display control module, and a subtitle graphics control module;

[0007] The audio and video import module is used to import audio and video files for processing;

[0008] The audio extraction module is used to extract audio information from the audio and video files. The audio information can be extracted by using audio extraction software and tools, such as FFmpeg, Audacity, etc.

[0009] The speech conversion module is used to identify the audio information, divide the audio information into audio information segments based on an endpoint detection algorithm, and convert the audio information segments into text based on a speech recognition algorithm (Speech Recognition is an important branch of the field of Natural Language Processing (NLP)) to obtain subtitle information; wherein the endpoint detection algorithm is common knowledge in the prior art and is directly applied here without modification, so it is not specifically described in this technical solution, and will not cause trouble to this technical solution based on the art;

[0010] The role recognition module is used to recognize the audio information of different roles in the audio information segment based on a first deep learning model, wherein the first deep learning model is a pre-trained model, such as the ECAPA-TDNN model of SpeechBrain;

[0011] The role identification module is also used to use different annotation contents to annotate the subtitle information corresponding to the audio information segments of different roles;

[0012] The emotion recognition module is used to set multiple emotion tags, and associate the emotion tags with the subtitle information that matches the emotion of the corresponding audio information segment, that is, if the emotion expressed by the audio information segment corresponding to a subtitle information is the same as or matches the emotion corresponding to an emotion tag, then the subtitle information is associated with the emotion tag, wherein the emotion tag can be set as needed, such as happy, angry, sad, sad, fearful, surprised, etc.; wherein the association between data can be realized through SQL tools;

[0013] The emotion recognition module is also used to train a second deep learning model based on the audio information segments corresponding to each subtitle information and the emotion tags associated with the subtitle information to obtain an emotion recognition model, which is used to output predicted emotion tags based on the input audio information segments. The present invention does not limit the specific deep learning model, and the deep learning model can be selected according to actual needs during the training process, so that the output accuracy of the trained model meets the set requirements.

[0014] The font graphic module is used to set a plurality of font styles and decorative graphic styles corresponding to each emotion label. The font style is the display style of the subtitle information. The decorative graphic style is used to display near the subtitle information and decorate the subtitle information. For example, if a subtitle information is "demonstration subtitle" and its emotion label is "happy", then the "demonstration subtitle" is displayed with a font style corresponding to the emotion label: happy, and the two ends or the top and bottom of the "demonstration subtitle" are displayed with decorative graphic styles corresponding to the emotion label: happy.

[0015] The subtitle display control module is used to obtain the audio duration of the audio information segment corresponding to each subtitle information, obtain the subtitle audio duration, and control the subtitle display duration of the subtitle information based on the subtitle audio duration of the subtitle information;

[0016] The subtitle display control module is also used to obtain the number of words in the subtitle information to obtain the subtitle length; obtain the time of the audio information segment corresponding to the subtitle information in the audio information, calculate the time interval of the audio information segments corresponding to each subtitle information, and control the position of the subtitle information based on the time interval, wherein the time interval is the time obtained by subtracting the end time of the audio information segment that starts earlier from the start time of the audio information segment that starts later; wherein the time interval can be positive or negative, and when the time interval is a positive number, it means that the time of the audio information segment that starts later does not overlap with the time of the audio information segment that starts earlier, and when the time interval is a negative number, it means that the time of the audio information segment that starts later overlaps with the time of the audio information segment that starts earlier;

[0017] The subtitle graphic control module is used to control the subtitle information to be displayed in a corresponding font style based on the emotion tag of the subtitle information, and to control the corresponding decorative graphic style to be displayed near the subtitle information, wherein the decorative graphic style can be emoticons, emoticons, etc.

[0018] Furthermore, the role identification module uses different annotation contents to annotate the subtitle information corresponding to the audio information of different roles, specifically including the following steps:

[0019] A1. Obtain the number of roles, generate a corresponding number of unique identification codes according to the number of roles, and associate the unique identification codes with the roles one by one;

[0020] A2. Mark the unique identification code before the audio information segment corresponding to each subtitle information; for example, if the unique identification code is role 1 and the subtitle information is "demonstration subtitle", the marked subtitle information is "role 1: demonstration subtitle";

[0021] A3. Set the role name corresponding to each unique identification code, and associate the role name with the corresponding unique identification code;

[0022] A4. Replace the marked unique identification code with the corresponding role name; for example, if the unique identification code is role 1 and the corresponding role name is Xiao Ming, and the marked subtitle information is "Role 1: Demonstration subtitles", the replaced subtitle information is "Xiao Ming: Demonstration subtitles", so that the system can randomly generate a unique identification code, and then use the role name to replace the unique identification code as needed, so that the system can identify different roles corresponding to the marked audio information, and it is also convenient to change the role name later, add or delete the audio information corresponding to the role, and change and correct the role name in the case of recognition errors. For example, the audio information corresponding to the two subtitle information corresponding to the role of Xiao Ming is respectively identified as role 2 and role 3, then both role 2 and role 3 can be associated with Xiao Ming by manual correction in the later stage, so as to change and correct the role name;

[0023] A5. Return to A1, recheck the number of roles, add or delete unique identification codes, and start a new cycle.

[0024] Furthermore, the role identification module is also used to divide the subtitle information so that the subtitle information corresponds to the role, including the following steps:

[0025] Identify, by a first deep learning model, whether the audio information segment contains only audio information of one character;

[0026] If yes, keep the subtitle information corresponding to the audio information segment unchanged;

[0027] If not, mark the audio information corresponding to each character in the audio information segment, and based on the speech recognition algorithm, convert the audio information corresponding to each character into subtitle information, that is, the audio information corresponding to one subtitle information corresponds to one character, and divide the audio information segment accordingly based on the converted subtitle information to obtain sub-audio information segments, each sub-audio information segment corresponds one-to-one with the converted subtitle information. For example, an audio information segment is 5 seconds long and contains audio information of character 1 and character 2, wherein the audio information of character 1 is 0-5 seconds, and the audio information of character 2 is 3-5 seconds. The converted subtitle information is subtitle information 1 corresponding to character 1, and subtitle information 2 corresponding to character 2. The two sub-audio information segments are the audio information segment of 0-5 seconds corresponding to subtitle information 1, and the audio information segment of 3-5 seconds corresponding to subtitle information 2.

[0028] Furthermore, before performing corresponding segmentation on the audio information segments based on the converted subtitle information, it is also necessary to detect whether the number of words in the subtitle information is less than or equal to the set number of words. If not, the subtitle information is divided into multiple subtitle information so that the number of words in each subtitle information is less than or equal to the set number of words. Then, based on the subtitle information, the audio information segments are correspondingly segmented to obtain sub-audio information segments, and each sub-audio information segment corresponds one-to-one to the sub-subtitle information.

[0029] Furthermore, the role identification module is also used to divide the subtitle information so that the subtitle information corresponds to the role, and further includes the following steps:

[0030] Determine whether an audio information segment contains multiple roles;

[0031] If so, determining whether the subtitle information converted from the audio information corresponding to the multiple roles is the same;

[0032] If so, record the roles corresponding to the same subtitle information, merge the same subtitle information into one subtitle information, and mark the recorded roles in front of the merged subtitle information. The roles can be marked by marking the unique identification codes corresponding to the roles, and then after setting the role names corresponding to the unique identification codes, the unique identification codes can be uniformly replaced with the corresponding role names.

[0033] In addition, in some embodiments, the subtitle information of different characters can be marked by using different colors, different fonts, different font weights, etc.

[0034] Furthermore, the emotion recognition module is also used to extract facial features of the character in the audio and video information, train the third deep learning model, and obtain a facial emotion recognition model, which is used to identify and extract the facial features of the character based on the audio and video information and predict the character's emotions, specifically including the following steps:

[0035] Recognize the character's face in the audio and video information based on the pre-trained object detection algorithm, and extract the character's facial image;

[0036] Extracting facial features of the character from the facial image of the character, training a second deep learning model based on the facial features of the character and the emotion labels corresponding to the facial features to obtain a facial emotion recognition sub-model, wherein the facial emotion recognition sub-model and the target detection algorithm form a facial emotion recognition model;

[0037] The audio and video information is input into the facial emotion recognition model, the character facial image in the audio and video information is automatically identified and extracted through the target detection algorithm, the facial emotion recognition sub-model is input to extract the character facial features in the character facial image, and the predicted emotion label is output based on the character facial features.

[0038] Further, the target detection algorithm is also used to identify the character corresponding to the character facial image when the character face is identified to obtain the character facial image, including the following steps:

[0039] According to the number of roles, a corresponding number of facial identification codes are generated, and each facial identification code is unique;

[0040] Associate each facial recognition code with a character one by one;

[0041] The character name is obtained, and the character name is associated with the corresponding facial recognition code, so that the character's audio information and subtitle information can be associated with the character's facial image through the character name.

[0042] Furthermore, if the predicted emotion label output by the facial emotion recognition model is different from the predicted emotion label output by the emotion recognition model, the emotion recognition module outputs the predicted emotion label output by the facial emotion recognition model. Because the micro-expressions of the face are more difficult to control than the voice, it is more accurate to judge emotions by recognizing micro-expressions through facial features.

[0043] Furthermore, the subtitle display control module controls the position of the subtitle information based on the time interval, and further includes the following steps:

[0044] Determine whether the start times of two audio information segments are the same;

[0045] If they are the same, determine whether the durations of the two audio information segments are equal;

[0046] If they are equal, the time interval is not calculated, and the subtitle information corresponding to the two audio information segments is controlled to be displayed simultaneously;

[0047] If they are not equal, the time interval is equal to the duration obtained by subtracting the end time of the shorter audio information segment from the start time of the longer audio information segment.

[0048] Furthermore, if the absolute value of the time interval is less than the set interval threshold, and the start times of the two audio information segments are different, the subtitle information of the audio information segment that starts earlier is controlled to be displayed first, and the subtitle information of the audio information segment that starts later is controlled to be displayed later, and the subtitle information displayed later is displayed at the position of the subtitle information displayed earlier, and the subtitle information displayed earlier is moved up and displayed in two lines with the subtitle information displayed later;

[0049] If the time interval is less than or equal to 0, and the start time of the two audio information segments is the same, the subtitle information corresponding to the two audio information segments is displayed in two lines at the same time, and the subtitle information corresponding to the audio information segment with a shorter duration is displayed on the top;

[0050] If the time interval is greater than 0, and the absolute value is greater than the set interval threshold, the subtitle information of the audio information segment that starts first is controlled to be displayed first, and the display ends after the set display time, and then the subtitle information of the audio information segment that starts later is displayed at the position of the subtitle information displayed first.

[0051] 1. Compared with the prior art, the present invention provides a real-time generation system for subtitles and display graphics thereof, which can identify different roles in audio information and mark the subtitles of different roles respectively by setting an audio and video import module, an audio extraction module, a voice conversion module, a role recognition module, an emotion recognition module, a font graphic module, a subtitle display control module, and a subtitle graphic control module, so as to facilitate the audience to understand the speaking roles more clearly and quickly through the subtitles when multiple roles are speaking.

[0052] 2. Compared with the prior art, the present invention provides a real-time generation system for subtitles and display graphics thereof, which can identify the emotions of a character by setting an audio and video import module, an audio extraction module, a voice conversion module, a character recognition module, an emotion recognition module, a font graphic module, a subtitle display control module, and a subtitle graphic control module, and use different set fonts and decorative graphics to display and decorate subtitles with different emotions, so that the audience can understand the emotions of the character through the subtitles, which is more interesting.

[0053] 3. Compared with the prior art, the present invention provides a real-time generation system for subtitles and display graphics thereof, which controls the number of words in each subtitle, calculates the time interval of subtitles with similar time corresponding to audio information, and controls the position of the barrage display based on the time interval, by setting an audio and video import module, an audio extraction module, a voice conversion module, a role recognition module, an emotion recognition module, a font graphic module, a subtitle display control module, and a subtitle graphic control module, so as to prevent each subtitle from being too long and ensure the display time of each subtitle so that the audience can watch it in time. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0055] Figure 1 A schematic diagram of the system structure provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0057] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0058] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, "multiple" means two or more, unless otherwise clearly and specifically defined. In addition, the terms "installed", "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal connection of two components. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0059] Example embodiments will be described more fully below with reference to the accompanying drawings, but the example embodiments may be embodied in different forms and should not be construed as limited to the embodiments set forth herein. On the contrary, the purpose of providing these embodiments is to make the present disclosure thorough and complete and to enable those skilled in the art to fully understand the scope of the present disclosure.

[0060] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.

[0061] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0062] The terms used herein are only used to describe specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms "a", "an" and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "comprising" and / or "made of" are used in this specification, the presence of the features, wholes, steps, operations, elements and / or components is specified, but the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof is not excluded.

[0063] The embodiments described herein may be described with reference to plan views and / or cross-sectional views by means of idealized schematic diagrams of the present disclosure. Therefore, the example illustrations may be modified according to manufacturing techniques and / or tolerances. Therefore, the embodiments are not limited to the embodiments shown in the accompanying drawings, but include modifications of the configurations formed based on the manufacturing process. Therefore, the regions illustrated in the accompanying drawings have schematic properties, and the shapes of the regions shown in the figures illustrate the specific shapes of the regions of the elements, but are not intended to be limiting.

[0064] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined as such herein.

[0065] See also Figure 1 , a real-time generation system of subtitles and display graphics thereof, comprising an audio and video import module, an audio extraction module, a voice conversion module, a role recognition module, an emotion recognition module, a font graphics module, a subtitle display control module, and a subtitle graphics control module;

[0066] The audio and video import module is used to import audio and video files for processing;

[0067] The audio extraction module is used to extract audio information from audio and video files. The audio information can be extracted by using audio extraction software and tools, such as FFmpeg, Audacity, etc.

[0068] The speech conversion module is used to recognize audio information, divide the audio information into audio information segments based on an endpoint detection algorithm, and convert the audio information segments into text based on a speech recognition algorithm (Speech Recognition is an important branch of the field of Natural Language Processing (NLP)) to obtain subtitle information; wherein the endpoint detection algorithm is common knowledge in the prior art and is directly applied without modification, so it is not specifically described in this technical solution, and will not cause trouble to this technical solution based on the field;

[0069] The role recognition module is used to recognize the audio information of different roles in the audio information segment based on the first deep learning model, wherein the first deep learning model is a pre-trained model, such as the ECAPA-TDNN model of SpeechBrain; the role recognition module is also used to use different annotation contents to annotate the subtitle information corresponding to the audio information segments of different roles, specifically including the following steps:

[0070] A1. Obtain the number of roles, generate a corresponding number of unique identification codes according to the number of roles, and associate the unique identification codes with the roles one by one;

[0071] A2. A unique identification code is marked before the audio information segment corresponding to each subtitle information; for example, if the unique identification code is role 1 and the subtitle information is "demonstration subtitle", the marked subtitle information is "role 1: demonstration subtitle";

[0072] A3. Set the role name corresponding to each unique identification code, and associate the role name with the corresponding unique identification code;

[0073] A4. Replace the marked unique identification code with the corresponding role name; for example, if the unique identification code is role 1 and the corresponding role name is Xiao Ming, and the marked subtitle information is "Role 1: Demonstration subtitles", the replaced subtitle information is "Xiao Ming: Demonstration subtitles", so that the system can randomly generate a unique identification code, and then use the role name to replace the unique identification code as needed, so that the system can identify different roles corresponding to the marked audio information, and it is also convenient to change the role name later, add or delete the audio information corresponding to the role, and change and correct the role name in the case of recognition errors. For example, the audio information corresponding to the two subtitle information corresponding to the role of Xiao Ming is respectively identified as role 2 and role 3, then both role 2 and role 3 can be associated with Xiao Ming by manual correction in the later stage, so as to change and correct the role name;

[0074] A5. Return to A1, recheck the number of roles, add or delete unique identification codes, and start a new cycle.

[0075] The role identification module is also used to divide the subtitle information so that the subtitle information corresponds to the role, including the following steps:

[0076] B1, identifying whether the audio information segment contains only audio information of one character through the first deep learning model;

[0077] B2. If yes, keep the subtitle information corresponding to the audio information segment unchanged; if not, mark the audio information corresponding to each role in the audio information segment, and convert the audio information corresponding to each role into subtitle information based on the speech recognition algorithm, that is, the audio information corresponding to one subtitle information corresponds to one role, and divide the audio information segment accordingly based on the converted subtitle information to obtain sub-audio information segments, each sub-audio information segment corresponds to the converted subtitle information one by one, for example, an audio information segment is 5 seconds long and contains audio information of role 1 and role 2, wherein the audio information of role 1 is 0-5 seconds, and the audio information of role 2 is 3-5 seconds, then the converted subtitle information is subtitle information 1 corresponding to role 1, and subtitle information 2 corresponding to role 2, then the two sub-audio information segments are the audio information segment of 0-5 seconds corresponding to subtitle information 1, and the audio information segment of 3-5 seconds corresponding to subtitle information 2.

[0078] The role identification module is also used to divide the subtitle information so that the subtitle information corresponds to the role, and further includes the following steps:

[0079] C1. Determine whether an audio information segment contains multiple roles;

[0080] C2. If yes, determine whether the subtitle information converted from the audio information corresponding to the multiple roles is the same;

[0081] C3. If so, record the roles corresponding to the same subtitle information, merge the same subtitle information into one subtitle information, and mark the recorded roles in front of the merged subtitle information. The roles can be marked by marking the unique identification codes corresponding to the roles, and then after setting the role names corresponding to the unique identification codes, the unique identification codes are uniformly replaced with the corresponding role names.

[0082] In addition, in some embodiments, the subtitle information of different characters can be marked by using different colors, different fonts, different font weights, etc.

[0083] The emotion recognition module is used to set multiple emotion tags, and associate the emotion tags with the subtitle information that matches the emotion of the corresponding audio information segment, that is, if the emotion expressed by the audio information segment corresponding to a subtitle information is the same as or matches the emotion corresponding to an emotion tag, then the subtitle information is associated with the emotion tag, wherein the emotion tag can be set as needed, such as happy, angry, sad, sad, fearful, surprised, etc.; wherein the association between data can be realized through SQL tools;

[0084] The emotion recognition module is also used to train a second deep learning model based on the audio information segments corresponding to each subtitle information and the emotion tags associated with the subtitle information to obtain an emotion recognition model, which is used to output predicted emotion tags based on the input audio information segments. The present invention does not limit the specific deep learning model, and the deep learning model can be selected according to actual needs during the training process so that the output accuracy of the trained model meets the set requirements.

[0085] The emotion recognition module is also used to extract the facial features of the characters in the audio and video information, train the third deep learning model, and obtain the facial emotion recognition model, which is used to identify and extract the facial features of the characters based on the audio and video information and predict the emotions of the characters, specifically including the following steps:

[0086] D1. Identify the character's face in the audio and video information based on the pre-trained target detection algorithm and extract the character's facial image;

[0087] The target detection algorithm is also used to identify the character corresponding to the character facial image when the character face is identified to obtain the character facial image, including the following steps:

[0088] (1) Generate a corresponding number of facial recognition codes based on the number of characters, and each facial recognition code is unique;

[0089] (2) Associating each facial recognition code with a character one by one;

[0090] (3) Obtaining the character name and associating the character name with the corresponding facial recognition code, so that the character's audio information and subtitle information can be associated with the character's facial image through the character name.

[0091] D2. Extract the facial features of the character from the facial image of the character, and train the second deep learning model based on the facial features of the character and the emotion labels corresponding to the facial features to obtain a facial emotion recognition sub-model. The facial emotion recognition sub-model and the target detection algorithm form a facial emotion recognition model;

[0092] D3. Input the audio and video information into the facial emotion recognition model, automatically identify and extract the character's facial image in the audio and video information through the target detection algorithm, input it into the facial emotion recognition sub-model to extract the character's facial features in the character's facial image, and output the predicted emotion label based on the character's facial features.

[0093] In one embodiment, if the predicted emotion label output by the facial emotion recognition model is different from the predicted emotion label output by the emotion recognition model, the emotion recognition module outputs the predicted emotion label output by the facial emotion recognition model. Because the micro-expressions of the face are more difficult to control than the voice, it is more accurate to judge emotions by recognizing micro-expressions through facial features.

[0094] The font graphic module is used to set a plurality of font styles and decorative graphic styles corresponding to each emotion label. The font style is the display style of the subtitle information. The decorative graphic style is used to display the subtitle information near the subtitle information and decorate the subtitle information. For example, if a subtitle information is "demonstration subtitle" and its emotion label is happy, then the "demonstration subtitle" is displayed with the font style corresponding to the emotion label: happy, and the two ends or the top and bottom of the "demonstration subtitle" are displayed with the decorative graphic style corresponding to the emotion label: happy.

[0095] The subtitle display control module is used to obtain the audio duration of the audio information segment corresponding to each subtitle information, obtain the subtitle audio duration, and control the subtitle display duration of the subtitle information based on the subtitle audio duration of the subtitle information;

[0096] The subtitle display control module is also used to obtain the number of words in the subtitle information to obtain the subtitle length; obtain the time of the audio information segment corresponding to the subtitle information in the audio information, calculate the time interval of the audio information segment corresponding to each subtitle information, and control the position of the subtitle information based on the time interval, wherein the time interval is the time obtained by subtracting the end time of the audio information segment that starts earlier from the start time of the audio information segment that starts later; wherein the time interval can be positive or negative, and when the time interval is a positive number, it means that the time of the audio information segment that starts later does not overlap with the time of the audio information segment that starts earlier, and when the time interval is a negative number, it means that the time of the audio information segment that starts later overlaps with the time of the audio information segment that starts earlier;

[0097] The subtitle display control module controls the position of the subtitle information based on the time interval, and also includes the following steps:

[0098] E1. Determine whether the start times of two audio information segments are the same;

[0099] E2. If they are the same, determine whether the durations of the two audio information segments are equal;

[0100] E3, if they are equal, the time interval is not calculated, and the subtitle information corresponding to the two audio information segments is controlled to be displayed simultaneously;

[0101] E4. If they are not equal, the time interval is equal to the duration obtained by subtracting the end time of the shorter audio information segment from the start time of the longer audio information segment.

[0102] The subtitle information position can be controlled based on the time interval by the following steps:

[0103] If the absolute value of the time interval is less than the set interval threshold, and the start times of the two audio information segments are different, the subtitle information of the audio information segment that starts earlier is controlled to be displayed first, and the subtitle information of the audio information segment that starts later is controlled to be displayed later, and the subtitle information displayed later is displayed at the position of the subtitle information displayed earlier, and the subtitle information displayed earlier is moved up and displayed in two lines with the subtitle information displayed later;

[0104] If the time interval is less than or equal to 0 and the start time of the two audio information segments is the same, the subtitle information corresponding to the two audio information segments is displayed in two lines at the same time, and the subtitle information corresponding to the audio information segment with a shorter duration is displayed on the top;

[0105] If the time interval is greater than 0 and the absolute value is greater than the set interval threshold, the subtitle information of the audio information segment that starts first is controlled to be displayed first, and the display ends after the set display time, and then the subtitle information of the audio information segment that starts later is displayed at the position of the subtitle information displayed first.

[0106] In one embodiment, before the audio information segment is segmented accordingly based on the converted subtitle information, it is also necessary to detect whether the number of words in the subtitle information is less than or equal to the set number of words. If not, the subtitle information is divided into a plurality of sub-subtitle information, so that the number of words in each sub-subtitle information is less than or equal to the set number of words, and then the audio information segment is segmented accordingly based on the sub-subtitle information to obtain sub-audio information segments, each of which corresponds to the sub-subtitle information. When the speech speed of the audio information segment is too fast and the subtitle information is too long, the number of words in each subtitle information can be controlled through this embodiment, and the position of the subtitle information can be controlled in combination with the time interval, so that each subtitle information is not too long, and the display time of each subtitle information is guaranteed, so that the audience can watch it in time.

[0107] The subtitle graphic control module is used to control the subtitle information to be displayed in a corresponding font style based on the emotional tag of the subtitle information, and to control the corresponding decorative graphic style to be displayed near the subtitle information, where the decorative graphic style can be emoticons, emoticons, etc.

[0108] The above description is only by way of illustration of certain exemplary embodiments of the present invention. It is undoubted that those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the above drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A system for real-time generation of subtitles and display graphics thereof, characterized in that: It includes audio and video import module, audio extraction module, voice conversion module, role recognition module, emotion recognition module, font graphic module, subtitle display control module, and subtitle graphic control module; The audio and video import module is used to import audio and video files to be processed; The audio extraction module is used to extract audio information from the audio and video files; The speech conversion module is used to identify the audio information, divide the audio information into audio information segments based on an endpoint detection algorithm, and convert the audio information segments into text based on a speech recognition algorithm to obtain subtitle information; The role identification module is used to identify audio information of different roles in the audio information segment based on the first deep learning model; The role identification module is also used to use different annotation contents to annotate the subtitle information corresponding to the audio information segments of different roles; The emotion recognition module is used to set a plurality of emotion tags and associate the emotion tags with the subtitle information of the corresponding audio information segment emotion matching; The emotion recognition module is also used to train a second deep learning model based on the audio information segments corresponding to each subtitle information and the emotion tags associated with the subtitle information to obtain an emotion recognition model, wherein the emotion recognition model is used to output a predicted emotion tag based on the input audio information segments; The font graphic module is used to set a plurality of font styles and decorative graphic styles corresponding to each emotion tag, the font style is the display style of the subtitle information, and the decorative graphic style is used to be displayed near the subtitle information to decorate the subtitle information; The subtitle display control module is used to obtain the audio duration of the audio information segment corresponding to each subtitle information, obtain the subtitle audio duration, and control the subtitle display duration of the subtitle information based on the subtitle audio duration of the subtitle information; The subtitle display control module is also used to obtain the number of words in the subtitle information to obtain the subtitle length; obtain the time of the audio information segment corresponding to the subtitle information in the audio information, calculate the time interval of the audio information segment corresponding to each subtitle information, and control the position of the subtitle information based on the time interval, wherein the time interval is the length obtained by subtracting the end time of the audio information segment starting earlier from the start time of the audio information segment starting later; The subtitle graphic control module is used to control the subtitle information to be displayed in a corresponding font style based on the emotion tag of the subtitle information, and to control the corresponding decorative graphic style to be displayed near the subtitle information; The role identification module uses different annotation contents to annotate the subtitle information corresponding to the audio information of different roles, specifically including the following steps: A1. Obtain the number of roles, generate a corresponding number of unique identification codes according to the number of roles, and associate the unique identification codes with the roles one by one; A2, marking the unique identification code before the audio information segment corresponding to each subtitle information; A3. Set the role name corresponding to each unique identification code, and associate the role name with the corresponding unique identification code; A4. Replace the marked unique identification code with the corresponding role name; A5, return to A1; The emotion recognition module is also used to extract facial features of the character in the audio and video information, train the third deep learning model, and obtain a facial emotion recognition model. The facial emotion recognition model is used to identify and extract the facial features of the character based on the audio and video information, and predict the character's emotions, specifically including the following steps: Recognize the character's face in the audio and video information based on the pre-trained object detection algorithm, and extract the character's facial image; Extracting facial features of the character from the facial image of the character, training a second deep learning model based on the facial features of the character and the emotion labels corresponding to the facial features to obtain a facial emotion recognition sub-model, wherein the facial emotion recognition sub-model and the target detection algorithm form a facial emotion recognition model; Input the audio and video information into the facial emotion recognition model, automatically identify and extract the character facial image in the audio and video information through the target detection algorithm, input the facial emotion recognition sub-model to extract the character facial features in the character facial image, and output the predicted emotion label based on the character facial features; If the predicted emotion label output by the facial emotion recognition model is different from the predicted emotion label output by the emotion recognition model, the emotion recognition module outputs the predicted emotion label output by the facial emotion recognition model.

2. A system for real-time generation of subtitles and display graphics thereof according to claim 1, characterized in that: The role identification module is also used to divide the subtitle information so that the subtitle information corresponds to the role, including the following steps: Identify, by a first deep learning model, whether the audio information segment contains only audio information of one character; If not, mark the audio information corresponding to each character in the audio information segment, convert the audio information corresponding to each character into subtitle information based on the speech recognition algorithm, and divide the audio information segment accordingly based on the converted subtitle information to obtain sub-audio information segments, each sub-audio information segment corresponds one-to-one to the converted subtitle information.

3. A system for real-time generation of subtitles and display graphics thereof according to claim 2, characterized in that: Before performing corresponding segmentation on the audio information segment based on the converted subtitle information, it is also necessary to detect whether the number of words in the subtitle information is less than or equal to the set number of words. If not, the subtitle information is divided into multiple subtitle information so that the number of words in each subtitle information is less than or equal to the set number of words. Then, the audio information segment is correspondingly segmented based on the subtitle information to obtain sub-audio information segments, and each sub-audio information segment corresponds one-to-one to the sub-subtitle information.

4. A system for real-time generation of subtitles and display graphics thereof according to claim 2, characterized in that: The role identification module is also used to divide the subtitle information so that the subtitle information corresponds to the role, and further comprises the following steps: Determine whether an audio information segment contains multiple roles; If so, determining whether the subtitle information converted from the audio information corresponding to the multiple roles is the same; If so, the roles corresponding to the same subtitle information are recorded, the same subtitle information is merged into one subtitle information, and the recorded roles are marked before the merged subtitle information.

5. The system for real-time generation of subtitles and display graphics thereof according to claim 1, characterized in that: The target detection algorithm is also used to identify the character corresponding to the character facial image when the character face is identified to obtain the character facial image, including the following steps: According to the number of roles, a corresponding number of facial recognition codes are generated; Associate each facial recognition code with a character one by one; Get the role name and associate the role name with the corresponding facial recognition code.

6. A system for real-time generation of subtitles and display graphics thereof according to claim 2, characterized in that: The subtitle display control module controls the position of the subtitle information based on the time interval, and further comprises the following steps: Determine whether the start times of two audio information segments are the same; If they are the same, determine whether the durations of the two audio information segments are equal; If they are equal, the time interval is not calculated, and the subtitle information corresponding to the two audio information segments is controlled to be displayed simultaneously; If they are not equal, the time interval is equal to the duration obtained by subtracting the end time of the shorter audio information segment from the start time of the longer audio information segment.

7. A system for real-time generation of subtitles and display graphics thereof according to claim 6, characterized in that: Controlling the position of the subtitle information based on the time interval comprises the following steps: If the absolute value of the time interval is less than the set interval threshold, and the start times of the two audio information segments are different, the subtitle information of the audio information segment that starts earlier is controlled to be displayed first, and the subtitle information of the audio information segment that starts later is displayed later, and the subtitle information displayed later is displayed at the position of the subtitle information displayed earlier, and the subtitle information displayed earlier is moved up and displayed in two lines with the subtitle information displayed later; If the time interval is less than or equal to 0, and the start time of the two audio information segments is the same, the subtitle information corresponding to the two audio information segments is displayed in two lines at the same time, and the subtitle information corresponding to the audio information segment with a shorter duration is displayed on the top; If the time interval is greater than 0, and the absolute value is greater than the set interval threshold, the subtitle information of the audio information segment that starts first is controlled to be displayed first, and the display ends after the set display time, and then the subtitle information of the audio information segment that starts later is displayed at the position of the subtitle information displayed first.

Images