Multimedia subtitle recognition display method

Abstract

The invention relates to the technical field of multimedia production, in particular to a multimedia subtitle recognition and display method, which comprises the following steps of: S1, acquiring a video to be processed, and extracting a subtitle area for the video to be processed; s2, identifying the subtitle area to obtain semantic information, and removing the subtitle area from the video to be processed to obtain a subtitle-removed video; and S3, adding subtitles in the subtitle-removed video according to the semantic information to form a subtitle-updated video. In order to solve the problem that watching is affected due to the fact that subtitles contained in an original film source are fuzzy in the secondary program making process in the prior art, in the scheme, the processes of extraction and semantic recognition for a subtitle area are introduced, so that the subtitle part in the film source is obtained through automatic analysis, then the original subtitle area is removed, and the subtitles are rendered again for embedding, and therefore the watching experience is improved. The problem that watching is affected due to the fact that original subtitles are fuzzy in the post-production process is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of multimedia production, and in particular to a multimedia subtitle recognition and display method. Background Art

[0002] Subtitles (subtitles of motion picture) refer to non-image content such as dialogues in television, film, and stage works displayed in text form. They also refer to the text processed in the post-production of film and television works. The commentary text and various texts that appear at the bottom of the movie screen or TV screen, such as the film title, cast list, lyrics, dialogues, descriptions, character introductions, place names, and years, are all called subtitles. The dialogue subtitles of film and television works generally appear at the bottom of the screen.

[0003] In the prior art, there are methods for adding subtitles to video programs. For example, Chinese patent CN200810222128.7 discloses a multi-channel subtitle editing and broadcasting method driven by an online program list, which belongs to the field of multi-channel subtitle editing and broadcasting of TV program production and broadcasting institutions in the field of radio and television. At present, the way of editing and broadcasting subtitles in most subtitle broadcasting systems has the disadvantages of large workload of subtitle production and easy broadcasting accidents. The method described in the present invention first obtains the online program list of all channels in the video broadcasting system; then generates the subtitle broadcast list of all channels according to the obtained online program list and the subtitle mapping table, subtitle template library and subtitle material library; finally, the generated subtitle broadcast list is transmitted to the broadcast subtitle machine of the corresponding channel according to the channel identifier; if the online program list of the channel changes, the subtitle broadcast list of the channel is updated. The method can couple the video broadcast and the subtitle broadcast tightly, thereby reducing the workload of manual subtitle production and avoiding broadcasting accidents caused by human operation errors.

[0004] However, in the actual implementation process, the inventor found that in the production process of some video programs, such as the production of online versions of movies, games, radio dramas and other programs, subtitles have been pre-added to the original film source. However, due to changes in the projection format, such as transcoding from game screen recordings, secondary compression, and bit rate control, the font format of the original subtitles may become blurred during the secondary production, affecting the viewing experience. Summary of the invention

[0005] In view of the above problems existing in the prior art, a multimedia subtitle recognition and display method is now provided.

[0006] The specific technical solutions are as follows:

[0007] A multimedia subtitle recognition and display method, comprising:

[0008] Step S1: obtaining a video to be processed, and extracting a subtitle area from the video to be processed;

[0009] Step S2: obtaining semantic information based on the subtitle region identification, and removing the subtitle region from the video to be processed to obtain a subtitle-free video;

[0010] Step S3: adding subtitles to the subtitle-less video according to the semantic information to form a subtitle-updated video.

[0011] On the other hand, the step S1 comprises:

[0012] Step S11: constructing multiple acquisition windows for the video to be processed;

[0013] Each of the acquisition windows includes a plurality of continuous image frames;

[0014] Step S12: for each acquisition window, frame averaging is performed to obtain an average image;

[0015] Step S13: identifying a color block area from the average image, and using the color block area as a window subtitle area;

[0016] Step S14: constructing a bounding rectangle for all the window subtitle areas as the subtitle area.

[0017] On the other hand, the step S2 comprises:

[0018] Step S21: for the subtitle area, respectively identify and obtain local semantic information;

[0019] Step S22: combining the local semantic information to obtain paragraph information;

[0020] Step S23: re-segmenting the paragraph information to obtain the semantic information.

[0021] On the other hand, the step S21 comprises:

[0022] Step S211: preprocessing the subtitle area to obtain a preprocessed image;

[0023] Step S212: segmenting the preprocessed image to obtain a word image;

[0024] Step S213: recognizing the word image to obtain the local semantic information.

[0025] On the other hand, after executing step S212 and before executing step S213, the method includes:

[0026] Step A212: determining the language type corresponding to the word image, and calling a language recognition template according to the language type;

[0027] In the step S213, the language recognition template is used for recognition.

[0028] On the other hand, the step S23 comprises:

[0029] Step S231: segmenting the paragraph information into single sentences to obtain single sentence semantics;

[0030] Step S232: re-map the time axis of the single sentence semantics to form the semantic information.

[0031] On the other hand, the step S2 further includes a subtitle removal process, which includes:

[0032] Step B21: for a single image frame, extract the subtitle area and adjacent images respectively;

[0033] Step B22: performing interpolation operation using the adjacent images to obtain an interpolated image;

[0034] Step B23: using the interpolated image to replace the adjacent image.

[0035] On the other hand, the step S3 comprises:

[0036] Step S31: Rendering a subtitle image according to the semantic information and a pre-built subtitle template;

[0037] Step S32: Add the subtitle image to the subtitle-removed video to form a subtitle-updated video.

[0038] The above technical solution has the following advantages or beneficial effects:

[0039] In view of the fact that the subtitles contained in the original film source in the prior art may cause blurring and other problems during the secondary production of the program, affecting the viewing experience, this solution introduces a process of extracting and semantically recognizing the subtitle area, thereby automatically parsing the subtitle part in the film source, removing the original subtitle area, and re-rendering the subtitles for embedding, thereby solving the problem that the original subtitles may become blurred and affect the viewing experience during the post-production process. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The embodiments of the present invention will be described more fully with reference to the attached drawings, which are provided for illustration and description only and are not intended to limit the scope of the present invention.

[0041] Figure 1 is an overall schematic diagram of an embodiment of the present invention;

[0042] Figure 2 This is a schematic diagram of step S1 in an embodiment of the present invention;

[0043] Figure 3 This is a schematic diagram of step S2 in an embodiment of the present invention;

[0044] Figure 4 This is a schematic diagram of step S21 in an embodiment of the present invention;

[0045] Figure 5 This is a schematic diagram of step A212 in an embodiment of the present invention;

[0046] Figure 6 This is a schematic diagram of step S23 in an embodiment of the present invention;

[0047] Figure 7 Schematic diagram of the subtitle removal process in an embodiment of the present invention;

[0048] Figure 8 Schematic diagram of step S3 in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0051] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0052] The present invention comprises:

[0053] A multimedia subtitle recognition and display method, such as Figure 1 As shown, including:

[0054] Step S1: obtaining a video to be processed, and extracting a subtitle area from the video to be processed;

[0055] Step S2: obtaining semantic information by identifying the subtitle area, and removing the subtitle area from the video to be processed to obtain a subtitle-free video;

[0056] Step S3: adding subtitles to the subtitle-less video according to the semantic information to form a subtitle-updated video.

[0057] Specifically, in view of the fact that the subtitles contained in the original film source in the prior art may cause blurring and other problems during the secondary production of the program, affecting the viewing experience, in this embodiment, a process of extracting and semantically recognizing the subtitle area is introduced, thereby automatically parsing the subtitle part in the film source, removing the original subtitle area, and re-rendering the subtitles for embedding, thereby solving the problem that the original subtitles may become blurry and affect the viewing experience during the post-production process.

[0058] During the implementation process, the above-mentioned multimedia subtitle recognition and display method is mainly configured as a software embodiment in a specific computer device, which is used to process the video to be processed that already has original subtitles, remove the original subtitle part, and extract the original subtitle part, recognize semantic information, render a new subtitle area and add it to the video to form a new subtitle update video.

[0059] Among them, the video to be processed refers to the film source involved in the production process, which usually has subtitles in a specific form, such as the film source with compressed subtitles, game recording video, etc.; the subtitle area refers to the part of the video to be processed that corresponds to the subtitles. Usually, the subtitle area will be distributed in specific frames in the video, roughly in the shape of rectangles of different lengths, and the height of the subtitle area is generally unchanged, and the font style remains unchanged. Semantic information refers to the actual text content of the corresponding sentence in each subtitle area, which removes style and other information relative to the original subtitle image. A subtitle-removed video refers to a video in which the image of the subtitle area is removed by image restoration technology, and the single frame image is restored using a background image. A subtitle-updated video refers to a video after the subtitles are re-rendered.

[0060] In one embodiment, Figure 2 As shown, step S1 includes:

[0061] Step S11: construct multiple acquisition windows for the video to be processed;

[0062] Each acquisition window includes multiple consecutive image frames;

[0063] Step S12: for each acquisition window, frame averaging is performed to obtain an average image;

[0064] Step S13: identifying a color block area from the average image, and using the color block area as a window subtitle area;

[0065] Step S14: constructing bounding rectangles for all window subtitle areas as subtitle areas.

[0066] Specifically, in order to achieve a better recognition effect in the subtitle area, in this embodiment, firstly, a plurality of acquisition windows are constructed with a preset step length for the video to be processed, each acquisition window has a predetermined length to cover continuous image frames, and the length of the acquisition window can be set longer to obtain multiple continuous image frames, some of which contain subtitle information. Then, for the acquisition window, frame averaging stack is performed to obtain an average image. Since the subtitle area contains text of a fixed style, the fonts will overlap into color blocks after frame averaging. Subsequently, the color block area is easily identified from the average image, and the color block area is used as the window subtitle area, and a bounding rectangle is constructed, and the image is cropped according to the bounding rectangle as the subtitle area on the frame.

[0067] In one embodiment, Figure 3 As shown, step S2 includes:

[0068] Step S21: for the subtitle area, respectively identify and obtain local semantic information;

[0069] Step S22: combining the local semantic information to obtain paragraph information;

[0070] Step S23: re-segment the paragraph information to obtain semantic information.

[0071] Specifically, in order to achieve a better semantic information recognition process, in this embodiment, first, the subtitle area in each frame of the image is input into the image text recognition model to obtain local semantic information. The local semantic information is the text part obtained by recognizing the text information contained in the image, and the interference background and style information in the image are removed. Then, since the original subtitles may split long sentences into two pages during the rendering process, it is necessary to reassemble the local semantic information to obtain paragraph information. Among them, the paragraph information can be segmented or not. In some subtitles with punctuation, end symbols such as Chinese period, English period, semicolon, etc. can be used as the basis for segmentation. Finally, the paragraph information is re-segmented to obtain semantic information, and re-mapped to the corresponding image frame according to the time sequence.

[0072] In one embodiment, Figure 4 As shown, step S21 includes:

[0073] Step S211: preprocessing the subtitle area to obtain a preprocessed image;

[0074] Step S212: segmenting the preprocessed image to obtain a word image;

[0075] Step S213: Recognize the word image to obtain local semantic information.

[0076] Specifically, in order to achieve a better recognition effect, in this embodiment, the subtitle area is first preprocessed, including noise removal, background filtering and grayscale processing. Subsequently, the preprocessed image is segmented using a grid template of corresponding size to obtain a word image, the grid template includes a plurality of grid lines, and the word image corresponding to the word area and the interval area between the words are determined by statistically analyzing the grids contained in the grayscale processed image, thereby removing the interval area. Finally, the word image is recognized to obtain local semantic information.

[0077] In one embodiment, Figure 5 As shown, after executing step S212 and before executing step S213, the following steps are included:

[0078] Step A212: determining the language type corresponding to the word image, and calling a language recognition template according to the language type;

[0079] In step S213, a language recognition template is used for recognition.

[0080] Specifically, in order to achieve better recognition effects, in this embodiment, after the words are split by the grid template, language prediction is performed for each word image obtained by the splitting, so as to determine the language type corresponding to the word image, and call the language recognition template according to the language type. Subsequently, in step S213, the corresponding language recognition template is used to recognize the word image, so as to obtain accurate recognition results.

[0081] In one embodiment, Figure 6 As shown, step S23 includes:

[0082] Step S231: segment the paragraph information into single sentences to obtain the single sentence semantics;

[0083] Step S232: re-map the time axis of the single sentence semantics to form semantic information.

[0084] Specifically, in order to achieve a better subtitle generation effect, in this embodiment, semantics are extracted through a semantic recognition model, such as a long short-term memory model, and then the paragraph information is segmented into single sentences according to the semantics to obtain single sentence semantics, so as to re-segment the reorganized paragraph into single sentences. Finally, the single sentence semantics are remapped to the time axis, and the image frames displayed by the single sentence are allocated, thereby forming complete semantic information.

[0085] In one embodiment, step S2 also includes a subtitle removal process, such as Figure 7 As shown, the subtitle removal process includes:

[0086] Step B21: for a single image frame, extract the subtitle area and adjacent images respectively;

[0087] Step B22: performing interpolation operation using adjacent images to obtain an interpolated image;

[0088] Step B23: Use the interpolated image to replace the adjacent image.

[0089] Specifically, in order to achieve a better image repair effect, in this embodiment, for each image frame with subtitles, the subtitle area is extracted respectively, and the adjacent adjacent image area is extracted according to the pixel position of the subtitle area. Subsequently, the adjacent image is interpolated to obtain an interpolated image, and the repair image corresponding to the area is fitted, and the interpolated image is used to replace the adjacent image to achieve a local repair effect.

[0090] In one embodiment, Figure 8 As shown, step S3 includes:

[0091] Step S31: Rendering a subtitle image according to semantic information and a pre-built subtitle template;

[0092] Step S32: Add the subtitle image to the subtitle-removed video to form a subtitle-updated video.

[0093] Specifically, in order to achieve better rendering effects, in this embodiment, for the target video to be generated, the corresponding subtitle style is first designed and a subtitle template is constructed. Then, the subtitle image is rendered according to the semantic information and the pre-constructed subtitle template, and then the subtitle image is added to the subtitle-free video to form a subtitle update video to achieve better rendering effects.

[0094] Those skilled in the art will appreciate that various aspects of the present invention, or possible implementations of various aspects, may be specifically implemented as systems, methods, or computer program products. Therefore, various aspects of the present invention, or possible implementations of various aspects, may take the form of complete hardware embodiments, complete software embodiments (including firmware, resident software, etc.), or embodiments of combined software and hardware aspects, all collectively referred to herein as "circuits," "modules," or "systems." In addition, various aspects of the present invention, or possible implementations of various aspects, may take the form of computer program products, which refer to computer instructions stored in a memory.

[0095] The memory may be a computer-readable signal medium or a computer-readable storage medium. Computer-readable storage media include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or apparatuses, or any suitable combination of the foregoing, such as random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable read-only memory (CD-ROM).

[0096] The processor in the computer reads the computer instructions stored in the memory, so that the processor can execute the functional actions specified in each step or the combination of steps in the flowchart; and generate a device for implementing the functional actions specified in each block or the combination of blocks in the block diagram.

[0097] It should be understood that the processor in the computer can be understood as one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components implemented to execute the aforementioned computer instructions.

[0098] Computer instructions can be executed completely on the user's local computer, partially on the user's local computer, as a separate software package, partially on the user's local computer and partially on a remote computer, or completely on a remote computer or server. It should also be noted that in some alternative embodiments, the functions noted in each step in the flow chart or each block in the block diagram may not occur in the order noted in the figure. For example, depending on the functions involved, two steps or two blocks shown in succession may actually be executed roughly simultaneously, or these blocks may sometimes be executed in reverse order.

[0099] The above are only preferred embodiments of the present invention, and are not intended to limit the implementation methods and protection scope of the present invention. Those skilled in the art should be aware that all solutions obtained by equivalent substitutions and obvious changes made using the description and illustrations of the present invention should be included in the protection scope of the present invention.

Claims

1. A multimedia subtitle recognition and display method, It is characterized in that include: Step S1: obtaining a video to be processed, and extracting a subtitle area from the video to be processed; Step S2: obtaining semantic information based on the subtitle region identification, and removing the subtitle region from the video to be processed to obtain a subtitle-free video; Step S3: adding subtitles to the subtitle-less video according to the semantic information to form a subtitle-updated video.

2. The multimedia subtitle recognition and display method according to claim 1, It is characterized in that The step S1 comprises: Step S11: constructing multiple acquisition windows for the video to be processed; Each of the acquisition windows includes a plurality of continuous image frames; Step S12: for each acquisition window, frame averaging is performed to obtain an average image; Step S13: identifying a color block area from the average image, and using the color block area as a window subtitle area; Step S14: constructing a bounding rectangle for all the window subtitle areas as the subtitle area.

3. The multimedia subtitle recognition and display method according to claim 1, It is characterized in that The step S2 comprises: Step S21: for the subtitle area, respectively identify and obtain local semantic information; Step S22: combining the local semantic information to obtain paragraph information; Step S23: re-segmenting the paragraph information to obtain the semantic information.

4. The multimedia subtitle recognition and display method according to claim 3, It is characterized in that The step S21 comprises: Step S211: preprocessing the subtitle area to obtain a preprocessed image; Step S212: segmenting the preprocessed image to obtain a word image; Step S213: recognizing the word image to obtain the local semantic information.

5. The multimedia subtitle recognition and display method according to claim 4, It is characterized in that After executing step S212 and before executing step S213, the following steps are included: Step A212: determining the language type corresponding to the word image, and calling a language recognition template according to the language type; In the step S213, the language recognition template is used for recognition.

6. The multimedia subtitle recognition and display method according to claim 3, It is characterized in that The step S23 comprises: Step S231: segment the paragraph information into single sentences to obtain single sentence semantics; Step S232: re-map the time axis of the single sentence semantics to form the semantic information.

7. The multimedia subtitle recognition and display method according to claim 1, It is characterized in that The step S2 further includes a subtitle removal process, which includes: Step B21: for a single image frame, extract the subtitle area and adjacent images respectively; Step B22: performing interpolation operation using the adjacent images to obtain an interpolated image; Step B23: using the interpolated image to replace the adjacent image.

8. The multimedia subtitle recognition and display method according to claim 1, It is characterized in that The step S3 comprises: Step S31: Rendering a subtitle image according to the semantic information and a pre-built subtitle template; Step S32: Add the subtitle image to the subtitle-removed video to form a subtitle-updated video.

Images