Video subtitle extraction method and device and electronic equipment

Abstract

The invention provides a video subtitle extraction method, a video subtitle extraction device and electronic equipment, after image data and audio data are extracted from an original video file, in order to accurately extract diversified subtitles from a video image, multi-scale subtitle image feature extraction is performed on the image data, and compared with traditional OCR (Optical Character Recognition) identification, the image subtitle extraction efficiency is improved. Meanwhile, voice recognition is performed on the audio data in the original video file, and the obtained voice recognition result is not influenced by the image quality, so that the defect of the subtitle recognition result extracted from the image data when the video image quality is poor can be overcome, and the accuracy of the subtitle recognition result is improved. Therefore, the caption file generated after the caption recognition result is corrected based on the voice recognition result can reliably meet the auxiliary requirements of audiences on video understanding or other processing requirements on the video captions.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and more specifically to a video subtitle extraction method, device and electronic equipment. Background Art

[0002] With the rapid development of multimedia technology, videos have been widely used in various fields such as the Internet, television broadcasting, education and training. Video subtitles can help viewers better understand and obtain information in the video, especially for the hearing-impaired and foreign language learners, video subtitles have become an important auxiliary tool.

[0003] However, the current video subtitle extraction method based on OCR (Optical Character Recognition) technology is often unable to accurately extract video subtitles due to factors such as diverse video subtitle styles, font sizes, or low video image quality, thereby affecting the auxiliary effect of video subtitles on viewers. Summary of the invention

[0004] In view of the above problems, this application provides the following solutions:

[0005] The first aspect of the present application provides a video subtitle extraction method, the video subtitle extraction method comprising:

[0006] Get the audio data and image data in the original video file;

[0007] Performing speech recognition on the audio data to obtain a speech recognition result of the audio data;

[0008] Performing multi-scale subtitle feature extraction on the image data, and obtaining a subtitle recognition result of the image data using the extracted multi-scale subtitle features;

[0009] Based on the speech recognition result, the subtitle recognition result is corrected to generate a subtitle file matching the original video file.

[0010] In a possible implementation, the correcting the subtitle recognition result based on the speech recognition result to generate a subtitle file matching the original video file includes:

[0011] Using the speech text in the speech recognition result, correcting the initial subtitle text in the subtitle recognition result to obtain a target subtitle text;

[0012] The target subtitle text is time-aligned with each video frame in the original video file to generate a subtitle file synchronized with the original video file.

[0013] In a possible implementation, the using the speech text in the speech recognition result to correct the initial subtitle text in the subtitle recognition result to obtain the target subtitle text includes:

[0014] Matching the initial subtitle text in the subtitle recognition result with the speech text in the speech recognition result, and determining a first text and a second text that are inconsistent in matching; the first text belongs to the initial subtitle text, and the second text belongs to the speech text;

[0015] If a first confidence level for the first text in the subtitle recognition result is less than a second confidence level for the second text in the speech recognition result, the first text in the initial subtitle text is corrected into the second text to obtain a target subtitle text.

[0016] In a possible implementation, the correcting the subtitle recognition result based on the speech recognition result to generate a subtitle file matching the original video file further includes at least one of the following implementation methods:

[0017] Determining that the initial subtitle text in the subtitle recognition result matches the speech text in the speech recognition result, and determining the initial subtitle text as the target subtitle text;

[0018] Determine a third text in the initial subtitle text in the subtitle recognition result that matches the speech text in the speech recognition result, and maintain the third text in the initial subtitle text unchanged.

[0019] In a possible implementation, extracting multi-scale subtitle features from the image data and obtaining a subtitle recognition result of the image data using the extracted multi-scale subtitle features includes:

[0020] Inputting the image data into a trained subtitle recognition model, and extracting subtitle features of different scales from the image data through multi-scale parallel octave convolution operations;

[0021] splicing the subtitle features of different scales to obtain target subtitle features;

[0022] The subtitle recognition result of the image data is obtained by decoding according to the target subtitle feature.

[0023] In a possible implementation, the training process of the subtitle recognition model includes:

[0024] The convolutional layer in the optical character recognition model is updated to a feature extraction network containing multi-scale parallel convolution kernels to obtain an initial subtitle recognition network; the feature extraction network is an Inception deep learning convolutional neural network constructed using multi-scale octave convolutional layers;

[0025] Acquire a subtitle data set; each subtitle image in the subtitle data set contains different subtitle attributes, and the subtitle attributes include one or more combinations of subtitle style, subtitle font size, and color;

[0026] Performing enhancement processing on the subtitle image, adding the enhanced subtitle image to the subtitle dataset, to obtain an enhanced subtitle dataset;

[0027] The initial subtitle recognition network is trained using each subtitle image in the enhanced subtitle data set to obtain a subtitle recognition model.

[0028] In a possible implementation, extracting subtitle features of different scales from the image data through a multi-scale parallel octave convolution operation includes:

[0029] Decomposing the feature map of the image data into high-frequency feature maps and low-frequency feature maps of different scales through multi-scale parallel octave convolution operations; the high-frequency feature map includes the subtitle outline features in the image data, and the low-frequency feature map includes the background features in the image data;

[0030] The high-frequency feature maps and the low-frequency feature maps of different scales are interactively integrated to obtain subtitle features of different scales in the image data.

[0031] A second aspect of the present application provides a video subtitle extraction device, the video subtitle extraction device comprising:

[0032] A data acquisition module, used for acquiring audio data and image data in the original video file;

[0033] A speech recognition module, used to perform speech recognition on the audio data to obtain a speech recognition result of the audio data;

[0034] A subtitle recognition module, used to extract multi-scale subtitle features from the image data, and obtain a subtitle recognition result of the image data using the extracted multi-scale subtitle features;

[0035] The subtitle file generation module is used to correct the subtitle recognition result based on the speech recognition result to generate a subtitle file matching the original video file.

[0036] A third aspect of the present application provides an electronic device, the electronic device comprising: at least one communication element, at least one memory and at least one processor, wherein:

[0037] The communication element is used to receive the original video file;

[0038] The processor is used to load and execute multiple computer instructions stored in the memory to implement the following steps:

[0039] Acquire audio data and image data in the original video file;

[0040] Performing speech recognition on the audio data to obtain a speech recognition result of the audio data;

[0041] Performing multi-scale subtitle feature extraction on the image data, and obtaining a subtitle recognition result of the image data using the extracted multi-scale subtitle features;

[0042] Based on the speech recognition result, the subtitle recognition result is corrected to generate a subtitle file matching the original video file.

[0043] It can be seen that in the face of a video with diverse subtitle attributes, poor picture quality, and complex background waiting to be extracted, that is, an original video file, in the video subtitle extraction method proposed in this application, the electronic device will extract image data and audio data from the original video file. In order to improve the accuracy of text recognition and extract diverse subtitles from the video image, multi-scale subtitle image feature extraction will be performed on the image data to ensure the accuracy and reliability of the subtitle recognition result obtained thereby, and speech recognition will be performed on the audio data in the original video file. Since the obtained speech recognition result is not affected by the image quality, it can make up for the deficiencies of the subtitle recognition result extracted from its image data when the video quality is poor, thereby improving the accuracy and completeness of the compensated video subtitles, so that the subtitle file generated after the subtitle recognition result is corrected based on the speech recognition result can reliably meet the audience's auxiliary needs for video understanding or other processing needs for video subtitles. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0045] Figure 1 A schematic diagram of the hardware structure of the electronic device proposed in Embodiment 1 of the present application;

[0046] Figure 2A schematic diagram of a system architecture for an optional application scenario of the video subtitle extraction method proposed in this application;

[0047] Figure 3 This is a schematic diagram of the hardware structure of the electronic device proposed in Example 2 of the present application;

[0048] Figure 4a A flowchart of a video subtitle extraction method proposed in Example 1 of the present application;

[0049] Figure 4b A schematic diagram of an octave convolution operation used to implement subtitle feature extraction in a video subtitle extraction method proposed in an embodiment of the present application;

[0050] Figure 5 A flowchart of a video subtitle extraction method proposed in Embodiment 2 of the present application;

[0051] Figure 6 A schematic diagram of a structure for implementing a multi-scale convolution operation in a feature extraction network in a subtitle recognition model used in a video subtitle extraction method proposed in an embodiment of the present application;

[0052] Figure 7 A flowchart of a video subtitle extraction method proposed in Embodiment 3 of the present application;

[0053] Figure 8 A schematic diagram of the structure of a video subtitle extraction device proposed in an embodiment of the present application. DETAILED DESCRIPTION

[0054] The embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation mode of the present application are only used to explain the specific embodiments of the present application and are not intended to limit the present application. The embodiments of the present application are described below in conjunction with the drawings. It is known to those of ordinary skill in the art that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0055] It should be understood that the terms "first", "second" etc. that the present disclosure relates to are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable in appropriate circumstances, and this is only to describe the distinction mode adopted by the objects of the same attribute when describing in the embodiments of the present application. Belonging to "if" can be interpreted as "at the time of ... " or "when ... " or "in response to determining" etc., can be understood according to the context. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusion, so that the process, method, product or equipment comprising a series of units need not be limited to those units, but can include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0056] In addition, the collection, use and processing of the data involved in this disclosure (including but not limited to data used for analysis, processing / storage / display, such as video images, video subtitles and video voice, etc.) shall, before using the technical solutions disclosed in the embodiments of this disclosure, inform the user of the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations and obtain the user's authorization.

[0057] The present application can be applied to various fields such as the Internet, television broadcasting, live broadcasting, education and training, and can be but not limited to being applied to applications with video processing functions or cloud services provided by cloud-side servers. The electronic device suitable for the video subtitle extraction method proposed in the present application will be introduced in detail below in conjunction with the accompanying drawings.

[0058] Reference Figure 1 , is a schematic diagram of the hardware structure of the electronic device proposed in the first embodiment of the present application. Combined with the above analysis, the electronic device 100 can be a terminal device or a server, so as to implement the video subtitle extraction method proposed in the embodiment of the present application through the electronic device.

[0059] In the actual application of the present application, the terminal device serving as the electronic device 100 can be installed with an application having a video processing function. The application or the login application webpage can provide a visual interface for the user to input relevant parameters on the interface (such as selecting the original video file of the subtitles to be extracted, the video extraction trigger operation or the subsequent processing operation based on the video subtitles, etc.). The terminal device can obtain corresponding processing results based on the received parameters, such as obtaining the original video file of the subtitles to be extracted from each video source, generating a subtitle file in a preset subtitle format using the target subtitle text extracted from the original video file, or filtering the hot content based on the extracted target subtitle text, or converting and generating a subtitle file in an output format suitable for special audiences (such as the hearing-impaired or foreign language learners), etc.

[0060] It should be understood that in one possible implementation, Figure 2 The system architecture diagram shown is applicable to an optional application scenario of the video subtitle extraction method proposed in the present application. The terminal device can also send the received user input parameters to the server. The server can obtain the processing results (such as the original video file, or the subtitle file of the original video file obtained by executing the video subtitle extraction method proposed in the present application, etc.) based on the received parameters, and feed the processing results back to the terminal device. In this embodiment, the terminal device can cooperate with the server to implement, and the application scenario can be determined according to the actual situation, and is not limited to Figure 2 Content shown.

[0061] The terminal device may be a smart phone, a tablet computer, a wearable device, an augmented reality (AR) device / virtual reality (VR) device, a vehicle-mounted device, a laptop computer, a robot, or a smart medical / transportation device, etc. The embodiment of the present application does not limit the product form of the terminal device. Figure 2 The product form of the terminal device shown is used as an example for description. The server may include one or more servers ( Figure 2 Only one server is used as an example for explanation), as analyzed above, the server can provide the method proposed in the embodiment of the present application for one or more terminal devices.

[0062] In combination with the above analysis, in the present application, the electronic device 100 implementing the video subtitle extraction method proposed in the embodiment of the present application is either a terminal device or a server, such as Figure 1 As shown, it may include at least one communication element 110, at least one memory 120 and at least one processor 130, and at least one communication element 110, at least one memory 120 and at least one processor 130 may communicate with each other via a bus. The bus may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 1 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0063] The communication element 110 can be used to receive the original video file. In practical applications, the communication element 110 may include a wireless communication element that supports wireless communication methods such as wifi, 5G / 6G, Global System of Mobile communication (GSM), General Packet Radio Service (GPRS), Bluetooth and / or near-field channel methods to achieve data transmission between the electronic device and other devices (such as databases, video sources, etc.), such as transmitting video files, subtitle files, etc. This application does not elaborate on the structure of each wireless communication element and its communication transmission mechanism.

[0064] It can be understood that the communication element 110 can also include one or more wired communication elements that support wired communication methods, such as a general-purpose input / output (GPIO) interface, a USB interface, a universal asynchronous receiver / transmitter (UART) interface, etc., to achieve data transmission between various components inside the electronic device. This application does not limit the type and quantity of wired communication elements, which can be determined according to the circumstances.

[0065] The memory 120 can be used to store computer instructions and data, and may include an instruction storage area and a data storage area. The data storage area may store various data, such as various model codes, video files, subtitle files, and various intermediate data generated during the video subtitle extraction process; the instruction storage area may store software units such as computer instructions required for an operating system, an application, and at least one function (such as image / audio data processing functions). In the embodiment of the present application, the memory 120 may provide the processor 130 with a plurality of computer instructions for implementing the video subtitle extraction method of the embodiment of the present application, as well as data or other software and hardware resources required during the execution of the video subtitle extraction method.

[0066] The processor 130, as the control center of the electronic device, can use various interfaces and lines to connect various parts of the entire electronic device, and implement the various steps of the video subtitle extraction method proposed in the embodiment of the present application by loading and executing multiple computer instructions stored in the memory. The implementation process can refer to the description of the corresponding part of the method embodiment below. Other components of the electronic device (such as communication elements, one or more input components and output components, etc.) can also be called to implement corresponding functions.

[0067] In the embodiment of the present application, the memory 120 may include a volatile memory, such as a random access memory (RAM), and may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard drive drive (HDD) or a solid state drive (SSD). The processor 130 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), a digital signal processor (DSP), an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). The types of the memory 120 and the processor 130 included in the electronic device and their working principles can be determined according to actual functional requirements.

[0068] It should be understood that Figure 1 The structure of the electronic device shown does not constitute a limitation on the electronic device in the embodiment of the present application. In practical applications, the electronic device may include Figure 1 More or less components as shown, or some components may be combined. In particular, when the electronic device 100 is a terminal device, such as Figure 3 As shown, the terminal device may also include at least one input component such as a touch sensing unit for sensing touch events on a touch display panel, a keyboard, a mouse, a camera, a microphone, etc.; at least one output component such as a display, a speaker, a vibration mechanism, an indicator light, etc.; an antenna, a radio frequency unit; a variety of sensors; a power module, etc. Figure 3 The listed components are not shown. The hardware structure can be determined according to the terminal device type and its functional requirements, and this application does not list them one by one here.

[0069] The video subtitles involved in this application are the subtitles in the video, which refer to the text displayed on the video screen during video playback. The video subtitles can present the voice of the speaker (such as a character or animation object) in the video in the form of text. They are embedded in the video in the form of images and are an integral part of the video content.

[0070] At present, in many scenarios, it is necessary to extract subtitles from videos in order to understand the content expressed in the video or conduct hot trend analysis through the extracted video subtitles. However, as described in the background technology part, there are technical problems in the method of extracting video subtitles based on the traditional OCR (Optical Character Recognition). In this regard, it is proposed to detect the subtitle area of ​​the video image (such as any implementation method such as color segmentation, edge detection and morphological operation), and then perform preprocessing such as denoising, binarization and image enhancement on the detected subtitle area image to increase the clarity and resolution of the subtitle area image and the contrast between the subtitle area image and the background area image. Although it can improve the OCR recognition accuracy of static images, it still faces challenges such as font diversity, complex background, dynamic changes, etc. when processing video subtitles, resulting in unsatisfactory subtitle recognition effect.

[0071] In order to improve the reliability and accuracy of video subtitle extraction, the video subtitle extraction method proposed in the embodiment of the present application improves the OCR recognition accuracy by improving the OCR recognition network, and combines the speech recognition results of the audio data in the video to make up for the shortcomings of OCR recognition, such as the influence of factors such as video quality and subtitle background complexity, which leads to inaccurate or incomplete OCR recognition results, etc., to further improve the accuracy of subtitle extraction, thereby reliably meeting the auxiliary needs of special audiences for video subtitles. The video subtitle extraction method of the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0072] Reference Figure 4a , is a flow chart of a video subtitle extraction method proposed in the first embodiment of the present application, which can be applied to electronic devices such as Figure 4a As shown, the video subtitle extraction method may include but is not limited to the following steps:

[0073] Step S401, obtaining audio data and image data in the original video file;

[0074] In the embodiment of the present application, the original video file refers to the video file from which subtitles are to be extracted, which can be any video file from which subtitles need to be extracted, such as any product advertising video, film and television drama video, recorded video or teaching video, etc. The present application does not impose any restrictions on the video type of the video file.

[0075] In actual applications, the electronic device can obtain the subtitles from a database or video source that stores one or more video files to be extracted, or it can obtain the video file input by the user through the interface of the electronic device (i.e., the terminal device), or it can act as a server of the electronic device to obtain the video file sent by the terminal device, etc. This application does not limit the method of obtaining the original video file, and the electronic device can flexibly implement it according to actual needs.

[0076] Combined with the above description of the technical solution of the present application, the present application will identify the corresponding types of data in the same original video file based on text recognition technology and speech recognition technology, that is, identify subtitle information from the same original video file through different subtitle extraction methods. Therefore, in order to execute the different subtitle extraction methods, the corresponding types of data corresponding to the processing of each subtitle extraction method can be first extracted from the same original video file, that is, image data suitable for text recognition technology, and audio data suitable for speech recognition technology. The present application does not limit the implementation method of how to extract audio data and image data from the video.

[0077] It is understandable that any original video file usually includes a series of continuous video frames. Step S401 can be the corresponding frame audio data and image data extracted from each video frame. In order to ensure the coherence and integrity of the subtitle content in the extracted audio data and image data and avoid missing the speaker's speech content, the audio data and image data of each video frame can be extracted based on the frequency of the original video file, thereby improving the reliability of obtaining the audio data and image data in the video.

[0078] In another possible implementation, due to the different speaking characteristics of different speakers, not all video frames contain audio data and subtitles. In order to reduce the amount of calculation, this embodiment can also set a corresponding data acquisition method based on the frequency and speaking characteristics of the original video file, and perform the acquisition operation of step S401 accordingly, thereby minimizing the image data extraction operation of the video frames without subtitles and the audio data extraction operation of the video frames without audio, thereby reducing the resource consumption generated by the data extraction process and improving the data acquisition efficiency.

[0079] For example, if the advertising slogan spoken by the speaker of some advertising videos is continuous for 2 / 3 / 5 seconds, etc., and the rest of the time is only an image without subtitles to be extracted, etc., the frame number can be set at intervals for the original video file containing a single speaker in combination with the video frequency, and continuous audio data and image data can be extracted from multiple continuous video frames; the characters in a certain film or TV series video will change with the plot, and the speaking time of the corresponding characters is determined according to the plot requirements, and the audio data and image data in the video frames with corresponding timestamps are extracted accordingly. The present application does not limit the implementation method of step S401.

[0080] Step S402, performing speech recognition on the audio data to obtain a speech recognition result of the audio data;

[0081] Following the above analysis, the present application can perform speech recognition on continuous audio data (i.e., audio files or audio data streams) extracted from the original video files based on Automatic Speech Recognition (ASR) technology, that is, convert the audio data into text data to obtain corresponding speech recognition results, such as speech text matching the audio data. The present application does not impose any restrictions on the speech recognition method.

[0082] In a possible implementation, the present application can call a voice assistant of an electronic device or trigger an external smart speaker or other device of the electronic device to convert the input audio data into text output based on artificial intelligence technology. This embodiment can call a speech recognition model to implement step S402, that is, input the acquired audio data into the speech recognition model, and output the corresponding speech recognition result. The speech recognition model can be an ASR model based on a recursive neural network (RNN) or a transform neural network (Transformer, also called a transformer). The network structure and working principle of the speech recognition model are not described in detail in this application, and the existing speech recognition model can be flexibly called according to actual needs.

[0083] Step S403, extracting multi-scale subtitle features from the image data, and obtaining a subtitle recognition result for the image data using the extracted multi-scale subtitle features;

[0084] In an embodiment of the present application, in order to accurately and reliably identify subtitles with various attributes such as style and font size in a video, the present application proposes to perform multi-scale subtitle feature extraction on each frame of image data, such as using convolution kernels of different scales to extract subtitle features from different scales of input image data in the same network layer. In this way, when faced with image data containing subtitles with different attributes such as style or font size, the present application can reliably and accurately identify subtitle features therefrom, thereby ensuring that the subtitle recognition results obtained thereby are accurate.

[0085] In a possible implementation, the present application can perform step S403 based on a trained subtitle recognition model, that is, input each frame of image data into the subtitle recognition model, and extract subtitle features of different scales from the image data through multi-scale parallel octave convolution operations. Compared with traditional convolution, the octave convolution (OctConv) can reduce the amount of calculation and memory usage without losing accuracy, thereby improving the efficiency and scalability of the subtitle recognition model.

[0086] In the above multi-scale parallel octave convolution operation process, refer to Figure 4b The octave convolution shown, X represents the feature map of the input image data, whose size is: W×H×C in, decompose the feature map X into a low-frequency feature map Xl and a high-frequency feature map Xh according to the ratio of the hyperparameter α. The sizes of these two feature maps are as follows: Figure 4b As shown, in the present application, the high-frequency feature map includes the subtitle contour features in the image data, and the low-frequency feature map includes the background features in the image data, that is, non-subtitle features. By interactively integrating high-frequency feature maps and low-frequency feature maps of different scales (that is, information interaction), the subtitles and background can be better distinguished, the recognition accuracy can be improved, and the accuracy and reliability of the subtitle features of different scales in the image data obtained thereby can be ensured.

[0087] Before the above interactive integration process, the low-frequency feature map can be downsampled first, such as Figure 4b As shown in the figure, its resolution is halved to increase its receptive field, and then the high- and low-frequency features are self-updated and interactively integrated. F(hh) and F(ll) represent the self-update of high-frequency features and low-frequency features respectively, and F(hl) and F(lh) represent the interactive integration of high- and low-frequency features. Both output high-frequency feature maps Yh and low-frequency feature maps Yl. Compared with traditional convolution, it enhances the ability of long-distance contextual information of input image data, improves the subtitle recognition performance of the model, and reduces the amount of calculation.

[0088] It can be seen that the present application can combine octave convolution to construct a feature extraction network capable of extracting subtitle feature maps of different scales, reliably extract subtitle features of multiple scales from the corresponding frame image, and quickly and accurately output the subtitle recognition results of the corresponding frame image data. It should be noted that the present application does not restrict the structure of the feature extraction network with multi-scale image feature extraction capabilities. The different sizes of multiple parallel convolution kernels contained in the convolution layer in the feature extraction network can be flexibly configured according to actual conditions, and combined with other network layers to improve the traditional OCR recognition model, thereby obtaining the subtitle recognition model required by the present application.

[0089] Step S404: based on the speech recognition result, correct the subtitle recognition result to generate a subtitle file that matches the original video file.

[0090] In the application of video subtitle extraction, due to the complex video recording environment or the scene of the video itself, the speech recognition results of the video are easily disturbed by the background sound and environmental noise in the video, making its recognition accuracy lower than the accuracy of the text recognition results of the image in the video. Therefore, in order to reduce the correction workload, this application proposes to correct the subtitle recognition results and determine the subtitles that match the original video file. However, in the case of a relatively clean voice environment (there is little noise interfering with the voice recognition subtitles, and the video scene can even be ignored), and the image quality (such as resolution or clarity) is poor, and the subtitle background is more complex, even if a multi-scale subtitle feature extraction method is adopted, it may also be due to interference that the extracted individual or adapted scale subtitle features are inaccurate or incomplete, reducing the accuracy of the subtitle recognition results, because the speech recognition results are more accurate at this time. Therefore, this application proposes to use the speech recognition results to correct the subtitle text that is not accurate enough in the subtitle recognition results, and improve the accuracy of the corrected subtitle recognition results.

[0091] It can be seen that the present application proposes a speech recognition result that is not affected by image quality to make up for the shortcomings of OCR recognition, especially in the face of a video with diverse subtitle attributes, poor image quality, and complex background waiting to extract subtitles, that is, an original video file. After extracting image data and audio data from it, in order to improve the accuracy of text recognition, a variety of subtitles are extracted from the video image. Multi-scale subtitle image feature extraction will be performed on the image data to ensure the accuracy and reliability of the subtitle recognition result obtained thereby. At the same time, the present application will perform speech recognition on the audio data in the original video file. Since the speech recognition process is independent of the image data, the quality of the video will not affect the speech recognition result, so that it can make up for the shortcomings of the subtitle recognition result extracted from its image data when the video quality is poor, and improve the accuracy and completeness of the compensated video subtitles, so that the generated subtitle file can reliably meet the audience's auxiliary needs for video understanding or other processing needs for video subtitles, such as generating a subtitle file in a text format that can be recognized by special groups such as the deaf and dumb.

[0092] Reference Figure 5 , is a flow chart of a video subtitle extraction method proposed in Embodiment 2 of the present application, which can describe an optional refinement of the video subtitle extraction method proposed above, such as Figure 5 As shown, the video subtitle extraction method proposed in this embodiment may include:

[0093] Step S501, obtaining each frame of video image and audio data in the original video file;

[0094] Step S502, preprocessing each frame of video image;

[0095] Step S503, extracting the subtitle area from each frame of the preprocessed video image to obtain image data corresponding to the subtitle area;

[0096] The implementation method for obtaining image data and audio data from the original video file includes but is not limited to the method described in this embodiment. During the implementation process, a suitable video processing tool can be selected to correctly and completely extract the image data and corresponding audio data of each frame of video where the subtitles are located in the original video file.

[0097] Based on this, the present application can input the acquired original video file into a video processing tool, such as the open source OpenCV (cross-platform computer vision and computer learning software library), etc., to decompose the original video file into single-frame video images, and output a series of continuous frame video images, or multiple groups of video images containing subtitles to be extracted (which can correspond to the implementation method of eliminating video images without subtitles). Each group of video images usually contains image data such as multiple continuous frame video images. In order to distinguish the subtitles of each frame of video images and facilitate the subsequent audio time alignment with the corresponding video frame, in the process of extracting each frame image data of the original video file, the corresponding timestamp is recorded, such as the frame number of the video frame where the each frame image data is located or the playback time in the entire original video file. The present application does not limit the recording method of the timestamp.

[0098] In one possible implementation, in order to reduce the interference of non-subtitle area images (such as non-text background area images, and other text area images that do not belong to subtitles, etc.) in each frame of video image on screen recognition, the present application proposes that the subtitle area can be first detected from each decomposed frame of video image. This can be achieved through appropriate image processing algorithms, such as image segmentation algorithms, edge detection algorithms, color segmentation algorithms, etc., performing color segmentation or edge detection on adjacent frames of video images, and comparing the detection results between video frames, so as to accurately determine the subtitle area in each frame of video image, extract image data of the subtitle area, so that the image data retains the complete subtitle features while reducing noise data as much as possible, thereby laying a solid foundation for subsequent text recognition.

[0099] In a possible implementation, in order to further improve the recognition reliability and efficiency of the subtitle area image data and reduce the problem of low OCR recognition accuracy due to poor image quality, the embodiment of the present application can first pre-process each frame of video image decomposed from the original video file, such as one or more combined pre-processing methods such as denoising, binarization and enhancement, to enhance the contrast between the subtitle area and the non-subtitle area (such as the background area) in the image, and then extract the image data of the subtitle area from each pre-processed frame of video image according to the method described above. The present application does not limit the image pre-processing method, and the pre-processing method that can enhance the image quality or the reliability of target (i.e., subtitle) recognition is applicable to the embodiment of the present application, and the present application does not give detailed examples one by one.

[0100] For the audio data in the original video file, this application can call audio processing tools, such as FFmpeg (a set of open source computer programs that can be used to record, convert digital audio and video, and convert them into streams), etc., and use its conversion and streaming audio and video functions to extract the audio signal in the original video file and output complete audio data, such as an audio file or an audio data stream, which records the complete video subtitle content in audio format. This application does not limit the implementation method of how to extract audio from video.

[0101] Step S504, inputting the audio data into a speech recognition model to obtain a speech recognition result of the audio data;

[0102] Step S505, inputting the image data into the trained subtitle recognition model, and extracting subtitle features of different scales from the image data through multi-scale parallel octave convolution operations;

[0103] In order to adapt to the diversity of attributes such as style, font size and color of subtitles in videos, and reduce or even eliminate the adverse effects of diverse subtitles on OCR recognition accuracy, this application will improve the OCR recognition model, retrain a subtitle recognition model that adapts to the diversity of video subtitles, and improve the recognition ability and robustness of the subtitle recognition model.

[0104] Combined with the above analysis, the present application can use octave convolution to replace the traditional convolution layer to achieve reliable extraction of subtitle features of multiple sizes. In one possible implementation, the present application calls the existing OCR recognition model and updates the single convolution layer (i.e., the feature extraction network layer used to extract image features) to a feature extraction network with multi-scale parallel convolution kernels, so that the trained subtitle recognition model has the ability to recognize text from videos with subtitles of various attributes, and can simultaneously adapt to the diversity of attributes such as font, font size, color, and style of video subtitles, thereby improving the accuracy of video subtitle OCR recognition.

[0105] Preferably, the feature extraction network can be an Inception deep learning convolutional neural network constructed using multi-scale octave convolution, that is, the octave convolution is used to replace the traditional convolution layer in the Inception network architecture, so as to distinguish the subtitle features and background features in the image data, and extract the accuracy and reliability of the required subtitle features. Among them, the Inception network architecture can flexibly select the structure of the Inception network series according to the actual situation, and is not limited to the network architecture described in this application.

[0106] In some embodiments, the above Inception network architecture can be at least one such as Figure 6 The Inception network connection structure shown in the figure needs to be explained that the convolution in the Inception network is an octave convolution, that is, the present application integrates the octave convolution into the Inception network, so as to extract multi-scale subtitle features at a finer scale, and can also increase the receptive field of the model by performing convolution operations at different scales, capture global features in the image data, and assist in identifying subtitle features to improve the accuracy of subtitle extraction. It can also reduce the number of model parameters and improve the efficiency of subtitle extraction on the basis of maintaining or improving the performance of the model.

[0107] It should be noted that the various Inception network structures in the above-mentioned feature extraction network of this application include but are not limited to Figure 6 The Inception V1 network architecture shown ( Figure 6 The convolution in the figure represents an octave convolution). Other networks in the Inception series can also be used. For example, to accelerate the training of the Inception network, a residual connection can be connected in parallel to connect a blank branch from the input to the output, thus obtaining a feature extraction network layer similar to the Inception V4 network structure. The convolution kernels of different scales include but are not limited to octave convolutions of three scales: 1×1 convolution kernel, 3×3 convolution kernel, and 5×5 convolution kernel, which can be flexibly configured as needed.

[0108] After improving each single convolutional layer of the OCR recognition model according to the method described above to obtain the initial subtitle recognition network, a subtitle data set containing diverse subtitles can be used, that is, each subtitle image contains different subtitle attributes (such as one or more combinations of subtitle style, subtitle font size and color, etc.), to improve the diversity of training data for training the initial subtitle recognition network, and obtain a subtitle recognition model with diversified subtitle recognition capabilities; and because the Inception deep learning convolutional neural network constructed by multi-scale octave convolution replaces the convolutional layer of the traditional OCR recognition model, the respective characteristics of the octave convolution and the Inception network architecture are utilized to efficiently and reliably distinguish between subtitles and background in the image, thereby improving the efficiency and accuracy of subtitle recognition in the image, thereby improving the performance and generalization ability of the subtitle recognition model required by the present application.

[0109] Preferably, the present application can also enhance the subtitle images in the subtitle data set to enrich the subtitle data set and enhance the robustness of the training model. Therefore, after obtaining the subtitle data set, the subtitle images therein can be subjected to one or more enhancement processes such as rotation, scaling, and adding noise, and the obtained enhanced subtitle images are added to the subtitle data set to obtain an enhanced subtitle data set, and then the individual subtitle images in the enhanced subtitle data set are used to train the initial subtitle recognition network to obtain a subtitle recognition model. The present application does not limit the model training method. For example, the subtitle image can be input into the initial subtitle recognition network, and multi-scale subtitle recognition can be performed through octave convolution operations to quickly and reliably identify the subtitle text in the subtitle image. After calculating the prediction loss of the subtitle text, the initial subtitle recognition network is trained by minimizing the prediction loss to obtain the desired subtitle recognition model.

[0110] Step S506, concatenating subtitle features of different scales to obtain target subtitle features;

[0111] Step S507, decoding to obtain a subtitle recognition result of the image data according to the target subtitle feature;

[0112] According to the relevant description of the subtitle recognition model described above, after initially extracting subtitle features of different scales from the input image data, the subtitle features of different scales can be spliced ​​so that the obtained target subtitle features form a richer feature expression to improve the accuracy of subtitle recognition.

[0113] It should be noted that in the process of feature extraction of image data, Figure 6 The Inception network shown in Figure 2 can perform multi-scale parallel octave convolution operations and pooling operations in parallel, such as Figure 6The maximum pooling layer of 3×3 scale shown in the figure can also be used as an average pooling layer. Optionally, in order to reduce the computational complexity of high-scale convolution kernels, Figure 6 By adding a 1×1 convolution kernel before the higher-scale convolution kernel in parallel and after the pooling layer, or decomposing the high-scale convolution kernel into multiple layers, the purpose of reducing the amount of calculation and improving network performance is achieved. This application does not limit the structure of each network layer contained in the feature extraction network in the model.

[0114] It can be seen that under different network structures, in the process of extracting features from input image data through parallel convolution kernels of different scales, the image data can be pooled through the pooling layer to reduce the amount of calculation and avoid overfitting. After that, the information between different channels (such as each convolution layer and pooling layer) is fused through cascade filters, that is, the extracted subtitle features and pooling features of different scales are fused and reduced in dimension, reducing the thickness of the extracted subtitle features and thus reducing the amount of calculation.

[0115] In the embodiment of the present application, since the subtitle area image data in each video frame is input into the subtitle recognition model, the interference of non-subtitle area information on the subtitle recognition process is reduced, and the subtitle features and background features in the image data are quickly and accurately distinguished, which is conducive to improving the accuracy and efficiency of subtitle recognition. In some other embodiments, if each frame of video image directly decomposed from the original video file is input into the subtitle recognition model, it is usually necessary to first detect the subtitle area in each frame of video image, and then perform multi-scale subtitle feature extraction, i.e., character recognition, on the subtitle area image to obtain rich and accurate target subtitle features, thereby predicting the subtitle recognition results of each frame of video image.

[0116] Step S508, using the speech text in the speech recognition result, correcting the initial subtitle text in the subtitle recognition result to obtain the target subtitle text;

[0117] As analyzed above for the technical solution of this application, if the video quality is poor and the contrast between the subtitles and the background is low, due to the limitations of OCR recognition, the subtitle text output by the subtitle recognition model may still contain errors or omissions, which will lead to inaccurate OCR recognition results. In this regard, this application will use the speech text in the speech recognition results that is not affected by the video image, that is, the text information converted from the speech signal in the same video, to correct the inaccurate initial subtitle text in the subtitle recognition results, and accurately and completely obtain the target subtitle text of the video. This application does not limit the correction implementation method.

[0118] It should be noted that in order to ensure the accuracy of the correction, the above-mentioned speech text and the initial subtitle text can be time-aligned, and then the speech text and the initial subtitle text can be compared and corrected to ensure the accuracy of the corrected target subtitle text. Therefore, the speech text in the speech recognition result of the present application can be a speech text sequence containing a timestamp, and the initial subtitle text in the subtitle recognition result can be an initial subtitle text sequence containing a timestamp, so as to achieve the comparison of the text with the same timestamp in the two text sequences. If the speech text with the same timestamp is used to correct the inaccurately recognized initial subtitle text, it is not limited to the correction implementation method described in this embodiment.

[0119] Step S509: Time-align the target subtitle text with each video frame in the original video file to generate a subtitle file synchronized with the original video file.

[0120] According to the method described above, the target subtitle text in the original video file is extracted, such as a subtitle text sequence with different timestamps. The subtitle file can be generated in the required subtitle format (such as SRT (SubRip Text) format or ASS (Advanced SubStation Alpha) format, etc.) by time alignment with each video frame in the original video file. The subtitle file can be directly used in a video player or embedded in a video file to store the subtitle file to meet the actual auxiliary needs of current viewers for video subtitles.

[0121] In summary, the embodiment of the present application decomposes and extracts the image data of the subtitle area in each frame of video image and the audio data in each frame of video from the original video file, and then inputs the image data of each frame into a trained subtitle recognition model with the ability to extract multi-scale subtitle features in parallel (i.e., adapt to the diversity of video subtitles). Through the multi-scale parallel octave convolution operation, the initial subtitle text with different font sizes, styles, colors and other attributes is better recognized and extracted. Compared with the traditional OCR recognition model, the diversity of extracted features is enhanced, and the accuracy and robustness of the subtitle recognition model are improved.

[0122] At the same time, in order to make up for the shortcomings of OCR text recognition, such as in an environment with complex background scenes but relatively clean voices, the OCR recognition results may be inaccurate or incomplete due to factors such as video quality and subtitle background complexity. This application proposes to use a speech recognition model to perform speech recognition on the audio data extracted from the original video file, so that the speech text converted from the audio signal is not affected by the image quality, and the text information corresponding to subtitles with poor image quality or low contrast between subtitles and the background is accurately obtained. Based on this, errors or omissions in the subtitle text in the subtitle recognition results can be corrected, and the accuracy of subtitle extraction can be further improved, thereby reliably meeting the auxiliary needs of special audiences for video subtitles.

[0123] It can be seen that the video subtitle extraction method proposed in this embodiment not only solves the problem of inaccurate traditional OCR recognition, but also compares with the OCR subtitle extraction method based on deep learning, such as extracting features from video images through convolutional neural networks (CNN), combined with the processing capability of recurrent neural networks (RNN) for sequence data, to identify and correct subtitles in continuous video frames, and input the corrected video frames into an end-to-end text recognition model (OCR model) to obtain the corresponding subtitle text. Although this video subtitle extraction method can improve the accuracy of OCR recognition to a certain extent, due to the limitations of OCR recognition, especially when video subtitles are intertwined with complex backgrounds (such as dynamically changing images, textures or noise), it still cannot completely and accurately extract video subtitles. This application combines speech recognition technology, uses the speech text recognized from the audio in the video, and corrects the initial subtitle text, thereby improving the reliability and accuracy of the initial subtitle text correction.

[0124] Reference Figure 7 , is a flow chart of a video subtitle extraction method proposed in Embodiment 3 of the present application, which can describe another optional refinement of the video subtitle extraction method proposed above, such as Figure 7 As shown, the video subtitle extraction method proposed in this embodiment may include:

[0125] Step S701, obtaining audio data and image data in the original video file;

[0126] Step S702, performing speech recognition on the audio data to obtain a speech recognition result of the audio data;

[0127] Step S703, extracting multi-scale subtitle features from the image data, and obtaining a subtitle recognition result for the image data using the extracted multi-scale subtitle features;

[0128] Regarding the implementation method of step S701-step S703, you can refer to the description content of the corresponding part of the above embodiment, such as step S401-step S403, or step S501-step S507, or combine it with other detailed description content of the implementation method, etc. This application will not go into details here.

[0129] Step S704, matching the initial subtitle text in the subtitle recognition result with the speech text in the speech recognition result, and determining a first text and a second text that are inconsistent in matching; the first text belongs to the initial subtitle text, and the second text belongs to the speech text;

[0130] Step S705, determining whether the first confidence level for the first text in the subtitle recognition result is less than the second confidence level for the second text in the speech recognition result, if not, proceeding to step S706; if yes, executing step S707;

[0131] In an embodiment of the present application, a suitable text matching algorithm can be used to match the initial subtitle text in the subtitle recognition result with the speech text in the speech recognition result, that is, the initial subtitle text corresponding to the same timestamp is compared with the speech text. If the match is inconsistent, it means that there is an erroneous text in the two recognized texts corresponding to the timestamp. In this regard, by comparing their respective confidence levels, it can be determined which recognized text is more credible, that is, more likely to be a real subtitle. If the confidence level of the speech text is higher, the initial subtitle text will be corrected accordingly to achieve correction of errors and omissions in the initial subtitle text, obtain a complete and accurate target subtitle text, reduce the workload of subtitle correction, and improve the efficiency of video subtitle extraction.

[0132] Among them, the speech recognition model and the subtitle recognition model each perform text recognition on the input data, and when predicting the text information corresponding to the input data (that is, the corresponding speech text and the initial subtitle text), the confidence level of the corresponding predicted text information can be output, such as the prediction probability / score, etc. The larger the value, the more accurate and reliable the corresponding text information is. This application does not elaborate on the prediction implementation process.

[0133] It should be noted that during the above-mentioned speech recognition process, the noise audio in the video may be recognized, resulting in the converted text information not belonging to the video subtitles. In this way, during the text matching process, there may be a situation where the speech text does not match the initial subtitle text. This will not be processed, and the texts of other timestamps can continue to be matched. Of course, in the above-mentioned text matching process, the speech text and the initial subtitle text that are irrelevant to the content will not be matched. Only when the speech text and the initial subtitle text are similar, it may be the recognition text of the same real subtitle. The speech text with higher confidence is used to correct the initial subtitle text corresponding to the same real subtitle (that is, the text semantics are similar).

[0134] Step S706, maintaining the first text in the initial subtitle text unchanged;

[0135] Step S707, correcting the first text in the initial subtitle text into a second text;

[0136] Step S708, determining that the correction of all initial subtitle texts is completed, and determining the corrected initial subtitle texts as target subtitle texts;

[0137] Step S709: Time-align the target subtitle text with each video frame in the original video file to generate a subtitle file synchronized with the original video file.

[0138] Following the above analysis, after using different recognition methods, the corresponding recognition text is identified from the same original video file, that is, the speech text and the initial subtitle text corresponding to each video frame. After text matching (such as matching after time alignment of two texts), it is determined that there is a first text with inconsistent matching in the initial subtitle text, that is, the initial subtitle text or part of its vocabulary corresponding to a certain timestamp, etc., and the second text in the speech text is correspondingly determined. After confidence comparison, it is determined that the confidence of the second text is higher, such as the first confidence of the first text is 0.8, and the second confidence of the second text is 0.9. The second text can be used to replace the first text in the initial subtitle text, that is, the speech recognition result is used to correct the errors in the subtitle recognition result (improved OCR recognition result), supplement the missing vocabulary, and improve the accuracy of the target subtitle text.

[0139] In the above text matching process, if the confidence of the first text is higher, it means that the voice text is unreliable at this time, and there is no need to correct the initial subtitle text of the timestamp, and the first text can be maintained unchanged, that is, the voice text of each frame synchronized with the initial subtitle text is matched, and the third text in the initial subtitle text in the subtitle recognition result that matches the voice text in the speech recognition result is determined, such as text information with any timestamp, and the third text in the initial subtitle text can be maintained unchanged.

[0140] In a possible implementation, if the text matching process of step S707 adopts a one-to-one matching between the two text sequences of speech text and initial subtitle text, according to the method described above, it is determined that the text match of the current timestamp is consistent, the text with the current timestamp in the initial subtitle text is maintained unchanged, and the text matching of the next timestamp is continued; it is determined that the text match of the current timestamp is inconsistent, according to the method described in steps S705 and S707, the second text in the speech text of the timestamp is used to correct the first text in the initial subtitle text, and the text matching of the next timestamp is continued until the correction of all initial subtitle texts is completed, and the final corrected initial subtitle text is determined as the target subtitle text.

[0141] In another possible implementation, for the text matching process of step S707, in addition to the implementation method of correcting all the initial subtitle texts step by step as described in the previous paragraph, the speech texts and the initial subtitle texts of each time stamp after time synchronization can also be matched simultaneously. Figure 7The described method maintains all the texts that match consistently in the initial subtitle text unchanged, and uses the corresponding text in the speech text to correct all the texts that do not match consistently. This process can be performed simultaneously to obtain an accurate and reliable target subtitle text, and also improves processing efficiency. Of course, the present application can also match the initial subtitle text with the texts with different timestamps in the speech text one by one in sequence in the manner described in the previous paragraph, and after obtaining the respective text matching results, perform unified processing on the different text matching results, such as the processing methods described in step S706 and step S707, to finally obtain a complete and accurate target subtitle text in the original video file.

[0142] It should be understood that after text matching, it is determined that the initial subtitle text in the subtitle recognition result is consistent with the speech text in the speech recognition result, indicating that the initial subtitle text obtained based on the improved OCR recognition is more credible, and there is no need to correct it, and the initial subtitle text is directly determined as the target subtitle text. Moreover, the correction implementation method proposed in the embodiment of the present application determines the subtitle text that needs to be corrected in the subtitle recognition result based on the confidence comparison result, and then replaces it with the time-aligned text in the speech recognition result, avoids the problem of directly using the text in the speech recognition result with a higher similarity to replace the text in the subtitle recognition result, because the confidence of the text in the speech recognition result with a higher similarity is lower, and it is not the real subtitle text that directly replaces the text in the subtitle recognition result, which affects the accuracy and reliability of video subtitle extraction, and improves the accuracy and reliability of video subtitle extraction.

[0143] A video subtitle extraction method provided by an embodiment of the present application is introduced above, and a device for executing the above video subtitle extraction method will be introduced below.

[0144] Reference Figure 8 , a schematic diagram of the structure of a video subtitle extraction device provided in an embodiment of the present application. Figure 8 As shown, the video subtitle extraction device may include:

[0145] The data acquisition module 810 is used to acquire the audio data and image data in the original video file;

[0146] The speech recognition module 820 is used to perform speech recognition on the audio data to obtain a speech recognition result of the audio data;

[0147] A subtitle recognition module 830 is used to extract multi-scale subtitle features from the image data, and obtain a subtitle recognition result of the image data using the extracted multi-scale subtitle features;

[0148] The subtitle file generation module 840 is used to correct the subtitle recognition result based on the speech recognition result to generate a subtitle file matching the original video file.

[0149] In a possible implementation, the subtitle file generation module 840 may include:

[0150] A correction unit, configured to correct the initial subtitle text in the subtitle recognition result by using the speech text in the speech recognition result to obtain a target subtitle text;

[0151] The subtitle generating unit is used to time-align the target subtitle text with each video frame in the original video file to generate a subtitle file synchronized with the original video file.

[0152] In a possible implementation, the correction unit may include:

[0153] A text matching unit, configured to match the initial subtitle text in the subtitle recognition result with the speech text in the speech recognition result, and determine a first text and a second text that are inconsistent in matching; the first text belongs to the initial subtitle text, and the second text belongs to the speech text;

[0154] A correction subunit is used to correct the first text in the initial subtitle text into the second text to obtain a target subtitle text if a first confidence level for the first text in the subtitle recognition result is less than a second confidence level for the second text in the speech recognition result.

[0155] In a possible implementation, the subtitle file generating module 840 may further include:

[0156] A first determination unit, configured to determine that an initial subtitle text in the subtitle recognition result matches a speech text in the speech recognition result, and determine the initial subtitle text as a target subtitle text;

[0157] The second determining unit is used to determine a third text in the initial subtitle text in the subtitle recognition result that matches the speech text in the speech recognition result, and maintain the third text in the initial subtitle text unchanged.

[0158] In a possible implementation, the data acquisition module 810 may include:

[0159] A video image acquisition unit, used to acquire each frame of video image in the original video file;

[0160] A preprocessing unit, used for preprocessing each frame of video image;

[0161] A subtitle region image extraction unit is used to extract the subtitle region from each frame of the preprocessed video image to obtain image data of the subtitle region of the corresponding frame;

[0162] The audio data acquisition unit is used to acquire the audio data in the original video file.

[0163] In a possible implementation, the subtitle recognition module 830 may include:

[0164] A feature extraction unit, used for inputting the image data into a trained subtitle recognition model, and extracting subtitle features of different scales from the image data through a multi-scale parallel octave convolution operation;

[0165] A splicing unit, used for splicing the subtitle features of different scales to obtain target subtitle features;

[0166] The decoding unit is used to decode and obtain the subtitle recognition result of the image data according to the target subtitle feature.

[0167] Among them, the training modules for training the subtitle recognition model include:

[0168] An initial subtitle recognition network construction unit is used to update the convolution layer in the optical character recognition model to a feature extraction network containing multi-scale parallel convolution kernels to obtain an initial subtitle recognition network; the feature extraction network is an Inception deep learning convolution neural network constructed using multi-scale octave convolution;

[0169] A subtitle data set acquisition unit, used to acquire a subtitle data set; each subtitle image in the subtitle data set contains different subtitle attributes, and the subtitle attributes include one or more combinations of subtitle style, subtitle font size, and color;

[0170] An enhancement processing unit, configured to perform enhancement processing on the subtitle image, and add the enhanced subtitle image to the subtitle data set to obtain an enhanced subtitle data set;

[0171] The training unit is used to train the initial subtitle recognition network using each subtitle image in the enhanced subtitle data set to obtain a subtitle recognition model.

[0172] Preferably, the speech recognition module 820 can be specifically used to input the audio data into a speech recognition model based on a recursive neural network or a transform neural network to obtain a speech recognition result.

[0173] In a possible implementation, the feature extraction unit may include:

[0174] A feature map decomposition unit, used to decompose the feature map of the image data into high-frequency feature maps and low-frequency feature maps of different scales through multi-scale parallel octave convolution operations; the high-frequency feature map includes subtitle outline features in the image data, and the low-frequency feature map includes background features in the image data;

[0175] The subtitle feature obtaining unit is used to interactively integrate the high-frequency feature maps and the low-frequency feature maps of different scales to obtain subtitle features of different scales in the image data.

[0176] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any of the video subtitle extraction methods provided in the embodiments of the present application.

[0177] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any video subtitle extraction method provided in the embodiment of the present application.

[0178] The computer-readable storage medium may be any available medium that can be stored by a computer or a data storage device such as a training device or a data center that includes one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0179] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Moreover, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0180] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0181] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loading and executing computer program instructions on a computer, the process or function described in accordance with the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices. The computer instruction can be stored in a computer-readable storage medium, or transmitted from a computer-readable storage medium to another computer-readable storage medium, for example, the computer instruction can be transmitted from a website site, a computer, a training device or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device or data center.

[0182] In addition, the various embodiments in this specification are described in a progressive or parallel manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. For the devices and electronic devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0183] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A video subtitle extraction method, characterized in that: The video subtitle extraction method comprises: Get the audio data and image data in the original video file; Performing speech recognition on the audio data to obtain a speech recognition result of the audio data; Performing multi-scale subtitle feature extraction on the image data, and obtaining a subtitle recognition result of the image data using the extracted multi-scale subtitle features; Based on the speech recognition result, the subtitle recognition result is corrected to generate a subtitle file matching the original video file.

2. The video subtitle extraction method according to claim 1, characterized in that: The method of correcting the subtitle recognition result based on the speech recognition result to generate a subtitle file matching the original video file includes: Using the speech text in the speech recognition result, correcting the initial subtitle text in the subtitle recognition result to obtain a target subtitle text; The target subtitle text is time-aligned with each video frame in the original video file to generate a subtitle file synchronized with the original video file.

3. The video subtitle extraction method according to claim 2, characterized in that: The method of using the speech text in the speech recognition result to correct the initial subtitle text in the subtitle recognition result to obtain the target subtitle text includes: Matching the initial subtitle text in the subtitle recognition result with the speech text in the speech recognition result, and determining a first text and a second text that are inconsistent in matching; the first text belongs to the initial subtitle text, and the second text belongs to the speech text; If a first confidence level for the first text in the subtitle recognition result is less than a second confidence level for the second text in the speech recognition result, the first text in the initial subtitle text is corrected into the second text to obtain a target subtitle text.

4. The video subtitle extraction method according to claim 3, characterized in that: The method of correcting the subtitle recognition result based on the speech recognition result to generate a subtitle file matching the original video file also includes at least one of the following implementation methods: Determining that the initial subtitle text in the subtitle recognition result matches the speech text in the speech recognition result, and determining the initial subtitle text as the target subtitle text; Determine a third text in the initial subtitle text in the subtitle recognition result that matches the speech text in the speech recognition result, and maintain the third text in the initial subtitle text unchanged.

5. The video subtitle extraction method according to any one of claims 1 to 4, characterized in that: The step of extracting multi-scale subtitle features from the image data and obtaining a subtitle recognition result of the image data using the extracted multi-scale subtitle features includes: Inputting the image data into a trained subtitle recognition model, and extracting subtitle features of different scales from the image data through multi-scale parallel octave convolution operations; splicing the subtitle features of different scales to obtain target subtitle features; The subtitle recognition result of the image data is obtained by decoding according to the target subtitle feature.

6. The video subtitle extraction method according to claim 5, characterized in that: The training process of the subtitle recognition model includes: The convolutional layer in the optical character recognition model is updated to a feature extraction network containing multi-scale parallel convolution kernels to obtain an initial subtitle recognition network; the feature extraction network is an Inception deep learning convolutional neural network constructed using multi-scale octave convolution; Acquire a subtitle data set; each subtitle image in the subtitle data set contains different subtitle attributes, and the subtitle attributes include one or more combinations of subtitle style, subtitle font size, and color; Performing enhancement processing on the subtitle image, adding the enhanced subtitle image to the subtitle dataset, to obtain an enhanced subtitle dataset; The initial subtitle recognition network is trained using each subtitle image in the enhanced subtitle data set to obtain a subtitle recognition model.

7. The video subtitle extraction method according to claim 5, characterized in that: The extracting subtitle features of different scales from the image data through multi-scale parallel octave convolution operation includes: Decomposing the feature map of the image data into high-frequency feature maps and low-frequency feature maps of different scales through multi-scale parallel octave convolution operations; the high-frequency feature map includes the subtitle outline features in the image data, and the low-frequency feature map includes the background features in the image data; The high-frequency feature maps and the low-frequency feature maps of different scales are interactively integrated to obtain subtitle features of different scales in the image data.

8. The video subtitle extraction method according to any one of claims 1 to 4, characterized in that: The obtaining of image data in the original video file comprises: Get each frame of video image in the original video file; Preprocessing each frame of video image; The subtitle area is extracted from each frame of the preprocessed video image to obtain image data of the subtitle area of ​​the corresponding frame.

9. A video subtitle extraction device, characterized in that: The video subtitle extraction device comprises: A data acquisition module, used for acquiring audio data and image data in the original video file; A speech recognition module, used to perform speech recognition on the audio data to obtain a speech recognition result of the audio data; A subtitle recognition module, used to extract multi-scale subtitle features from the image data, and obtain a subtitle recognition result of the image data using the extracted multi-scale subtitle features; The subtitle file generation module is used to correct the subtitle recognition result based on the speech recognition result to generate a subtitle file matching the original video file.

10. An electronic device, characterized in that: The electronic device comprises: at least one communication element, at least one memory and at least one processor, wherein: The communication element is used to receive the original video file; The processor is used to load and execute multiple computer instructions stored in the memory to implement the following steps: Acquire audio data and image data in the original video file; Performing speech recognition on the audio data to obtain a speech recognition result of the audio data; Performing multi-scale subtitle feature extraction on the image data, and obtaining a subtitle recognition result of the image data using the extracted multi-scale subtitle features; Based on the speech recognition result, the subtitle recognition result is corrected to generate a subtitle file matching the original video file.

Images