Subtitle positioning method, electronic device and medium

Through audio fingerprint matching and text detection network models, efficiently identify and copy subtitle information, the problem of time-consuming and low accuracy of subtitle positioning in the prior art is solved, and fast and accurate subtitle positioning and video cropping are achieved.

CN114071184BActive Publication Date: 2024-12-03TENCENT MUSIC ENTERTAINMENT TECH (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111333945.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-11-11
Publication Date
2024-12-03
Estimated Expiration
2041-11-11

AI Technical Summary

Technical Problem

The existing video subtitle positioning methods are time-consuming and prone to missed or missed detection while ensuring accuracy. Especially when processing horizontal music short films to vertical versions, an efficient subtitle positioning method is required. To avoid truncating subtitles.

Method used

By matching the song audio signals in the video file with the preset audio fingerprint library, determining the time position of the target song and lyrics, filtering out keyframe pictures for subtitle positioning, using text detection network model to identify the subtitle content, and using the appearance frequency and position offsets, Rate determines the subtitle category and copy the subtitle information to other frame pictures.

Benefits of technology

It significantly reduces the time cost of subtitle positioning, improves the accuracy and efficiency of subtitle positioning, avoids missed and mis-checked subtitles, and ensures the integrity of video cropping.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114071184B_ABST
    Figure CN114071184B_ABST
Patent Text Reader

Abstract

The present application discloses a subtitle positioning method, electronic device and medium, which matches the audio fingerprint of the song audio signal in the video file with the preset audio fingerprint library to determine the target song matched by the audio fingerprint and the time position of the song audio signal in the target song; wherein the audio fingerprint library records the correspondence between the song and the song fingerprint information. According to the distribution time of the lyrics of the target song and the time position of the song audio signal in the target song, the video clip corresponding to each line of lyrics in the video file is determined. A preset number of frame images are selected from each video clip as the frame images to be analyzed, and only the frame images to be analyzed need to be subtitled. Relying on the subtitle information in the frame images to be analyzed, the subtitle information of the remaining frame images in each video clip can be quickly determined, which greatly reduces the time spent on subtitle positioning while ensuring the accuracy of subtitle positioning.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a subtitle positioning method, an electronic device and a computer-readable storage medium. Background Art

[0002] With the popularization of mobile Internet, users spend much more time browsing information on mobile phones than on other media. Mobile phones are more suitable for vertical video browsing, but most existing music videos (MVs) are horizontal, which requires us to crop vertical videos. The cropping process should avoid the problem of truncating the lyrics and subtitles in the video, so the original lyrics and subtitles need to be removed, and the positioning of the lyrics and subtitles is a prerequisite for subtitle removal.

[0003] At present, most of the solutions for locating lyrics and subtitles in videos convert videos into image sequences and then detect lyrics and subtitles frame by frame. Lyrics and subtitles detection technologies mainly include traditional edge detection-based methods and deep learning-based methods.

[0004] Traditional edge detection methods are highly dependent on the design of empirical parameters, and the detection accuracy is not high on large-scale data. Based on deep learning methods, if frame-by-frame detection is used, it will take a long time. For example, a 4-minute video with 25 frames per second includes 6,000 frames of pictures, and even the fastest deep learning model takes 10 minutes. In order to reduce the computing time, after splitting the video into time-series pictures, these pictures can be clustered and then sampled for subtitle detection. However, if there are no subtitles in the extracted pictures, the subtitles of the clip will be missed, and the situation where the subtitle color is close to the background color will also result in missed detection. In addition, deep learning models will produce more false detections, such as misdetecting background text, advertising text, and text on clothes as lyrics subtitles.

[0005] It can be seen that how to reduce the time spent on subtitle positioning while ensuring the accuracy of subtitle positioning is a problem that technical personnel in this field need to solve. Summary of the invention

[0006] In view of this, the purpose of this application is to provide a subtitle positioning method, device, electronic device and computer-readable storage medium, which can reduce the time spent on subtitle positioning while ensuring the accuracy of subtitle positioning. The specific scheme is as follows:

[0007] In a first aspect, the present application discloses a subtitle positioning method, comprising:

[0008] Matching the audio fingerprint of the song audio signal in the video file with a preset audio fingerprint library to determine the target song matched by the audio fingerprint and the time position of the song audio signal in the target song; wherein the audio fingerprint library records the correspondence between the song and the song fingerprint information;

[0009] Determine the video clip corresponding to each line of lyrics in the video file according to the lyrics distribution time of the target song and the time position of the song audio signal in the target song;

[0010] Filtering a preset number of frame images from each of the video clips as frame images to be analyzed;

[0011] The subtitle information in the frame picture to be analyzed is determined, and the subtitle information of the remaining frame pictures in each of the video segments is determined based on the subtitle information in the frame picture to be analyzed.

[0012] Optionally, matching the audio fingerprint of the song audio signal in the video file with a preset audio fingerprint library to determine a target song matched by the audio fingerprint and a time position of the song audio signal in the target song includes:

[0013] Extracting a song audio signal from a video file and converting the song audio signal into a speech spectrogram;

[0014] Based on each peak point in the speech spectrogram, determining the audio fingerprint of the song audio signal;

[0015] The audio fingerprint is matched with the fingerprint information of each song in the audio fingerprint library to determine the target song matched by the audio fingerprint and the time position of the song audio signal in the target song.

[0016] Optionally, determining the video clip corresponding to each line of lyrics in the video file according to the lyrics distribution time of the target song and the time position of the song audio signal in the target song includes:

[0017] According to the time position of the song audio signal in the target song, extracting a song segment matching the song audio signal from the target song;

[0018] Determine the time range corresponding to each line of lyrics in the song segment according to the lyrics distribution time of the song segment;

[0019] Based on the time range corresponding to each line of lyrics in the song segment and the frame rate of the video file, the video segment corresponding to each line of lyrics in the video file is determined.

[0020] Optionally, the subtitle information includes position information and text category; and determining the subtitle information in the frame picture to be analyzed includes:

[0021] Using a text detection network model, identifying position information corresponding to the text content in the frame image to be analyzed;

[0022] According to the position information corresponding to the text content in the frame image to be analyzed, the text category corresponding to each of the text contents is determined.

[0023] Optionally, determining the text category corresponding to each text content according to the position information corresponding to the text content in the frame image to be analyzed includes:

[0024] If the occurrence frequency of the target text in the same area of ​​the frame image to be analyzed is less than a preset frequency threshold, the text content of the target text is determined to be background text; and / or

[0025] If the occurrence frequency of the target text in the same area of ​​the frame image to be analyzed is not less than a preset frequency threshold, and the text endpoint position offset rate of the target text is less than a preset change rate threshold, the text content of the target text is determined to be a logo text; and / or

[0026] If the occurrence frequency of the target text in the same area of ​​the frame image to be analyzed is not less than the preset frequency threshold, and the text endpoint position offset rate of the target text is not less than the preset change rate threshold, the text content of the target text is determined to be lyrics-type text.

[0027] Optionally, the preset frequency threshold includes a first preset frequency threshold, a second preset frequency threshold and a third preset frequency threshold; wherein the first preset frequency threshold is greater than the second preset frequency threshold; the second preset frequency threshold is greater than the third preset frequency threshold;

[0028] Determining the text category corresponding to each text content according to the position information corresponding to the text content in the frame image to be analyzed includes:

[0029] If the frequency of occurrence of text at the first area position of the frame image to be analyzed is greater than a first preset frequency threshold, and the position offset rate of the text endpoints at the first area position is less than a preset change rate threshold, the text content at the first area position is determined to be a logo-type text; and / or

[0030] If the frequency of occurrence of the text at the second area position of the frame image to be analyzed is greater than a second preset frequency threshold, and the position offset rate of the text endpoints at the second area position is not less than the preset change rate threshold, the text content at the second area position is determined to be lyrics-like text; and / or

[0031] If the occurrence frequency of the text at the third area position of the frame image to be analyzed is less than a third preset frequency threshold, the text content at the third area position is determined to be background text.

[0032] Optionally, determining the text category corresponding to each text content according to the position information corresponding to the text content in the frame image to be analyzed includes:

[0033] According to the position information corresponding to the text content in the frame image to be analyzed, the frame image to be analyzed is converted into a binary image; in the binary image, two different numbers are used to represent the pixel points in the text area and the pixel points in the non-text area respectively;

[0034] Add the numbers corresponding to the pixels at the same position in the binary image to obtain the superposition value corresponding to each pixel;

[0035] According to the category value range to which the superposition value corresponding to each pixel point belongs, the text category corresponding to each text content is determined; wherein each text category has its corresponding category value range; the text categories include logo text, lyrics text and background text.

[0036] Optionally, the determining the subtitle information of the remaining frame pictures in each of the video segments based on the subtitle information in the frame picture to be analyzed includes:

[0037] Copying the position information corresponding to the logo text to the remaining frame images in each of the video clips;

[0038] The position information corresponding to each line of lyrics in the lyrics-like text is copied to the remaining frame images in the video clip with the same lyrics.

[0039] In a second aspect, the present application discloses a subtitle positioning device, including a matching unit, a video segment determination unit, a screening unit, and a subtitle determination unit;

[0040] The matching unit is used to match the audio fingerprint of the song audio signal in the video file with a preset audio fingerprint library to determine the target song matched by the audio fingerprint and the time position of the song audio signal in the target song; wherein the audio fingerprint library records the correspondence between the song and the song fingerprint information;

[0041] The video segment determination unit is used to determine the video segment corresponding to each line of lyrics in the video file according to the lyrics distribution time of the target song and the time position of the song audio signal in the target song;

[0042] The screening unit is used to screen out a preset number of frame images from each of the video clips as frame images to be analyzed;

[0043] The subtitle determination unit is used to determine the subtitle information in the frame picture to be analyzed, and determine the subtitle information of the remaining frame pictures in each of the video segments based on the subtitle information in the frame picture to be analyzed.

[0044] Optionally, the matching unit includes an extraction subunit, a conversion subunit and a determination subunit;

[0045] The extraction subunit is used to extract the song audio signal from the video file;

[0046] The conversion subunit is used to convert the song audio signal into a speech spectrogram;

[0047] The determination subunit is used to determine the audio fingerprint of the song audio signal based on each peak point in the speech spectrogram; match the audio fingerprint with the fingerprint information of each song in the audio fingerprint library to determine the target song matched by the audio fingerprint and the time position of the song audio signal in the target song.

[0048] Optionally, the video segment determination unit includes a capture subunit and a determination subunit;

[0049] The interception subunit is used to intercept a song segment matching the song audio signal from the target song according to the time position of the song audio signal in the target song;

[0050] The determination subunit is used to determine the time range corresponding to each line of lyrics in the song clip according to the lyrics distribution time of the song clip; based on the time range corresponding to each line of lyrics in the song clip and the frame rate of the video file, determine the video clip corresponding to each line of lyrics in the video file.

[0051] Optionally, the subtitle information includes position information and text category; the subtitle determination unit includes an identification subunit and a category determination subunit;

[0052] The identification subunit is used to identify the position information corresponding to the text content in the frame image to be analyzed by using the text detection network model;

[0053] The category determination subunit is used to determine the text category corresponding to each text content according to the position information corresponding to the text content in the frame image to be analyzed.

[0054] Optionally, the category determination subunit is used to determine the text content of the target text as background text if the frequency of occurrence of the target text in the same area position in the frame image to be analyzed is less than a preset frequency threshold; and / or if the frequency of occurrence of the target text in the same area position in the frame image to be analyzed is not less than a preset frequency threshold, and the text endpoint position offset rate of the target text is less than a preset change rate threshold, then determine the text content of the target text as logo text; and / or if the frequency of occurrence of the target text in the same area position in the frame image to be analyzed is not less than a preset frequency threshold, and the text endpoint position offset rate of the target text is not less than the preset change rate threshold, then determine the text content of the target text as lyrics text.

[0055] Optionally, the preset frequency threshold includes a first preset frequency threshold, a second preset frequency threshold and a third preset frequency threshold; wherein the first preset frequency threshold is greater than the second preset frequency threshold; the second preset frequency threshold is greater than the third preset frequency threshold;

[0056] The category determination subunit is used to determine the text content at the first area position of the frame image to be analyzed as logo-type text if the frequency of text appearance at the first area position of the frame image to be analyzed is greater than a first preset frequency threshold, and the offset rate of the text endpoint position at the first area position is less than a preset change rate threshold; and / or if the frequency of text appearance at the second area position of the frame image to be analyzed is greater than a second preset frequency threshold, and the offset rate of the text endpoint position at the second area position is not less than the preset change rate threshold, then determine the text content at the second area position as lyric-type text; and / or if the frequency of text appearance at the third area position of the frame image to be analyzed is less than a third preset frequency threshold, then determine the text content at the third area position as background-type text.

[0057] Optionally, the category determination subunit is used to convert the frame image to be analyzed into a binary image based on the position information corresponding to the text content in the frame image to be analyzed; two different numbers are used in the binary image to represent the pixel points on the text area and the pixel points on the non-text area respectively; the numbers corresponding to the pixel points at the same position in the binary image are added to obtain the superposition value corresponding to each pixel point; based on the category value range to which the superposition value corresponding to each pixel point belongs, the text category corresponding to each of the text contents is determined; wherein each text category has its corresponding category value range; the text categories include logo text, lyrics text and background text.

[0058] Optionally, the subtitle determination unit is used to copy the position information corresponding to the logo-type text on the remaining frame images in each of the video clips; and copy the position information corresponding to each line of the lyrics in the lyrics-type text on the remaining frame images in the video clips with the same lyrics.

[0059] In a third aspect, the present application discloses an electronic device, comprising:

[0060] Memory, used to store computer programs;

[0061] The processor is used to execute the computer program to implement the aforementioned disclosed subtitle positioning method.

[0062] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program, wherein the computer program implements the aforementioned disclosed subtitle positioning method when executed by a processor.

[0063] In the present application, the audio fingerprint of the song audio signal in the video file is matched with the preset audio fingerprint library to determine the target song matched by the audio fingerprint and the time position of the song audio signal in the target song; wherein the audio fingerprint library records the corresponding relationship between the song and the song fingerprint information. The audio fingerprint can be information generated based on the characteristics of the song audio signal for uniquely identifying the song audio signal. The song fingerprint information is generated in the same way as the audio fingerprint. By matching the audio fingerprint with the preset audio fingerprint library, the target song to which the song audio signal belongs and the time position of the song audio signal in the target song can be determined. Based on the duration of each line of lyrics in the song and the number of frame images played per unit time in the video file, it can be known that the same line of lyrics often appears in multiple frame images at the same time. If each frame image in the video file is analyzed, it is a repeated analysis for the frame images containing the same lyrics. In order to reduce the time spent on subtitle positioning, the video clip corresponding to each line of lyrics in the video file can be determined according to the lyrics distribution time of the target song and the time position of the song audio signal in the target song. A preset number of frame images are selected from each video clip as the frame images to be analyzed. After screening out the frame images to be analyzed, it is only necessary to locate the subtitles of the frame images to be analyzed and determine the subtitle information in the frame images to be analyzed. Relying on the subtitle information in the frame images to be analyzed, the subtitle information of the remaining frame images in each video clip can be quickly determined. In this technical solution, by matching the audio fingerprint, the target song to which the song audio signal belongs and the time position of the song audio signal in the target song can be determined. According to the distribution time of the lyrics of the target song and the time position of the song audio signal in the target song, a preset number of frame images can be screened out from the video clip corresponding to each line of lyrics as the frame images to be analyzed. Relying on the subtitle positioning results of the frame images to be analyzed, the subtitle positions of all frame images in the video file can be quickly determined, which greatly reduces the time spent on subtitle positioning while ensuring the accuracy of subtitle positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0065] Figure 1 A system hardware composition framework diagram applicable to a subtitle positioning method provided in this application;

[0066] Figure 2 A flow chart of a subtitle positioning method provided for this application;

[0067] Figure 3 A flowchart of a method for selecting a frame image to be analyzed provided in this application;

[0068] Figure 4 A flowchart of a method for marking subtitles for each frame of a video file provided by the present application;

[0069] Figure 5 A flowchart of a method for determining a text category provided in an embodiment of the present application;

[0070] Figure 6a A schematic diagram of a frame image provided for this application;

[0071] Figure 6b The application provided for Figure 6a Schematic diagram of subtitle positioning based on frame images;

[0072] Figure 6c The application provided Figure 6a Another frame image with the same lyrics;

[0073] Figure 7 A schematic diagram of the structure of a subtitle positioning device provided in this application;

[0074] Figure 8 A structural diagram of an electronic device provided for this application. DETAILED DESCRIPTION

[0075] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0076] The positioning of lyrics subtitles is a prerequisite for subtitle removal. When implementing subtitle positioning, the traditional edge detection method has low detection accuracy on large-scale data. The deep learning method, if using frame-by-frame detection, will take a long time. If all frame images in the video file are clustered and then sampled for subtitle detection, it is easy to miss or misdetect, resulting in reduced accuracy of subtitle positioning.

[0077] Therefore, the embodiments of the present application provide a subtitle positioning method, device, electronic device and computer-readable storage medium. The subtitle positioning solution provided in the embodiments of the present application is mainly aimed at subtitles with regular content and time distribution of subtitles, mainly taking songs as an example.

[0078] like Figure 1The following is a system hardware composition framework diagram applicable to a subtitle positioning method provided in an embodiment of the present application. In the conventional manner, video files shot are often played in a horizontal screen mode. When the user terminal has a requirement to play the video file in a vertical screen mode, a vertical screen play instruction of the video file can be sent to the server. The user terminal can be a portable electronic device such as a mobile phone.

[0079] In practical applications, when converting a video file played in horizontal mode to a video file played in vertical mode, in order to ensure the integrity of the subtitles of the video file, it is necessary to locate the location of the subtitles in the video file. After determining the subtitle information of all frame images in the video file, the video file can be vertically cropped based on the subtitle information of the frame images, so that the cropped video file is suitable for vertical screen playback. In practical applications, one server can be used to perform the two processes of subtitle positioning and vertical screen cropping of video files. Two servers can also be used, one server is used to perform subtitle positioning of video files, and the other server is used to perform vertical screen cropping of video files. Figure 1 In the example, two servers are used to perform subtitle positioning and vertical screen cropping of video files. Server 1 can be used to perform subtitle positioning of video files, and server 2 can be used to perform vertical screen cropping of video files. Server 1 and server 2 can provide subtitle positioning and vertical screen cropping services for multiple user terminals. Figure 1 The example in this section is the interaction between a server and a user terminal.

[0080] When server 1 locates subtitles for a video file, the video file may be transmitted from a user terminal to server 1. In addition, the video file may also be obtained by server 1 from a third-party storage terminal. The source of the video file is not limited here.

[0081] In order to realize the subtitle positioning of each frame of the picture in the video file, the server 1 can rely on the target song that the song audio signal matches in the video file and the time position of the song audio signal in the target song to determine the video clip corresponding to each line of lyrics in the video file. In order to quickly determine the target song that the song audio signal matches, an audio fingerprint library can be set on the server 1, in which a large number of corresponding relationships between songs and song fingerprint information can be recorded. By matching the audio fingerprint of the song audio signal in the video file with the preset audio fingerprint library, the target song that the audio fingerprint matches and the time position of the audio signal in the target song can be determined.

[0082] Each line of lyrics in a song has its corresponding playback duration, so each line of lyrics is often contained in multiple frame images. The video clip corresponding to each line of lyrics in the video file can be determined based on the lyrics distribution time of the target song and the time position of the song audio signal in the target song. In practical applications, it is only necessary to select a preset number of frame images from each video clip as the frame images to be analyzed, perform subtitle positioning on the frame images to be analyzed, and determine the subtitle information in the frame images to be analyzed. Based on the subtitle information in the frame images to be analyzed, the subtitle information of the remaining frame images in each video clip can be quickly determined, which greatly reduces the number of frame images that need to perform subtitle positioning analysis. While ensuring the accuracy of subtitle positioning, the time spent on subtitle positioning is greatly reduced.

[0083] Figure 2 A flow chart of a subtitle positioning method provided in an embodiment of the present application. Figure 2 As shown, the subtitle positioning method may include the following steps

[0084] S201: Matching the audio fingerprint of the song audio signal in the video file with a preset audio fingerprint library to determine the target song for audio fingerprint matching and the time position of the song audio signal in the target song.

[0085] Among them, the audio fingerprint library records the correspondence between songs and song fingerprint information.

[0086] Audio fingerprints can be information generated based on the characteristics of a song audio signal to uniquely identify a song audio signal. Song fingerprint information is generated in the same way as audio fingerprints. By matching the audio fingerprint with a preset audio fingerprint library, the target song to which the song audio signal belongs and the time position of the audio signal in the target song can be determined.

[0087] In an embodiment of the present application, a song audio signal can be extracted from a video file and converted into a speech spectrogram; based on each peak point in the speech spectrogram, an audio fingerprint of the song audio signal is determined; the audio fingerprint is matched with each song fingerprint information in an audio fingerprint library to determine a target song for audio fingerprint matching and a time position of the song audio signal in the target song.

[0088] In a specific implementation, the method of determining the audio fingerprint based on the peak point may include selecting a set of adjacent peak points from each peak point, and encoding the set of adjacent peak points into the audio fingerprint.

[0089] Each adjacent peak point set encoding can obtain a sub-fingerprint, and the sub-fingerprints corresponding to each adjacent peak point set can be combined as an audio fingerprint.

[0090] Among them, selecting a set of adjacent peak points from each peak point can be to use any peak point in the speech spectrum graph as the center of the circle, preset a distance threshold as the radius to determine the coverage range, and combine all peak points within the coverage range whose time points are greater than the time points of the center of the circle into a set of adjacent peak points. The set of adjacent peak points only includes peak points within a certain range and whose time points are greater than the time points of the center of the circle, that is, peak points behind the center of the circle, so that repeated sub-fingerprints can be avoided. Of course, the set of adjacent peak points can also be selected according to other strategies, which are not limited here.

[0091] In practical applications, a hash algorithm can be used to encode a set of neighboring peak points into an audio fingerprint to reduce the possibility of fingerprint collision.

[0092] The peak points in the speech spectrogram represent the most representative frequency value of a song at each moment. Each peak point corresponds to a mark (f, t) composed of frequency and time, which can be understood as a coordinate. The marks corresponding to each peak point and its n neighboring peak points can form a set of neighboring peak points, which can then be encoded into an audio fingerprint in the form of hash coding.

[0093] For example, if the peak point as the center of the circle is represented as (f0, t0), and its n neighboring peak point sets are represented as (f1, t1), (f2, t2), ..., (fn, tn), then (f0, t0) is combined with each of its neighboring peak points to obtain each pair of combination information, such as (f0, f1, t1-t0), (f0, f2, t2-t0), ..., (f0, fn, tn-t0), and then each pair of combination information can be encoded into a sub-fingerprint in the form of hash coding. All sub-fingerprints are combined as the audio fingerprint of the song audio signal.

[0094] S202: Determine the video clip corresponding to each line of lyrics in the video file according to the distribution time of the lyrics of the target song and the time position of the song audio signal in the target song.

[0095] Lyric distribution time refers to the time range corresponding to each word in each line of lyrics in the song. In actual applications, QRC lyrics (lyrics files of QQ music player) can accurately control each word. The format of each line of QRC lyrics is: [start time ms, duration ms] lyrics (start time ms, duration ms).

[0096] The start time in square brackets indicates the start time of a line of lyrics in the entire song, the duration in square brackets indicates the time taken for a line of lyrics to be played, the start time in parentheses indicates the start time of each word, and the duration indicates the time taken for this word to be played.

[0097] For example, the lyrics are "I am used to being buried deep in the fog", and the corresponding lyrics format is: [6156, 1996] I am (6156, 147) used to (6303, 154) being (6457, 224) being (6681, 187) being (6868, 324) being (7192, 395) being (7587, 565). The 6156 in the square brackets indicates that the start time of this lyric in the whole song is 6156ms, and 1996 indicates that the time taken for this lyric to be played is 1996ms. Taking any one of the characters as an example, such as "deep", the corresponding 6681 indicates that the start time of the character "deep" in the whole song is 6681ms, and 187 indicates that the time taken for the character "deep" to be played in the lyrics "I am used to being buried deep in the fog" is 187ms.

[0098] In the embodiment of the present application, the lyrics distribution time of the target song can be determined based on the QRC lyrics.

[0099] According to the time position of the song audio signal in the target song, the lyrics content in the target song that matches the song audio signal can be determined.

[0100] For example, the duration of the target song is 4 minutes, the time position of the song audio signal in the target song is the 2nd minute, and the duration of the song audio signal is 1 minute. Then the lyrics content corresponding to the 2nd minute to the 3rd minute of the target song is the lyrics content that matches the song audio signal.

[0101] Based on the duration of each line of lyrics in the song and the number of frames played per unit time in the video file, it can be known that the same line of lyrics often appears in multiple frames at the same time. In the embodiment of the present application, multiple consecutive frames with the same line of lyrics can be referred to as a video clip. According to the distribution time of the lyrics of the target song and the time position of the song audio signal in the target song, the video clip corresponding to each line of lyrics in the video file can be determined.

[0102] S203: Filter out a preset number of frame images from each video segment as frame images to be analyzed.

[0103] If each frame of the video file is analyzed, the frame containing the same lyrics is analyzed repeatedly. In order to reduce the time spent on subtitle positioning, in the embodiment of the present application, a preset number of frames can be selected from each video clip as the frames to be analyzed.

[0104] The value of the preset number can be set according to actual needs, for example, the preset number can be set to 2.

[0105] For example, suppose a video clip contains 48 frames, and these 48 frames contain the same lyrics. In order to reduce the time spent on subtitle positioning of the frames, 2 frames can be selected from the 48 frames for subtitle positioning analysis. In practical applications, 2 frames can be selected from the 48 frames as the frames to be analyzed, or the first and last frames can be selected in chronological order. After determining the subtitle information of these 2 frames, the subtitle information corresponding to the lyrics and logo text can be directly copied to the remaining 46 frames.

[0106] S204: Determine the subtitle information in the frame picture to be analyzed, and determine the subtitle information of the remaining frame pictures in each video segment based on the subtitle information in the frame picture to be analyzed.

[0107] In the embodiment of the present application, the existing subtitle positioning technology can be used to position the subtitles of the frame picture to be analyzed. After determining the subtitle information in the frame picture to be analyzed, the subtitle information of the remaining frame pictures in each video segment is determined based on the subtitle information in the frame picture to be analyzed.

[0108] The subtitle information may include text category and position information, wherein the position information may be expressed in the form of coordinates.

[0109] The text category refers to the category to which the text content of the subtitles belongs, which can include lyrics text, logo text, and background text.

[0110] Background text will hardly affect the vertical screen cropping of video files, so you can directly ignore the background text of each frame in the video file.

[0111] Lyrics and logo texts will affect the vertical screen cropping of the video file, so when determining the subtitle information of the remaining frame images, the lyrics and logo texts appearing in the frame images are mainly marked.

[0112] Logo text is often a text with fixed content and a relatively fixed position in each frame of a video file, for example, the logo of the software that plays the current video file, or the logo of the program that plays the current video file, etc. After determining the position information corresponding to the logo text in the frame to be analyzed, the position information can be applied to the remaining frames in each video clip.

[0113] For lyrics, each frame in a video clip contains the same lyrics, and the lyrics in different video clips are different, but the position of the lyrics in each video clip is relatively fixed, such as the lyrics appearing at the bottom of the frame. After determining the position information corresponding to the lyrics in the frame to be analyzed, the position information corresponding to each line of lyrics in the frame to be analyzed can be applied to the remaining frames of the video clip with the same lyrics.

[0114] In the present application, the audio fingerprint of the song audio signal in the video file is matched with the preset audio fingerprint library to determine the target song matched by the audio fingerprint and the time position of the song audio signal in the target song; wherein the audio fingerprint library records the corresponding relationship between the song and the song fingerprint information. The audio fingerprint can be information generated based on the characteristics of the song audio signal for uniquely identifying the song audio signal. The song fingerprint information is generated in the same way as the audio fingerprint. By matching the audio fingerprint with the preset audio fingerprint library, the target song to which the song audio signal belongs and the time position of the song audio signal in the target song can be determined. Based on the duration of each line of lyrics in the song and the number of frame images played per unit time in the video file, it can be known that the same line of lyrics often appears in multiple frame images at the same time. If each frame image in the video file is analyzed, it is a repeated analysis for the frame images containing the same lyrics. In order to reduce the time spent on subtitle positioning, the video clip corresponding to each line of lyrics in the video file can be determined according to the lyrics distribution time of the target song and the time position of the song audio signal in the target song. A preset number of frame images are selected from each video clip as the frame images to be analyzed. After screening out the frame images to be analyzed, it is only necessary to locate the subtitles of the frame images to be analyzed and determine the subtitle information in the frame images to be analyzed. Relying on the subtitle information in the frame images to be analyzed, the subtitle information of the remaining frame images in each video clip can be quickly determined. In this technical solution, by matching the audio fingerprint, the target song to which the song audio signal belongs and the time position of the song audio signal in the target song can be determined. According to the distribution time of the lyrics of the target song and the time position of the song audio signal in the target song, a preset number of frame images can be screened out from the video clip corresponding to each line of lyrics as the frame images to be analyzed. Relying on the subtitle positioning results of the frame images to be analyzed, the subtitle positions of all frame images in the video file can be quickly determined, which greatly reduces the time spent on subtitle positioning while ensuring the accuracy of subtitle positioning.

[0115] Figure 3 A flowchart of a method for selecting a frame image to be analyzed provided in an embodiment of the present application. Figure 3 As shown, the method for selecting a frame image to be analyzed may include the following steps

[0116] S301: based on the time position of the song audio signal in the target song, extracting a song segment that matches the song audio signal from the target song.

[0117] In actual applications, the song contained in the video file may not be a complete song, but a song segment. When performing the screening of the frame image to be analyzed, it is necessary to determine the lyrics content that matches the song audio signal. Therefore, in the embodiment of the present application, it is possible to determine which part of the song the audio signal corresponds to based on the time position of the audio signal in the target song, so as to extract the song segment that matches the song audio signal in the target song.

[0118] S302: Determine the time range corresponding to each line of lyrics in the song segment according to the distribution time of the lyrics in the song segment.

[0119] In an embodiment of the present application, the lyrics distribution time of a song segment can be determined based on the QRC lyrics. The lyrics distribution time includes the time range corresponding to each word in each sentence of the song, such as the start time and duration of each word.

[0120] S303: Based on the time range corresponding to each line of lyrics in the song segment and the frame rate of the video file, determine the video segment corresponding to each line of lyrics in the video file.

[0121] The frame rate of a video file may be the number of frames of the video file played in a unit of time. Generally, the frame rate of a video file may be 25fps.

[0122] For example, if the duration of a line of lyrics is 1920ms and the video file plays 25 frames per second, it can be determined that the line of lyrics will appear in 1920 / 1000*25=48 consecutive frames of pictures, and these 48 consecutive frames of pictures are a video clip. Similarly, the video clip corresponding to each line of lyrics in the video file can be determined.

[0123] In the embodiment of the present application, the song segment corresponding to the song audio signal is determined, and the video segment corresponding to each lyric in the video file is determined according to the time range corresponding to each lyric in the song segment and the frame rate of the video file. When performing the screening of the frame images to be analyzed, at least one frame image to be analyzed is screened out from each video segment, thereby ensuring that all the screened frame images to be analyzed can contain all the lyrics corresponding to the song segment, while reducing the number of frame images for subtitle positioning analysis, making the screened frame images to be analyzed more representative, thereby ensuring the accuracy of performing subtitle positioning analysis on the video file.

[0124] Figure 4A flowchart of a method for marking subtitles for each frame of a video file provided in an embodiment of the present application. Figure 4 As shown, the method for marking subtitles for each frame of a video file may include the following steps:

[0125] S401: Using a text detection network model, identifying position information corresponding to text content in a frame image to be analyzed.

[0126] The text detection network model can adopt the existing relatively mature VGG16 network architecture to realize the subtitle positioning of the frame image to be analyzed.

[0127] The input of the text detection network model is a color image, and the network output is two probability maps of the same size as the input image, one of which represents the probability that each pixel belongs to a text, and the other represents the probability that each pixel belongs to between two texts. Combining the two output probability maps, a black-and-white image of the same size as the input image can be obtained, where the black area represents text content and the white area represents no text content.

[0128] S402: Determine the text category corresponding to each text content according to the position information corresponding to the text content in the frame image to be analyzed.

[0129] The text categories may include lyrics text, background text and logo text.

[0130] For background text, the probability of its appearance in all the frame images to be analyzed is low. For lyrics text and logo text, the probability of its appearance in all the frame images to be analyzed is high. The text content of logo text is relatively fixed, so the regional position corresponding to the logo text in the frame image to be analyzed is relatively fixed. As for lyrics text, since different lyrics are of different lengths, the regional position corresponding to the lyrics text in different frame images to be analyzed will change. Therefore, in the embodiment of the present application, according to the position information corresponding to the text content in the frame image to be analyzed, the probability of the text content appearing in all the frame images to be analyzed at the same regional position can be determined. And according to the change of the position information corresponding to the text content, the change of the regional position of the text content in different frame images to be analyzed can be determined, thereby determining the text category corresponding to each text content.

[0131] Figure 5 A flowchart of a method for determining a text category provided in an embodiment of the present application, the method comprising:

[0132] S501: Determine whether the occurrence frequency of the target text in the same area of ​​the frame image to be analyzed is less than a preset frequency threshold.

[0133] The value of the preset frequency threshold can be set according to actual needs. The purpose of setting the preset frequency threshold is to distinguish the probability of occurrence of logo text, background text and lyrics text.

[0134] In practical applications, the preset frequency threshold may be set to a numerical value.

[0135] Considering that background text rarely appears in all frames to be analyzed, if the frequency of occurrence of the target text in the same area of ​​the frame to be analyzed is less than the preset frequency threshold, it means that the text content of the target text should be background text, and S502 can be executed at this time.

[0136] If the frequency of the target text in the same area of ​​the frame image to be analyzed is not less than the preset frequency threshold, it means that the text content of the target text should be a logo text or a lyric text. At this time, the target text can be further judged, that is, S503 is executed.

[0137] S502: Determine the text content of the target text as background text.

[0138] If the occurrence frequency of the target text in the same area of ​​the frame image to be analyzed is less than a preset frequency threshold, the text content of the target text can be determined as background text.

[0139] For example, assuming that the preset frequency threshold is 30%, there are 10 frame images to be analyzed, and text appears in an area of ​​only one of the 10 frame images to be analyzed, while no text appears in the same area of ​​the other frame images to be analyzed. At this time, the frequency of text appearance in the same area of ​​the frame images to be analyzed is 1 / 10=10%, and 10% is less than the preset frequency threshold of 30%. At this time, the text content can be determined as background text.

[0140] S503: Determine whether the character endpoint position offset rate of the target character is less than a preset change rate threshold.

[0141] The text endpoint position deviation rate can be used to represent the change in the position of the text content in each frame of the image.

[0142] Considering that in actual applications, when the text content changes, the vertex coordinates of the connected area occupied by the text content in the frame image will change. Therefore, in an embodiment of the present application, the text endpoint position offset rate can be determined based on the change in the vertex coordinates of the connected area occupied by the text content.

[0143] Figure 6a , Figure 6b and Figure 6c A schematic diagram of an application scenario applicable to a subtitle positioning method provided in an embodiment of the present application, Figure 6aThis is a schematic diagram of a frame image. Figure 6a The "QQ Music" written in the upper left corner is the logo of this frame of picture, which is a logo-type text. Figure 6a The lyrics "I'm used to being buried deep in the fog" written at the bottom center of the frame image are the lyrics on the frame image. Figure 6a The background text in the center of the frame is "I slept through spring and didn't wake up until dawn. I heard birds singing everywhere. At night, there were sounds of wind and rain. How many flowers have fallen?"

[0144] Figure 6b Yes Figure 6a Schematic diagram of subtitle positioning for frame images. According to the traditional method Figure 6a The subtitles can be positioned by using the frame image to determine the location of each text area. Figure 6b It is presented in the form of coordinates. The four vertex coordinates of the first area where the logo text "QQ Music" is located are (x1, y1), (x2, y2), (x3, y3) and (x4, y4); among them, the values ​​of x1 and x3 are the same, the values ​​of x2 and x4 are the same; the values ​​of y1 and y2 are the same, and the values ​​of y3 and y4 are the same. The four vertex coordinates of the second area where the lyrics text "I am used to being buried deep in the fog" are located are (x5, y5), (x6, y6), (x7, y7) and (x8, y8); among them, the values ​​of x5 and x7 are the same, the values ​​of x6 and x8 are the same; the values ​​of y5 and y6 are the same, and the values ​​of y7 and y8 are the same. The four vertex coordinates of the third area where the background text "I sleep in spring and hear birds singing everywhere at dawn. At night, the sound of wind and rain makes me wonder how many flowers have fallen" are located are (x9, y9), (x 10 ,y 10 )、(x 11 ,y 11 ) and (x 12 ,y 12 ), since the third area where the background text is located is not distributed in the positive direction, the values ​​of the four vertex coordinates are different.

[0145] by Figure 6b Taking the frame picture shown as an example of the completed subtitle positioning, when identifying the text category of the target text in the frame picture to be analyzed, the changes in the vertex coordinate positions of the target text in the same area position in multiple frame pictures to be analyzed can be evaluated. Taking the logo-type text "QQ Music" as an example, the logo-type text appears in a similar position in each frame picture to be analyzed, all located in the upper left corner of the frame picture to be analyzed. And the vertex coordinates of the connected area occupied by the logo-type text in multiple frame pictures to be analyzed will hardly change. Therefore, by comparing the changes in the vertex coordinates of the target text in the same area position in multiple frame pictures to be analyzed, it can be identified whether the current target text is a logo-type text.

[0146] Considering that in actual applications, whether it is a logo text or a lyric text, the height of the text content will basically not change, only the length of the space occupied by the text will change. Therefore, the text endpoint position offset rate of the target text can be determined based on the change in the length of the connected area occupied by the text content.

[0147] In a specific implementation, the vertex coordinates of the connected area occupied by the text content at the first area position of the first frame image to be analyzed can be used as reference coordinates. Based on the reference coordinates, the length of the connected area can be calculated and used as the reference length. Then, the difference between the length of the connected area occupied by the text content of the remaining frame images to be analyzed and the reference length is calculated respectively, and the ratio of the absolute value of each difference to the reference length is used as the initial change rate. The average value of the initial change rate is used as the text endpoint position offset rate of the target text.

[0148] Combination Figure 6b In the frame picture shown, taking the logo text "QQ Music" as an example, the length of the connected area occupied by it is x2-x1 or x4-x3. Taking the lyrics text "I am used to being buried deep in the fog", the length of the connected area occupied by it is x6-x5 or x8-x7. By analogy, the length of the connected area occupied by the text content in each frame picture to be analyzed can be calculated.

[0149] The preset change rate threshold is set to evaluate the position deviation rate of the text endpoints, so as to distinguish between logo text and lyrics text. The value of the preset change rate threshold can be set according to actual needs. Considering that the connected area occupied by logo text is relatively fixed, the value of the preset change rate threshold can be set lower, for example, it can be set to 10%.

[0150] If the text endpoint position offset rate of the target text is less than the preset change rate threshold, it means that the vertex coordinates of the connected area occupied by the target text change little, and the probability that the target text belongs to the logo type text is higher. Therefore, if the text endpoint position offset rate of the target text is less than the preset change rate threshold, S504 can be executed.

[0151] If the text endpoint position offset rate of the target text is not less than the preset change rate threshold, it means that the vertex coordinates of the connected area occupied by the target text have changed greatly, and the probability that the target text belongs to lyrics is higher. Therefore, if the text endpoint position offset rate of the target text is not less than the preset change rate threshold, S505 can be executed.

[0152] S504: Determine the text content of the target text as a symbol-type text.

[0153] When the character endpoint position deviation rate of the target character is less than a preset change rate threshold, the character content of the target character may be determined as a logo-type character.

[0154] S505: Determine the text content of the target text as lyrics-related text.

[0155] When the text endpoint position offset rate of the target text is not less than a preset change rate threshold, the text content of the target text can be determined as lyrics-type text.

[0156] In the embodiment of the present application, in addition to being set in the form of a numerical value, the preset frequency threshold can also be set to correspond to different text categories based on the probability that the logo text, lyrics text, and background text may each appear in all frames to be analyzed. In order to distinguish the preset frequency thresholds corresponding to different text categories, the preset frequency threshold corresponding to the logo text can be called the first preset frequency threshold; the preset frequency threshold corresponding to the lyrics text can be called the second preset frequency threshold; and the preset frequency threshold corresponding to the background text can be called the third preset frequency threshold.

[0157] Taking setting three preset frequency thresholds as an example, in a specific implementation, for the recognition of logo-type text, it can be determined whether the frequency of text occurrence in the first area position of the frame image to be analyzed is greater than the first preset frequency threshold, and whether the text endpoint position offset rate in the first area position is less than the preset change rate threshold.

[0158] The first area position is used to indicate an area where characters whose appearance frequency is greater than a first preset frequency threshold and whose endpoint position offset rate is less than a preset change rate threshold are located.

[0159] The value of the first preset frequency threshold can be set according to actual needs. The purpose of setting the first preset frequency threshold is to identify logo text. Considering that logo text generally appears at a fixed position in each frame of a video file, the value of the first preset frequency threshold can be set higher, for example, it can be set to 95%.

[0160] If the frequency of occurrence of text at the first area position of the frame image to be analyzed is greater than the first preset frequency threshold, and the text endpoint position offset rate at the first area position is less than the preset change rate threshold, the text content at the first area position is determined to be logo-type text.

[0161] For example, assuming that there are 10 frame images to be analyzed, and the same text content exists at the same position in the upper left corner of these 10 frame images to be analyzed, the frequency of text appearance is 100%, which is greater than the first preset frequency threshold of 95%; and the text endpoint position offset rate is 0%, which is less than the preset change rate threshold of 10%. At this time, the text content can be judged as logo-type text.

[0162] For the recognition of lyrics, it can be determined whether the frequency of occurrence of the text in the second area position of the frame image to be analyzed is greater than the second preset frequency threshold, and whether the position offset rate of the text endpoint in the second area position is not less than the preset change rate threshold.

[0163] The value of the second preset frequency threshold can be set according to actual needs. The purpose of setting the second preset frequency threshold is to identify lyrics-like text. Considering that lyrics-like text generally appears in a relatively close position in each frame of the video file, the value of the second preset frequency threshold can be set higher. Considering that the position where lyrics-like text appears in the frame image is not as fixed as the position where logo-like text appears in the frame image, the value of the second preset frequency threshold can be set slightly lower than the first preset frequency threshold. For example, the value of the second preset frequency threshold can be set to 85%.

[0164] If the frequency of occurrence of text in the second area position of the frame image to be analyzed is greater than the second preset frequency threshold, and the text endpoint position offset rate in the second area position is not less than the preset change rate threshold, the text content in the second area position is determined to be lyrics-type text.

[0165] For example, assuming that there are 10 frame images to be analyzed, there is text content in the bottom center position of these 10 frame images to be analyzed. The bottom center position of the frame images to be analyzed is the second area position. At this time, the frequency of occurrence of text in the second area position is 10 / 10=100%, and 100% is greater than the second preset frequency threshold of 85%. 1 character is 1 length. Assuming that the lengths of the connected areas corresponding to the characters in the second area of ​​the 10 frames to be analyzed are 10, 8, 5, 6, 12, 15, 7, 2, 9, and 4 respectively, according to the calculation method of the character endpoint position offset rate introduced above, the character endpoint position offset rate is [(10-8) / 10+(10-5) / 10+(10-6) / 10+(12-10) / 10+(15-10) / 10+(10-7) / 10+(10-2) / 10+(10-9) / 10+(10-4) / 10] / 9=0.4=40%, 40% is greater than the preset change rate threshold of 10%. At this time, the text content can be determined as lyrics.

[0166] For the recognition of background text, it may be determined whether the frequency of occurrence of text in the third region of the frame image to be analyzed is less than a third preset frequency threshold.

[0167] The third area position is used to indicate an area where characters whose appearance frequency is less than a third preset frequency threshold are located.

[0168] The value of the third preset frequency threshold can be set according to actual needs. The purpose of setting the third preset frequency threshold is to identify background text. Considering that background text generally appears in certain frame images of video files, the value of the third preset frequency threshold can be set lower, for example, it can be set to 30%.

[0169] If the occurrence frequency of the text at the third area position of the frame image to be analyzed is less than the third preset frequency threshold, the text content at the third area position is determined to be background text.

[0170] In an embodiment of the present application, in addition to determining the text category corresponding to each text content based on the probability and area position of the text appearance in the frame image to be analyzed, the text category corresponding to each text content can also be determined by assigning values ​​to the pixels of the frame image to be analyzed based on the text and superimposing the numbers corresponding to the pixels.

[0171] In a specific implementation, the frame image to be analyzed can be converted into a binary image according to the position information corresponding to the text content in the frame image to be analyzed; two different numbers are used in the binary image to represent the pixel points in the text area and the pixel points in the non-text area respectively.

[0172] For the convenience of distinction and calculation, the number "1" can be used to represent the pixel points in the text area, and the number "0" can be used to represent the pixel points in the non-text area. For the convenience of introduction, the number "1" represents the pixel points in the text area, and the number "0" represents the pixel points in the non-text area.

[0173] The numbers corresponding to the pixels at the same position in the binary image are added together to obtain the superposition value corresponding to each pixel; the text category corresponding to each text content is determined based on the category value range to which the superposition value corresponding to each pixel belongs; wherein each text category has its corresponding category value range.

[0174] The text categories may include logo text, lyrics text and background text.

[0175] Generally, logo-like text appears at the same position in each frame of the image to be analyzed. Therefore, for logo-like text, the superposition value obtained by adding the numbers corresponding to the pixels at the same position is higher.

[0176] Lyrics text appears in the same or similar position in each frame of the image to be analyzed. Therefore, for lyrics text, the superposition value obtained by adding the numbers corresponding to the pixels at the same position is higher, but lower than the superposition value corresponding to the logo text.

[0177] Background text may only appear in certain frames of the frame images to be analyzed. Therefore, for background text, the superposition value obtained by adding the numbers corresponding to the pixels at the same position is lower.

[0178] In practical applications, the category value range can be obtained based on the number of frame images to be analyzed, the number of pixels representing the text area, and a set ratio.

[0179] For example, the number of frames to be analyzed is 10, and the number representing the pixel point on the text area is "1". The corresponding ratio of the logo text is 0.8 to 1 (excluding 0.8, including 1), the corresponding ratio of the lyrics text is 0.4 to 0.8 (excluding 0.4, including 0.8), and the corresponding ratio of the background text is 0 to 0.4 (including 0, including 0.4). Then the category value range corresponding to the logo text is 10*0.8=8, 10*1=10, that is, 8 to 10 (excluding 8, including 10); the category value range corresponding to the lyrics text is 10*0.4=4, 10*0.8=8, that is, 4 to 8 (excluding 4, including 8); the category value range corresponding to the background text is 10*0=0, 10*0.4=4, that is, 0 to 4 (excluding 0, including 4).

[0180] Considering that in actual applications, logo text, lyrics text and background text generally contain multiple characters, these continuous text contents can form a connected area. Therefore, when determining the category value range to which the superposition value corresponding to each pixel point belongs, the superposition values ​​of each pixel point in the connected area where the text is located can be added and averaged, and the obtained average value can be compared with the set category value range to determine the text category corresponding to each text content.

[0181] For example, assuming that there are 10 frame images to be analyzed, and there are 8 consecutive pixels in the upper left corner of each frame image to be analyzed, and the numbers corresponding to them are all "1", then these 8 pixels can be regarded as a connected area, and the superposition value of each pixel in the 10 frame images to be analyzed in the connected area is 10. The superposition values ​​of each pixel in the connected area are added and averaged, that is, (10+10+10+10+10+10+10+10) / 8=10, which belongs to the category value range of 8 to 10. Therefore, it can be determined that the text content corresponding to these 8 pixels is logo text.

[0182] In an embodiment of the present application, different marking methods can be set for different text categories based on the impact of different text categories when the video file is cropped vertically and the presentation form of different text categories in the video file.

[0183] In a specific implementation, the position information corresponding to the logo text can be copied to the remaining frames in each video clip; the position information corresponding to each line of lyrics text can be copied to the remaining frames in the video clip with the same lyrics. Considering that background text has almost no effect on vertical cropping, the position information corresponding to background text can be directly ignored.

[0184] Combined with the above Figure 6a , Figure 6b and Figure 6c . Figure 6c For Figure 6a Another frame picture with the same lyrics, Figure 6b The text category and location information of the determined frame image, Figure 6b The location information is presented in the form of coordinates, which can be Figure 6b The position information of the logo text and lyrics text is directly copied to Figure 6c frame picture, thereby completing the subtitle positioning of the 6c frame picture.

[0185] In the embodiment of the present application, the text category corresponding to each text content can be determined according to the position information corresponding to the text content in the frame image to be analyzed. After the text content in the frame image to be analyzed is marked, the subtitle position of all frame images in the video file can be marked according to the position information corresponding to each text category in the frame image to be analyzed, which effectively improves the efficiency of subtitle marking of the video file.

[0186] Figure 7 A schematic diagram of the structure of a subtitle positioning device provided in an embodiment of the present application. Figure 7 As shown, the subtitle locating device may include a matching unit 71, a video segment determining unit 72, a screening unit 73 and a subtitle determining unit 74;

[0187] The matching unit 71 is used to match the audio fingerprint of the song audio signal in the video file with a preset audio fingerprint library to determine the target song for audio fingerprint matching and the time position of the song audio signal in the target song; wherein the audio fingerprint library records the corresponding relationship between the song and the song fingerprint information;

[0188] A video segment determination unit 72 is used to determine the video segment corresponding to each lyric in the video file according to the distribution time of the lyrics of the target song and the time position of the song audio signal in the target song;

[0189] A screening unit 73, used to screen out a preset number of frame images from each video segment as frame images to be analyzed;

[0190] The subtitle determination unit 74 is used to determine the subtitle information in the frame picture to be analyzed, and determine the subtitle information of the remaining frame pictures in each video segment based on the subtitle information in the frame picture to be analyzed.

[0191] Optionally, the matching unit includes an extraction subunit, a conversion subunit and a determination subunit;

[0192] An extraction subunit, used for extracting a song audio signal from a video file;

[0193] A conversion subunit, for converting a song audio signal into a speech spectrogram;

[0194] The determination subunit is used to determine the audio fingerprint of the song audio signal based on each peak point in the speech spectrogram; the audio fingerprint is matched with the fingerprint information of each song in the audio fingerprint library to determine the target song for audio fingerprint matching and the time position of the song audio signal in the target song.

[0195] Optionally, the video segment determination unit includes a capture subunit and a determination subunit;

[0196] The interception subunit is used to intercept the song segment matching the song audio signal from the target song according to the time position of the song audio signal in the target song;

[0197] A determination subunit is used to determine the time range corresponding to each line of lyrics in the song clip according to the lyrics distribution time of the song clip; based on the time range corresponding to each line of lyrics in the song clip and the frame rate of the video file, determine the video clip corresponding to each line of lyrics in the video file.

[0198] Optionally, the subtitle information includes position information and text category; the subtitle determination unit includes an identification subunit and a category determination subunit;

[0199] The recognition subunit is used to use the text detection network model to identify the position information corresponding to the text content in the frame image to be analyzed;

[0200] The category determination subunit is used to determine the text category corresponding to each text content according to the position information corresponding to the text content in the frame image to be analyzed.

[0201] Optionally, the category determination subunit is used to determine the text content of the target text as background text if the frequency of occurrence of the target text in the same area position in the frame image to be analyzed is less than a preset frequency threshold; and / or if the frequency of occurrence of the target text in the same area position in the frame image to be analyzed is not less than a preset frequency threshold, and the text endpoint position offset rate of the target text is less than a preset change rate threshold, then determine the text content of the target text as logo text; and / or if the frequency of occurrence of the target text in the same area position in the frame image to be analyzed is not less than a preset frequency threshold, and the text endpoint position offset rate of the target text is not less than a preset change rate threshold, then determine the text content of the target text as lyrics text.

[0202] Optionally, the preset frequency threshold includes a first preset frequency threshold, a second preset frequency threshold and a third preset frequency threshold; wherein the first preset frequency threshold is greater than the second preset frequency threshold; the second preset frequency threshold is greater than the third preset frequency threshold;

[0203] A category determination subunit is used to determine the text content at the first area position of the frame image to be analyzed as logo-type text if the frequency of text appearance at the first area position of the frame image to be analyzed is greater than a first preset frequency threshold, and the text endpoint position offset rate at the first area position is less than a preset change rate threshold; and / or if the frequency of text appearance at the second area position of the frame image to be analyzed is greater than a second preset frequency threshold, and the text endpoint position offset rate at the second area position is not less than the preset change rate threshold, then determine the text content at the second area position as lyric-type text; and / or if the frequency of text appearance at the third area position of the frame image to be analyzed is less than a third preset frequency threshold, then determine the text content at the third area position as background-type text.

[0204] Optionally, a category determination subunit is used to convert the frame image to be analyzed into a binary image based on the position information corresponding to the text content in the frame image to be analyzed; two different numbers are used in the binary image to represent the pixel points on the text area and the pixel points on the non-text area respectively; the numbers corresponding to the pixel points at the same position in the binary image are added to obtain the superposition value corresponding to each pixel point; based on the category value range to which the superposition value corresponding to each pixel point belongs, the text category corresponding to each text content is determined; wherein each text category has its corresponding category value range; the text categories include logo text, lyrics text and background text.

[0205] Optionally, the subtitle determination unit is used to copy the position information corresponding to the logo text to the remaining frame images in each video segment; and copy the position information corresponding to each line of lyrics in the lyrics text to the remaining frame images in the video segment with the same lyrics.

[0206] In the present application, the audio fingerprint of the song audio signal in the video file is matched with the preset audio fingerprint library to determine the target song matched by the audio fingerprint and the time position of the song audio signal in the target song; wherein the audio fingerprint library records the corresponding relationship between the song and the song fingerprint information. The audio fingerprint can be information generated based on the characteristics of the song audio signal for uniquely identifying the song audio signal. The song fingerprint information is generated in the same way as the audio fingerprint. By matching the audio fingerprint with the preset audio fingerprint library, the target song to which the song audio signal belongs and the time position of the song audio signal in the target song can be determined. Based on the duration of each line of lyrics in the song and the number of frame images played per unit time in the video file, it can be known that the same line of lyrics often appears in multiple frame images at the same time. If each frame image in the video file is analyzed, it is a repeated analysis for the frame images containing the same lyrics. In order to reduce the time spent on subtitle positioning, the video clip corresponding to each line of lyrics in the video file can be determined according to the lyrics distribution time of the target song and the time position of the song audio signal in the target song. A preset number of frame images are selected from each video clip as the frame images to be analyzed. After screening out the frame images to be analyzed, it is only necessary to locate the subtitles of the frame images to be analyzed and determine the subtitle information in the frame images to be analyzed. Relying on the subtitle information in the frame images to be analyzed, the subtitle information of the remaining frame images in each video clip can be quickly determined. In this technical solution, by matching the audio fingerprint, the target song to which the song audio signal belongs and the time position of the song audio signal in the target song can be determined. According to the distribution time of the lyrics of the target song and the time position of the song audio signal in the target song, a preset number of frame images can be screened out from the video clip corresponding to each line of lyrics as the frame images to be analyzed. Relying on the subtitle positioning results of the frame images to be analyzed, the subtitle positions of all frame images in the video file can be quickly determined, which greatly reduces the time spent on subtitle positioning while ensuring the accuracy of subtitle positioning.

[0207] Furthermore, an embodiment of the present application also provides an electronic device. Figure 8 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be regarded as any limitation on the scope of use of the present application.

[0208] Figure 8The present invention provides a schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the subtitle positioning method disclosed in any of the above embodiments. In addition, the electronic device 20 in this embodiment may be a server.

[0209] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0210] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon may include an operating system 221, a computer program 222 and video data 223, etc. The storage method can be temporary storage or permanent storage.

[0211] The operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, so as to realize the operation and processing of the massive video data 223 in the memory 22 by the processor 21, which can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the subtitle positioning method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks. The data 223 can include various video data collected by the electronic device 20.

[0212] Furthermore, an embodiment of the present application also discloses a storage medium, in which a computer program is stored. When the computer program is loaded and executed by a processor, the steps of the subtitle positioning method disclosed in any of the aforementioned embodiments are implemented.

[0213] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0214] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0215] The above is a detailed introduction to a subtitle positioning method, device, electronic device and computer-readable storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A subtitle positioning method, characterized in that: include: Matching the audio fingerprint of the song audio signal in the video file with a preset audio fingerprint library to determine the target song matched by the audio fingerprint and the time position of the song audio signal in the target song; wherein the audio fingerprint library records the correspondence between the song and the song fingerprint information; Determine the video clip corresponding to each line of lyrics in the video file according to the lyrics distribution time of the target song and the time position of the song audio signal in the target song; Filtering a preset number of frame images from each of the video clips as frame images to be analyzed; Determine the subtitle information in the frame picture to be analyzed, and determine the subtitle information of the remaining frame pictures in each of the video segments based on the subtitle information in the frame picture to be analyzed; The subtitle information includes position information and text category; based on the change of the position information corresponding to the text content, the change of the position of the area where the text content is located in different frame images to be analyzed is determined, so as to determine the text category corresponding to each text content; the text category refers to the category to which the text content of the subtitle belongs, including lyrics text, logo text and background text; the text endpoint position offset rate is used to characterize the change of the position area where the text content is located in each frame image; the vertex coordinates of the connected area occupied by the text content at the first area position of the first frame image to be analyzed are used as reference coordinates, and the length of the connected area is calculated based on the reference coordinates, and the length is used as the reference length; the difference between the length of the connected area occupied by the text content of the remaining frame images to be analyzed and the reference length is calculated respectively, and the ratio of the absolute value of each difference to the reference length is used as the initial change rate; the average value of all initial change rates is used as the text endpoint position offset rate of the target text.

2. The subtitle positioning method according to claim 1, characterized in that: The step of matching the audio fingerprint of the song audio signal in the video file with a preset audio fingerprint library to determine the target song matched by the audio fingerprint and the time position of the song audio signal in the target song includes: Extracting a song audio signal from a video file and converting the song audio signal into a speech spectrogram; Based on each peak point in the speech spectrogram, determining the audio fingerprint of the song audio signal; The audio fingerprint is matched with the fingerprint information of each song in the audio fingerprint library to determine the target song matched by the audio fingerprint and the time position of the song audio signal in the target song.

3. The subtitle positioning method according to claim 1, characterized in that: Determining the video clip corresponding to each line of lyrics in the video file according to the lyrics distribution time of the target song and the time position of the song audio signal in the target song comprises: According to the time position of the song audio signal in the target song, extracting a song segment matching the song audio signal from the target song; Determine the time range corresponding to each line of lyrics in the song segment according to the lyrics distribution time of the song segment; Based on the time range corresponding to each line of lyrics in the song segment and the frame rate of the video file, the video segment corresponding to each line of lyrics in the video file is determined.

4. The subtitle positioning method according to claim 1, characterized in that: The subtitle information includes position information and text category; and determining the subtitle information in the frame picture to be analyzed includes: Using a text detection network model, identifying position information corresponding to the text content in the frame image to be analyzed; According to the position information corresponding to the text content in the frame image to be analyzed, the text category corresponding to each of the text contents is determined.

5. The subtitle positioning method according to claim 4, characterized in that: Determining the text category corresponding to each text content according to the position information corresponding to the text content in the frame image to be analyzed includes: If the occurrence frequency of the target text in the same area of ​​the frame image to be analyzed is less than a preset frequency threshold, the text content of the target text is determined to be background text; and / or If the occurrence frequency of the target text in the same area of ​​the frame image to be analyzed is not less than a preset frequency threshold, and the text endpoint position offset rate of the target text is less than a preset change rate threshold, the text content of the target text is determined to be a logo text; and / or If the occurrence frequency of the target text in the same area of ​​the frame image to be analyzed is not less than the preset frequency threshold, and the text endpoint position offset rate of the target text is not less than the preset change rate threshold, the text content of the target text is determined to be lyrics-type text.

6. The subtitle positioning method according to claim 5, characterized in that: The preset frequency threshold includes a first preset frequency threshold, a second preset frequency threshold and a third preset frequency threshold; wherein the first preset frequency threshold is greater than the second preset frequency threshold; the second preset frequency threshold is greater than the third preset frequency threshold; Determining the text category corresponding to each text content according to the position information corresponding to the text content in the frame image to be analyzed includes: If the frequency of occurrence of text at the first area position of the frame image to be analyzed is greater than a first preset frequency threshold, and the position offset rate of the text endpoints at the first area position is less than a preset change rate threshold, the text content at the first area position is determined to be a logo-type text; and / or If the frequency of occurrence of the text at the second area position of the frame image to be analyzed is greater than a second preset frequency threshold, and the position offset rate of the text endpoints at the second area position is not less than the preset change rate threshold, the text content at the second area position is determined to be lyrics-like text; and / or If the occurrence frequency of the text at the third area position of the frame image to be analyzed is less than a third preset frequency threshold, the text content at the third area position is determined to be background text.

7. The subtitle positioning method according to claim 4, characterized in that: Determining the text category corresponding to each text content according to the position information corresponding to the text content in the frame image to be analyzed includes: According to the position information corresponding to the text content in the frame image to be analyzed, the frame image to be analyzed is converted into a binary image; in the binary image, two different numbers are used to represent the pixel points in the text area and the pixel points in the non-text area respectively; Add the numbers corresponding to the pixels at the same position in the binary image to obtain the superposition value corresponding to each pixel; According to the category value range to which the superposition value corresponding to each pixel point belongs, the text category corresponding to each text content is determined; wherein each text category has its corresponding category value range; the text categories include logo text, lyrics text and background text.

8. The subtitle positioning method according to any one of claims 5 to 7, characterized in that: The step of determining the subtitle information of the remaining frame pictures in each of the video segments based on the subtitle information in the frame picture to be analyzed includes: Copying the position information corresponding to the logo text to the remaining frame images in each of the video clips; The position information corresponding to each line of lyrics in the lyrics-like text is copied to the remaining frame images in the video clip with the same lyrics.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the subtitle positioning method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: Used to store a computer program, which, when executed by a processor, implements the subtitle positioning method according to any one of claims 1 to 8.