Video subtitle extraction method, device and computer-readable storage medium

Abstract

The embodiment of the present application discloses a video subtitle extraction method, device and computer-readable storage medium, wherein the method comprises: obtaining a target video, and determining a target subtitle area of ​​the target video based on multiple image frames in the target video. The subtitle area of ​​each image frame in the target video is determined by the target subtitle area, and the initial subtitle text of the target video is obtained based on the subtitle area of ​​each image frame, wherein the initial subtitle text contains multiple initial subtitle clauses with an arrangement order. Based on the similarity between each adjacent initial subtitle clause, multiple target subtitle clauses with an arrangement order are obtained from the multiple initial subtitle clauses, wherein the characters between adjacent target subtitle clauses are different from each other, and finally the target subtitle text of the target video is obtained based on the multiple target subtitle clauses with an arrangement order. The present application can solve the problem of repeated subtitle extraction, and the operation is simple, the subtitle extraction effect is good, and the applicability is strong.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a video subtitle extraction method, device and computer-readable storage medium. Background Art

[0002] With the development of Internet technology, various karaoke applications provide people with indispensable leisure and entertainment activities such as singing, recording singing videos, and uploading videos. K-song applications can allow users to upload music-related videos in the application, such as a cover video of a song that the user has sung and recorded. In order to push videos associated with the user's preferred songs (such as a cover video of the song) to optimize the push strategy of the karaoke application video, it is necessary to locate the song name by automatically identifying the lyrics and subtitles in the video uploaded by the user, so as to associate the video with the identified song name.

[0003] The inventors of the present application discovered during the course of research and practice that, in the prior art, in the process of identifying lyrics and subtitles in a video, by intercepting image frames at intervals in the video and identifying text, there would be a problem of repeated extraction of certain lyrics, resulting in poor lyrics extraction effect, which leads to a low success rate in locating song names, an inability to accurately push related videos, and a poor user experience. Summary of the invention

[0004] The embodiments of the present application provide a video subtitle extraction method, device and computer-readable storage medium, which can solve the problem of repeated subtitle extraction, are simple to operate, have good subtitle extraction effect and strong applicability.

[0005] In a first aspect, an embodiment of the present application provides a video subtitle extraction method, the method comprising:

[0006] Acquire a target video, and determine a target subtitle area of ​​the target video based on a plurality of image frames in the target video;

[0007] Determining the subtitle area of ​​each image frame in the target video based on the target subtitle area, and obtaining the initial subtitle text of the target video based on the subtitle area of ​​each image frame, wherein the initial subtitle text has a plurality of initial subtitle clauses in an arranged order;

[0008] Based on the similarity between each adjacent initial subtitle sentence, a plurality of target subtitle sentences having an arrangement order are obtained from the plurality of initial subtitle sentences, wherein the characters between adjacent target subtitle sentences are different from each other;

[0009] The target subtitle text of the target video is obtained based on the multiple target subtitle clauses having an arrangement order.

[0010] In a possible implementation manner, determining the target subtitle area of ​​the target video based on the multiple image frames in the target video includes:

[0011] Acquire multiple image frames in the target video, and input the multiple image frames into a pixel classification model to obtain the probability that each pixel in each image frame output by the pixel classification model belongs to a subtitle area;

[0012] Based on the probability that each pixel point in each of the above image frames belongs to the subtitle area, obtaining subtitle positioning information of the above target video;

[0013] A target subtitle area of ​​the target video is determined based on the subtitle positioning information.

[0014] In a possible implementation, the acquiring the subtitle location information of the target video based on the probability that each pixel point in each image frame belongs to the subtitle area includes:

[0015] Based on the probability that each pixel point in each of the above image frames belongs to the subtitle area, obtaining the number of times each of the above pixel points is determined to be the subtitle area in each of the above image frames;

[0016] A target pixel is determined from the pixels based on the number of times the pixel is determined as a subtitle area in the image frames, and subtitle positioning information of the target video is acquired based on the target pixel.

[0017] In a possible implementation, determining a target pixel from each pixel based on the number of times each pixel is determined as a subtitle area in each image frame, and acquiring subtitle positioning information of the target video based on the target pixel includes:

[0018] Determine from the above-mentioned pixels the pixel points whose number of times of being determined as the subtitle area in the above-mentioned image frames is greater than or equal to the number threshold as target pixel points to obtain a plurality of target pixel points, wherein the plurality of target pixel points constitute one or more pixel point sets;

[0019] From the one or more pixel point sets, select one or more target pixel point sets including a number of the target pixel points greater than a preset number;

[0020] A target pixel point set in a designated display area is selected from the one or more target pixel point sets, and subtitle positioning information of the target video is obtained based on the selected target pixel point set.

[0021] In a possible implementation, the subtitle positioning information includes the coordinates of the starting point of the region, the length of the region, and the width of the region; and determining the target subtitle region of the target video based on the subtitle positioning information includes:

[0022] Determine the preset area extension size based on the area length and area width included in the subtitle positioning information, and determine the target area starting coordinates, target area length and target area width based on the area starting point coordinates, area length, area width and the preset area extension size;

[0023] An area consisting of the target area starting coordinates, the target area length and the target area width is determined as the target subtitle area.

[0024] In a possible implementation, determining the subtitle area of ​​each image frame in the target video based on the target subtitle area, and obtaining the initial subtitle text of the target video based on the subtitle area of ​​each image frame includes:

[0025] Acquire multiple image frames of the target video based on a preset frame extraction frequency, and determine the subtitle area of ​​each image frame from the multiple image frames based on the target subtitle area;

[0026] Based on the subtitle area of ​​each image frame, target sub-images of each image frame are cut out from each image frame to obtain a plurality of target sub-images, and the plurality of target sub-images are sorted according to the order of appearance of the image frames to which the target sub-images belong in the target video;

[0027] An image to be recognized is generated based on the sorted multiple target sub-images, and character recognition is performed on the image to be recognized to obtain an initial subtitle text of the target video.

[0028] In a possible implementation, the similarity degree includes an edit distance and a length distance; and the step of obtaining a plurality of target subtitle sentences having an arrangement order from the plurality of initial subtitle sentences based on the similarity degree between the adjacent initial subtitle sentences includes:

[0029] Detecting the minimum target number of editing operations required to transform any initial subtitle sentence into its adjacent previous initial subtitle sentence, and obtaining the editing distance of any initial subtitle sentence based on the target number of editing operations;

[0030] Detecting the number of characters that differ between any initial subtitle sentence and its adjacent previous initial subtitle sentence, and obtaining the length distance of any initial subtitle sentence based on the number of characters that differ;

[0031] A target subtitle sentence is determined from the plurality of initial subtitle sentences based on the edit distance and the length distance of each of the initial subtitle sentences.

[0032] In a possible implementation, determining a target subtitle sentence from the plurality of initial subtitle sentences based on the edit distance and the length distance of each of the initial subtitle sentences includes:

[0033] Determine the first initial subtitle clause among the multiple initial subtitle clauses as an alternative subtitle clause;

[0034] If the difference between the edit distance and the length distance of any initial subtitle clause after the first initial subtitle clause is not less than a preset threshold, the initial subtitle clause and the previous initial subtitle clause adjacent to the initial subtitle clause are both determined as the candidate subtitle clauses; otherwise, the previous initial subtitle clause adjacent to the initial subtitle clause is removed and the initial subtitle clause is determined as the candidate subtitle clause to determine the candidate subtitle clause from the multiple initial subtitle clauses; wherein the previous initial subtitle clause adjacent to the second initial subtitle clause in the multiple initial subtitle clauses is the first initial subtitle clause;

[0035] All the determined candidate subtitle sentences are determined as target subtitle sentences.

[0036] In a second aspect, an embodiment of the present application provides a computer device, the computer device comprising: a processor, a memory, and a network interface;

[0037] The above-mentioned processor is connected to a memory and a network interface, wherein the network interface is used to provide a data communication function, the above-mentioned memory is used to store program codes, and the above-mentioned processor is used to call the above-mentioned program codes to execute the method in the first aspect of the embodiment of the present application.

[0038] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, which stores instructions. When the instructions are executed on the above-mentioned terminal device, the terminal device executes the video subtitle extraction method provided by the above-mentioned first aspect and / or any possible implementation method of the first aspect, and can also achieve the beneficial effects of the method provided by the first aspect.

[0039] In a fourth aspect, an embodiment of the present application provides a video subtitle extraction device, the device comprising:

[0040] An acquisition module, used to acquire a target video;

[0041] A target subtitle region generating module, used for determining a target subtitle region of the target video based on a plurality of image frames in the target video acquired by the acquisition module;

[0042] An initial subtitle text generation module is used to determine the subtitle area of ​​each image frame in the target video based on the target subtitle area of ​​the target subtitle area generation module, and obtain the initial subtitle text of the target video based on the subtitle area of ​​each image frame, wherein the initial subtitle text includes a plurality of initial subtitle clauses in an arrangement order;

[0043] A target subtitle text generation module, configured to obtain a plurality of target subtitle sentences having an arrangement order from the plurality of initial subtitle sentences based on the similarity between the adjacent initial subtitle sentences of the initial subtitle text generation module, wherein the characters of the adjacent target subtitle sentences are different from each other;

[0044] The target subtitle text generation module is further used to obtain the target subtitle text of the target video based on the multiple target subtitle clauses having an arrangement order.

[0045] In an embodiment of the present application, a target subtitle area can be obtained from a target video to obtain the position distribution of the target subtitle text in each image frame of the video, and the initial subtitle text (including multiple initial subtitle clauses with an arrangement order) can be obtained from each image frame of the target video through the target subtitle area. Based on the similarity between each adjacent clause in each initial subtitle clause, the target subtitle clause extraction is performed on the above multiple initial subtitle clauses to obtain multiple target subtitle clauses with an arrangement order. In the process of extracting the above target subtitle clauses, the initial subtitle clauses that are incomplete or repeated with the subtitles in the target video can be removed, so that the target subtitle text can be obtained through multiple target subtitle clauses. The problem of repeated subtitle extraction can be solved, the operation is simple, the subtitle extraction effect is good, and the applicability is strong. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 It is a schematic diagram of the system architecture provided by the embodiment of the present application;

[0048] Figure 2 It is a flowchart of a video subtitle extraction method provided in an embodiment of the present application;

[0049] Figure 3 It is a schematic diagram of a scenario of the video subtitle extraction method provided by an embodiment of the present application;

[0050] Figure 4is another scenario schematic diagram of the video subtitle extraction method provided by an embodiment of the present application;

[0051] Figure 5 is another scenario schematic diagram of the video subtitle extraction method provided by an embodiment of the present application;

[0052] Figure 6 is another scenario schematic diagram of the video subtitle extraction method provided by an embodiment of the present application;

[0053] Figure 7a is a schematic diagram of text detection for optical character recognition;

[0054] Figure 7b It is a text recognition diagram of optical character recognition;

[0055] Figure 8 is another scenario schematic diagram of the video subtitle extraction method provided by an embodiment of the present application;

[0056] Fig. 9 is a structural diagram of a video subtitle extraction device provided in an embodiment of the present application;

[0057] Fig.10 It is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0059] The video subtitle extraction method provided in the embodiment of the present application is applicable to applications with video viewing and / or uploading functions (such as video applications, live broadcast applications, short video applications, and karaoke applications, etc.), and can process the video containing subtitles in the above application (for the convenience of description, the target video can be used as an example to illustrate) to obtain the subtitle text from the target video (which can be the target subtitle text, that is, the subtitle text that is completely consistent with the subtitles in the target video). The categories of the above target video include but are not limited to movies, news, variety shows, training courses, and user-generated content (UGC) videos, etc. The above subtitles can be the title of the movie, the cast, lyrics, dialogues, character or background introductions, place names, and era information. It can be determined according to the actual application scenario requirements, and is not limited here. The target video in the above application can be further processed based on the target subtitle text to achieve a better user experience. For example, the target video can be associated with a relevant category label based on the target subtitle text, so as to classify the above target video to place the target video in the corresponding video channel, so that users can watch their favorite videos on video channels of different categories. Alternatively, the target video can also be subject to a health review based on the target subtitle text. Developers of applications (such as video applications and live broadcast applications) can obtain the target subtitle text of the target video to evaluate whether the content of the target video is healthy and compliant, and remove target videos with unqualified content (such as sensitive information or illegal information) from the shelves to ensure the legality of the application.

[0060] In addition, during the use of the relevant application (such as a karaoke application), the user can not only sing the favorite songs when using the karaoke application, but also upload the target video (such as a UGC video) to the karaoke application. The above-mentioned UGC video can be a video shot and edited by the user through a device with a video recording function such as a mobile phone, a digital camera, and a tablet computer. Alternatively, it can also be a video obtained by the user from the Internet and edited by the obtained video. For example, user A records the target video of the cover song B (the target video contains the lyrics of song B) through relevant devices (mobile phones, cameras, etc.), and based on the above-mentioned target video, the corresponding target subtitle text can be obtained, and the name of song B is obtained based on the target subtitle text. The cover video of song B uploaded by user A is associated with song B (for the convenience of description, the song associated with the target video is called the target song), that is, when other users use the karaoke application (such as opening the application page related to song B), they can view the cover video of song B uploaded by user A on the relevant application page, so as to push the above-mentioned target video more effectively and improve the user's experience. If the target subtitle text is greatly different from the lyrics of the corresponding target song or there are repeated lyrics when the song name is obtained based on the target subtitle text, the accuracy of locating the song name will be affected. Therefore, in the process of extracting the target subtitle text of the target video, the obtained subtitle text is further processed (such as deduplication) to obtain a target subtitle text that is consistent with the lyrics of the corresponding song and has no duplication, which is very important for accurately locating and associating the corresponding song. For the convenience of description, the following will be used as an example to extract the target subtitle text (i.e., the complete lyrics of the corresponding song) for the UGC video uploaded by the user during the development and / or use of the K song application.

[0061] In the video subtitle extraction method provided in the embodiment of the present application, the position distribution of the target subtitle text in each image frame of the video is obtained by obtaining the target subtitle area from the target video, and the initial subtitle text (including multiple initial subtitle clauses) is obtained from each image frame of the target video through the target subtitle area. Based on the similarity between each adjacent clause in each initial subtitle clause, the target subtitle clause extraction is performed on the above multiple initial subtitle clauses to obtain multiple target subtitle clauses. In the above target subtitle clause extraction process, the initial subtitle clauses that are significantly different from the lyrics of the corresponding song (such as incomplete sentences) or repeated can be removed, so that the target subtitle text can be obtained through multiple target subtitle clauses, which can solve the problem of repeated subtitle extraction, simple operation, good subtitle extraction effect, and strong applicability.

[0062] See also Figure 1 , Figure 1 Schematic diagram of the system architecture provided by the embodiment of the present application. Figure 1As shown, the system architecture may include a business server 100 and a terminal cluster, and the terminal cluster may include: terminal devices 200a, terminal device 200b, terminal device 200c, ..., terminal device 200n and other terminal devices. Among them, the above-mentioned business server 100 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud database, cloud service, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and big data and artificial intelligence platform. The terminal device (including terminal device 200a, terminal device 200b, terminal device 200c, ..., terminal device 200n) can be a smart phone, tablet computer, laptop computer, desktop computer, PDA, mobile Internet device (mobile internet device, MID), wearable device (such as smart watch, smart bracelet, etc.), smart computer, smart car and other smart terminals. Among them, the business server 100 can establish a communication connection with each terminal device in the terminal cluster, and the terminal devices in the terminal cluster can also establish a communication connection. In other words, the business server 100 can establish a communication connection with each terminal device in the terminal device 200a, the terminal device 200b, the terminal device 200c, ..., the terminal device 200n. For example, a communication connection can be established between the terminal device 200a and the business server 100. A communication connection can be established between the terminal device 200a and the terminal device 200b, and a communication connection can also be established between the terminal device 200a and the terminal device 200c. Among them, the above-mentioned communication connection does not limit the connection method, and can be directly or indirectly connected through a wired communication method, or directly or indirectly connected through a wireless communication method, etc., which can be determined according to the actual application scenario, and this application does not limit it here.

[0063] It should be understood that Figure 1 Each terminal device in the terminal cluster shown in the figure may be installed with an application client. When the application client runs in each terminal device, it can be respectively connected to the above Figure 1The business servers 100 shown in the figure perform data interaction, so that the business server 100 can receive business data from each terminal device (such as UGC videos uploaded by users through terminal devices). Among them, the application client can be a social application, instant messaging application, live broadcast application, game application, short video application, video application, music application, K song application, shopping application, novel application, payment application, etc., which has the function of displaying text, images, audio and video and other data information. It can be determined according to the actual application scenario requirements and is not limited here. Among them, the application client can be an independent client or an embedded sub-client integrated in a client (such as an instant messaging client, a social client, etc.), which can be determined according to the actual application scenario and is not limited here. Taking the K song application as an example, when a user uses the K song application through a terminal device, he can not only sing his favorite songs, but also upload UGC videos related to music (such as a song cover video that a user has covered and recorded) to the business server 100 through the terminal device, and the business server 100 receives the UGC video sent by the above user. Each terminal device (terminal device 200a, terminal device 200b, terminal device 200c, ..., terminal device 200n) that establishes a communication connection with the service server 100 can watch the UGC video sent by the user through the karaoke application. In addition, after receiving the UGC video sent by the user through the karaoke application, the service server 100 can extract the video subtitles from the video and associate the target song with the extracted subtitles to better push the video. The method provided in the embodiment of the present application can be performed by Figure 1 The service server 100 shown in the figure may also be executed by a terminal device (such as Figure 1 The method may be executed by any one of the terminal devices 200a, 200b, ..., 200n shown in the figure, or may be executed jointly by the terminal device and the service server. The specific method may be determined according to the actual application scenario and is not limited here.

[0064] In some feasible implementations, the target video can be sent to the business server 100 by the terminal device 200a (installed with a karaoke application), and the business server 100 extracts video subtitles based on the received target video to obtain the subtitle text of the target video (which can be called the target subtitle text). The terminal device 200a can send the target video (such as a cover video of song B recorded by user A) to the business server 100 through the karaoke application, and the business server 100 obtains the target subtitle area based on the received target video, thereby obtaining the position distribution of the target subtitle text in each image frame of the video. The initial subtitle text (including multiple initial subtitle clauses) is obtained from each image frame of the target video through the target subtitle area, and the target subtitle clauses are extracted from the above multiple initial subtitle clauses based on the similarity between each adjacent clause in each initial subtitle clause to obtain multiple target subtitle clauses. In the above target subtitle clause extraction process, the initial subtitle clauses that are significantly different from or repeated from the lyrics of the corresponding target song can be removed, so that the business server 100 obtains the target subtitle text that is completely consistent with the lyrics of the song corresponding to the target video (i.e., the target song) through multiple target subtitle clauses. It solves the problem of repeated subtitle extraction, is simple to operate, has good subtitle extraction effect and strong applicability.

[0065] For the convenience of description, the service server will be used as the execution subject of the method provided in the embodiment of the present application, and an implementation method of extracting video subtitles through the service server will be specifically described through an embodiment.

[0066] See also Figure 2 , Figure 2 It is a flow chart of a video subtitle extraction method provided in an embodiment of the present application.

[0067] like Figure 2 As shown, the method comprises the following steps:

[0068] S101, obtaining a target video, and determining a target subtitle area of ​​the target video based on a plurality of image frames in the target video.

[0069] In some feasible implementations, the business server can obtain the target video (which can be a UGC video, which can be a cover video of the target song) sent by the terminal device (such as the terminal device 200a) through the application client (such as a karaoke application) installed thereon, and determine the target subtitle area of ​​the target video (i.e., the area where the target subtitle text is located in the target video) based on the multiple image frames received in the target video, so that the target subtitle text (i.e., the subtitle text consistent with the lyrics of the target song) can be obtained from the target video based on the above target subtitle area. Specifically, the business server can obtain multiple image frames in the target video, and the above multiple image frames can be multiple image frames extracted from the target video that carry the initial subtitle clause. Please refer to Figure 3 , Figure 3 Schematic diagram of a scenario of the video subtitle extraction method provided by the embodiment of the present application. Figure 3 As shown, the image frame 30 in the figure may be one of multiple image frames extracted from the target video, and the image frame 30 includes an initial subtitle clause 31 and other video images, and the content of the initial subtitle clause 31 is: "I am used to being buried deep in the fog and competing with myself". Here, the area where the initial subtitle clause 31 is located in the image frame 30 is the subtitle area of ​​the image frame, and the target subtitle area of ​​the target video can be obtained based on the subtitle area of ​​the image frame and the subtitle areas of other extracted image frames.

[0070] In some feasible implementations, the acquired multiple image frames can be input into a pixel classification model (which can be a pixel classification model based on the VGG16 network architecture), and the probability that each pixel in each image frame belongs to the subtitle area of ​​each image frame can be obtained through the model, so that the target subtitle area of ​​the target video can be obtained based on the probability that each pixel in each image frame belongs to the subtitle area of ​​each image frame. Here, the subtitle area of ​​the above-mentioned image frame can include a text area (the text part in the subtitle area) and a non-text area (other areas in the subtitle area except the text, such as the gap between the text). Specifically, for the image frames input into the pixel classification model, a first probability map of the probability that each pixel in each image frame belongs to the text area and a second probability map of the probability that each pixel belongs to the non-text area can be obtained (the size of the probability map can be the same as that of each image frame), and a binary map of the subtitle area of ​​each image frame can be obtained based on the first probability map and the second probability map of each image frame, and the binary map of the subtitle area of ​​each image frame reflects the specific position of the subtitle area of ​​each image frame.

[0071] See also Figure 4 , Figure 4 FIG. 1 is another schematic diagram of a video subtitle extraction method provided by an embodiment of the present application. Figure 4 As shown, the image frame 30 in the figure can be one of multiple image frames extracted from the target video. After the image frame is input into the pixel classification model, a first probability map and a second probability map are obtained. Based on the first probability map and the second probability map, a subtitle region binary map 40 of the image frame 30 can be obtained. The subtitle region binary map 40 includes a subtitle region 41 (i.e., a white region in the subtitle region binary map) of the image frame 30 obtained based on the first probability map and the second probability map. It can be understood that there are other positions in the subtitle region binary map 40 that can be identified as subtitle regions ( Figure 4 not shown).

[0072] In some feasible implementations, inter-frame synthesis can be performed based on the above-mentioned multiple subtitle area binary images, that is, the total number of times each pixel is judged as a subtitle area (including text area and non-text area) in each image frame can be counted, and the target pixel is determined based on the total number of times each pixel is judged as a subtitle area, and the subtitle positioning information of the above-mentioned target video can be obtained based on the above-mentioned target pixel. For example, if the above-mentioned determination threshold is 2, all pixels that are judged as subtitle areas with a number greater than or equal to 2 are taken as target pixels to obtain multiple target pixels, and the above-mentioned multiple target pixels can constitute one or more pixel sets. It can be understood that the above-mentioned one or more pixel sets can be one or more target subtitle areas of the above-mentioned target video. If there are multiple target subtitle areas, the target subtitle area related to the target subtitle text can be extracted from the multiple target subtitle areas. Based on the above-mentioned multiple target pixels obtained, a target subtitle area binary image can be obtained, and the above-mentioned target subtitle area binary image is a synthesis of the above-mentioned multiple subtitle area binary images, which reflects the target subtitle area of ​​the above-mentioned target video as a whole.

[0073] See also Figure 5 , Figure 5 FIG. 1 is another schematic diagram of a video subtitle extraction method provided by an embodiment of the present application. Figure 5 As shown, Figure 5 The inter-frame comprehensive map 50 in the figure includes a subtitle area 51 and a subtitle area 52 (that is, the total number of times each pixel in the above subtitle area is determined to be a subtitle area in each image frame is greater than or equal to 2), wherein the subtitle area 511 in the subtitle area 51 indicates that the total number of times each pixel in the area is determined to be a subtitle area in each image frame is 4 times, the subtitle area 512 indicates that the total number of times each pixel in the area is determined to be a subtitle area in each image frame is 6 times, and the subtitle area 513 indicates that the total number of times each pixel in the area is determined to be a subtitle area in each image frame is 8 times. The total number of times each pixel in the subtitle area 52 is determined to be a subtitle area in each image frame is 6 times. If the number of times a pixel is determined to be a subtitle area in each image frame reaches 2 times or more, then the pixel belongs to the target pixel in the target subtitle area binary map, and multiple target pixels can constitute one or more pixel sets, and each pixel set can be a target subtitle area. The above target subtitle area binary map is as shown in FIG. Figure 5 As shown in the binary image 60 in FIG. 6 , the binary image 60 includes a target subtitle area 61 and a target subtitle area 62. It can be understood that among the pixels in the target subtitle area binary image determined to belong to the target subtitle area, some pixels do not belong to the target subtitle text (for example, other subtitles in the target video except the lyrics subtitles, such as Figure 562 in the target subtitle area). One or more target subtitle areas containing target pixel points greater than a preset number (e.g., greater than 150 pixel points) can be extracted from the multiple target subtitle areas (or pixel point sets) in the target subtitle area binary image. A target subtitle area in a specified display area (e.g., the bottom of the display area of ​​the image frame, i.e., the lyrics in the target subtitle are located at the bottom) is obtained from the one or more target subtitle areas, and the positioning information of the target video is extracted based on the target subtitle area, thereby effectively avoiding the inclusion of some pixel points that do not belong to the target subtitle text into the target subtitle area, and further improving the accuracy of the target subtitle area. Please refer to Figure 5 , Figure 5 After the target subtitle area is extracted from the binary image 60, a binary image 70 can be obtained. The binary image 70 includes a target subtitle area 71, that is, the target subtitle area 62 irrelevant to the target subtitle text in the binary image 60 is removed. The positioning information of the above-mentioned target video can be obtained more accurately based on the binary image.

[0074] In some feasible implementations, the positioning information of the target video can be obtained based on the target subtitle area binary map, and the target subtitle area of ​​the target video can be determined by the positioning information. Specifically, the positioning information includes the coordinates of the starting point of the area, the length of the area, and the width of the area. In order to avoid the situation where the subtitle area in each image frame is obtained by the target subtitle area determined by the positioning information, and the characters in the intercepted subtitle area are incomplete, which affects the extraction of the target subtitle text, the preset area extension size is determined based on the area length and the area width in the positioning information, so as to determine the starting coordinates of the target area, the length of the area, and the width of the target area based on the starting coordinates of the area, the length of the area, and the width of the target area, and the target subtitle area determined by the starting coordinates of the target area, the length of the area, and the width of the target area can prevent the situation where incomplete characters appear when the subtitle area in each image frame is intercepted, thereby improving the accuracy of the target subtitle text extraction. For example, based on the binary map of the target subtitle area (such as Figure 5 The binary image 70 in the image can obtain the positioning information: the starting coordinates of the area (x, y), the length of the area w and the width of the area h. And the preset area expansion size is determined as follows: based on the subtitle area determined by the above positioning information, the left and right sides of the area are expanded by 0.1*w respectively, and the upper and lower sides of the area are expanded by 0.1*h respectively. That is, the target starting coordinates are determined to be (x-0.1*w, y-0.1*h), the length of the target area is 1.2*w, and the width of the target area is 1.2*h. By expanding the target subtitle area based on the preset area expansion size, it is avoided that when the subtitle area in each image frame is obtained based on the target subtitle area, there are incomplete characters in the subtitle area captured, so that the extraction of the target subtitle text is more complete and accurate.

[0075] S102, determining the subtitle area of ​​each image frame in the target video based on the target subtitle area, and obtaining the initial subtitle text of the target video based on the subtitle area of ​​each image frame.

[0076] In some feasible implementations, the business server can obtain multiple image frames of the target video based on a preset frame extraction frequency (for example, obtaining one image frame every 60 seconds) (until the target video ends), and determine the subtitle area of ​​the above multiple image frames through the above target subtitle area to extract the target sub-image of each image frame from the above each image frame, and the above target sub-image includes the initial subtitle clauses in each image frame, so that the initial subtitle text (which may include multiple initial subtitle clauses) can be obtained based on the obtained multiple target sub-images.

[0077] In some feasible implementations, since the target subtitle clauses in the target subtitle text are arranged in a certain order (i.e., the order in which the target subtitle clauses appear in the target video, or the order in which they appear in the target song), the target sub-images are sorted in the order in which the image frames to which the target sub-images belong appear in the target video, and the target subtitle text consistent with the lyrics of the target song can be obtained (for example, by recognizing and obtaining text through optical character recognition (OCR) technology) from the sorted target sub-images (which can generate images to be recognized). See Figure 6 , Figure 6 FIG. 1 is another schematic diagram of a video subtitle extraction method provided by an embodiment of the present application. Figure 6 As shown, Figure 6It includes some continuous image frames among multiple image frames acquired based on the target video (for example, one image frame is acquired every 60 seconds), including image frame 80, image frame 81 and image frame 82. Image frame 80 includes the target sub-image 801 of the image frame (the subtitle area of ​​the image frame is determined by the above-mentioned target subtitle area, for example, the starting coordinates of the area are (x1-0.1*w1, y1-0.1*h1), the area length is 1.2*w1, and the area width is 1.2*h1. Thus, the subtitle sub-image of the image frame is obtained based on the above-mentioned subtitle area), and the target sub-image 801 includes the initial subtitle clause of the image frame "I am used to being buried deep". Similarly, the target sub-image 811 in image frame 81 includes the initial subtitle clause of the image frame "I am used to being buried deep in the fog", and the target sub-image 821 in image frame 82 includes the initial subtitle clause of the image frame "I am used to being buried deep in the fog and competing with myself". After obtaining the above-mentioned target sub-images, the above-mentioned multiple target sub-images are sorted according to the order of appearance of the image frames to which each target sub-image belongs in the target video, and after sorting, they can be spliced ​​into the image to be recognized 83. The image to be recognized 83 contains the target sub-images in the above-mentioned image frames 80, image frames 81 and image frames 82 arranged in order. Character recognition is performed on the image to be recognized 83, and multiple initial subtitle clauses ("I am used to being buried deep", "I am used to being buried deep in the fog" and "I am used to being buried deep in the fog and competing with myself") can be obtained from the image to be recognized (according to the order of each subtitle clause). Similarly, for other image frames in the above-mentioned target video, the same method can be used to obtain the target sub-images of each image frame to obtain the target sub-images of all image frames, and the above-mentioned target sub-images are sorted according to the order of appearance of the image frames to which each target sub-image belongs in the target video to obtain the image to be recognized, and finally the initial subtitle text of the above-mentioned target video is obtained by OCR technology, which is simple to operate and has high efficiency in obtaining the initial subtitle text.

[0078] In some feasible implementations, extracting the initial subtitle text from the above-mentioned image to be recognized by using OCR technology may include text detection and text recognition. For details, please refer to Figure 7a , Figure 7a Figure 1 is a diagram of text detection in optical character recognition. Figure 7a As shown, in the process of optical character recognition, in order to detect the image based on the input (i.e. Figure 7aThe image to be detected 90 in the figure contains the text "test text") to obtain the text detection result. The text detection first passes through a feature extraction network composed of Compact Inception blocks, then passes through a feature enhancement network composed of several adaptive-sized recurrent neural networks (RNN), and finally passes through a Box generation and edge refinement network to obtain the text detection result (such as Figure 7a See the text detection box in Figure 91. Figure 7b , Figure 7b This is a diagram of text recognition in optical character recognition. Figure 7b As shown, based on Figure 7a The text detection results obtained by text detection will be Figure 7b The text to be recognized 92 in the text recognition model is input into the text recognition model. The above text recognition model adds horizontal asymmetric convolution and comprehensive multi-scale receptive field features, which enhances the network's support for multi-scale fonts. At the same time, it draws on the refined convolution method in Fine-grained Recognition to effectively enhance the image feature extraction in cases of similar characters and blurred characters. The text recognition model can be used to identify and obtain the characters "test text" in the text to be recognized 92. Through the above text detection and text recognition, the initial subtitle text of the above target video can be obtained from the image to be recognized obtained by sorting the above target sub-images in a certain order. The operation is simple, the recognition accuracy is high, and the initial subtitle text acquisition effect is good.

[0079] S103, based on the similarity between each adjacent initial subtitle sentence, obtaining a plurality of target subtitle sentences having an arrangement order from the plurality of initial subtitle sentences.

[0080] In some feasible implementations, the service server can detect the similarity of each initial subtitle clause in the initial subtitle text, and the similarity includes the editing distance and the length distance. The service server can extract the target subtitle clauses from the multiple initial subtitle clauses in the initial subtitle text based on the similarity to obtain multiple target subtitle clauses with an arrangement order, and the characters between two adjacent subtitle clauses in the multiple target subtitle clauses with an arrangement order are different, so that there are no repeated target subtitle clauses in the final target subtitle text. Specifically, the service server can detect the minimum number of target editing operations (the target number of editing operations refers to the editing operations in units of single characters) required to convert each initial subtitle clause in the initial subtitle text into its adjacent previous initial subtitle clause. The minimum number of target editing operations can be called the editing distance of each initial subtitle clause. At the same time, the number of characters different between each initial subtitle clause and its adjacent previous initial subtitle clause can be detected, and the length distance of each initial subtitle clause is obtained based on the number of characters different. The above-mentioned edit distance and length distance both reflect the similarity between each initial subtitle sentence and its adjacent previous initial subtitle sentence. The smaller the edit distance and the length distance, the greater the similarity. If the edit distance and the length distance of the initial subtitle sentence are both 0, the initial subtitle sentence is the same as its adjacent previous initial subtitle sentence. The above-mentioned multiple initial subtitle sentences can be extracted based on the above-mentioned edit distance and the length distance to obtain multiple target subtitle sentences with an arrangement order.

[0081] The following is a specific example to illustrate the process of obtaining the edit distance and length distance. Figure 8 , Figure 8 FIG. 1 is another schematic diagram of a video subtitle extraction method provided by an embodiment of the present application. Figure 8 As shown, Figure 8The initial subtitle text 100 therein is obtained by intercepting target sub-images from multiple image frames of the target video, sorting them to generate an image to be recognized, and performing character recognition on the image to be recognized. The above initial subtitle text 100 contains 14 initial subtitle clauses (where the 8th initial subtitle clause has no content, that is, the initial subtitle text in the target sub-image corresponding to this clause does not contain any characters). Taking the 2nd initial subtitle clause "Our company is used to being deeply buried in the fog by ourselves" as an example, the previous initial subtitle clause adjacent to this initial subtitle clause is "I am used to being deeply buried". The minimum number of target editing operations required to convert the 2nd initial subtitle clause into its previous adjacent initial subtitle clause is 4 times, including 3 deletion operations (deleting "fog", "in", and "by ourselves") and 1 replacement operation (replacing "company" with "I"), then the edit distance of the 2nd initial subtitle clause is 4. At the same time, it is detected that the number of characters in the 2nd initial subtitle clause is 8, and the number of characters in its previous adjacent initial subtitle clause is 5, with a difference of 3 characters, then the length distance of the 2nd initial subtitle clause is 3, thus obtaining the edit distance and length distance of the 2nd initial subtitle clause "Our company is used to being deeply buried in the fog by ourselves". From this, the edit distance and length distance of each clause from the 2nd initial subtitle clause to the 14th initial subtitle clause can be obtained (since the 8th initial subtitle clause has no content, this initial subtitle clause has no edit distance and length distance), and the specific process will not be elaborated. Refer to Figure 8 In Table 101 therein, the edit distances and length distances of the 2nd and 3rd initial subtitle clauses are 4 and 3 respectively, the edit distances and length distances of the 4th, 6th, 7th, 12th, and 14th initial subtitle clauses are all 0, the edit distances and length distances of the 5th and 13th initial subtitle clauses are 11 and 7 respectively, the edit distances and length distances of the 9th initial subtitle clause are 6 and 2 respectively, and the edit distances and length distances of the 11th initial subtitle clause are both 2.

[0082] In some feasible implementations, the service server can determine the candidate subtitle clauses from the above multiple initial subtitle clauses based on the edit distance and length distance of each initial subtitle clause, and determine all the determined candidate subtitle clauses as target subtitle clauses to obtain the target subtitle text. Specifically, first determine the first initial subtitle clause among the multiple initial subtitle clauses as the candidate subtitle clause. For any initial subtitle clause after the first initial subtitle clause, if the difference between the edit distance and the length distance of the initial subtitle clause is not less than the preset threshold, then determine the above any initial subtitle clause and its adjacent previous initial subtitle clause as candidate subtitle clauses. Otherwise, remove the adjacent previous initial subtitle clause of any initial subtitle clause, and determine any initial subtitle clause as a candidate subtitle clause. Through the determination rules of the above candidate subtitle clauses, there can be no repeated candidate subtitle clauses in the final candidate subtitle clauses, that is, the above candidate subtitle clauses can constitute the target subtitle text. The operation is simple, the target subtitle text extraction effect is good, and the applicability is strong.

[0083] Please see again Figure 8 , from Figure 8The example of extracting multiple target subtitle clauses from multiple initial subtitle clauses in the initial subtitle text 100 is used for explanation. First, the first initial subtitle clause "I am used to burying deeply" is determined as an alternative subtitle clause, and the above preset threshold is 3. For the second initial subtitle clause, the difference between the edit distance and the length distance of the second initial subtitle clause is 1, which is less than the preset threshold 3, then the first initial subtitle clause is removed, and the second initial subtitle clause is determined as an alternative subtitle clause. If the difference between the edit distance and the length distance of the third initial subtitle clause is less than the preset threshold, then the second initial subtitle clause is removed, and the third initial subtitle clause is determined as an alternative subtitle clause. If the difference between the edit distance and the length distance of the fourth initial subtitle clause is less than the preset threshold, then the third initial subtitle clause is removed, and the fourth initial subtitle clause is determined as an alternative subtitle clause. That is, after the above process, the current alternative subtitle clauses only have the fourth initial subtitle clause. If the difference between the edit distance and the length distance of the fifth initial subtitle clause is greater than the preset threshold, the fifth initial subtitle clause is newly determined as the candidate subtitle clause while retaining the fourth initial subtitle clause, that is, the current candidate subtitle clauses include the fourth initial subtitle clause and the fifth initial subtitle clause. Next, the candidate subtitle clauses are extracted from the 14 initial subtitle clauses in the initial subtitle text 100 by the same method, and the final candidate subtitle clauses are determined to include the fourth, seventh, twelfth and fourteenth initial subtitle texts in the above 14 initial subtitle clauses. Based on the above multiple candidate subtitle clauses, the target subtitle clauses 102 of the initial subtitle text 100 are obtained: "I am used to being buried deep in the fog and competing with myself", "Soul and head separated", "I like to be buried in the fog and not be raised" and "Wake up at dawn". The candidate subtitle clauses are determined from the above multiple initial subtitle clauses based on the edit distance and length distance of each initial subtitle clause to obtain multiple different target subtitle clauses.

[0084] In some feasible implementations, the above-mentioned multiple target subtitle clauses can be stored in a designated storage space in the business server, and the above-mentioned designated storage space can be a text list. Before extracting multiple target subtitle clauses based on the above-mentioned multiple initial subtitle clauses, the above-mentioned text list can be empty. In the process of extracting alternative subtitle clauses based on the edit distance and length distance of each of the above-mentioned initial subtitle clauses, the first initial subtitle clause can be first placed in the first storage unit of the above-mentioned text, that is, the first initial subtitle clause is determined as the first alternative subtitle clause. At the same time, for any initial subtitle clause after the first initial subtitle clause, if the difference between the edit distance and the length distance of the initial subtitle clause is not less than a preset threshold, then any initial subtitle clause is added to the latest free storage unit of the above-mentioned text list, which means that the previous initial subtitle clause adjacent to it is determined as an alternative subtitle clause. Otherwise, the adjacent previous initial subtitle clause is removed from the text list, and any of the initial subtitle clauses is added to the latest free storage unit of the text list, which means that the adjacent previous initial subtitle clause of any of the initial subtitle clauses is removed, and only any of the initial subtitle clauses is determined as an alternative subtitle clause. Through the above process, the initial subtitle clauses contained in the text list are the alternative subtitle clauses finally extracted, and the alternative subtitle clauses constitute multiple target subtitle clauses. In the process of extracting the target subtitle clauses, the initial subtitle clauses that are greatly different from or repeated in the lyrics of the corresponding target song are removed, which solves the problem of repeated extraction of subtitles. The operation is simple, the subtitle extraction effect is good, and the applicability is strong.

[0085] S104, obtaining a target subtitle text of a target video based on a plurality of target subtitle clauses having an arrangement order.

[0086] In some feasible implementations, the service server can obtain the target subtitle text of the target video based on the above multiple target subtitle clauses in the order of arrangement. For example, please refer to Figure 8 ,from Figure 8 A plurality of target subtitle clauses 102 with an arrangement order are extracted from a plurality of initial subtitle clauses in the initial subtitle text 100: "I am used to burying myself in the fog and competing with myself", "My soul and head are separated", "I like to bury myself in the fog and not be lifted up", and "Wake up at dawn". The above-mentioned multiple target subtitle clauses constitute the target subtitle text of the target video: "I am used to burying myself in the fog and competing with myself, my soul and head are separated, I like to bury myself in the fog and not be lifted up, and wake up at dawn." In the process of obtaining the above-mentioned target subtitle text, the initial subtitle clauses that are different from or repeated in the subtitles in the target video are removed, which solves the problem of repeated extraction of subtitles. The operation is simple, the subtitle extraction effect is good, and the applicability is strong.

[0087] In some feasible implementations, the business server can also obtain the name of the target song based on the target subtitle text, and associate the target video with the name of the target song to push the target video more effectively. For example, user A records the target video of the cover song B (the target video contains the lyrics of song B) through relevant equipment (mobile phone, camera, etc.), and user A uploads the target video to the business server through the karaoke application. The business server receives and obtains the corresponding target subtitle text based on the target video, and the target subtitle text is consistent with the lyrics of song B. The business server can obtain the name of song B based on the target subtitle text, and associate the cover video of song B uploaded by user A with song B. When other users use the karaoke application (such as opening the application page related to song B), they can view the cover video of song B uploaded by user A on the relevant application page, thereby pushing the target video more effectively and improving the user experience.

[0088] In an embodiment of the present application, the service server can obtain the target video (which can be sent by the terminal device through the application client (such as a karaoke application) installed thereon), and obtain multiple image frames in the target video at the same time, and the above multiple image frames can be multiple image frames carrying initial subtitle clauses extracted from the target video. The multiple image frames obtained are input into a pixel classification model (which can be a pixel classification model based on the VGG16 network architecture), and the probability that each pixel in each image frame belongs to the subtitle area of ​​each image frame can be obtained through the model. Then, the total number of times each pixel is judged as a subtitle area in each image frame is counted, and the target pixel is determined based on the total number of times each pixel is judged as a subtitle area. If the number of times a pixel is judged as a subtitle area in each image frame reaches a judgment threshold (for example, 2 times or more), then the pixel belongs to the target pixel in the target subtitle area binary map. Multiple target pixels can constitute one or more pixel sets, and each pixel set can be a target subtitle area. One or more target subtitle areas containing target pixel points greater than a preset number (for example, greater than 150 pixel points) can be extracted from the multiple target subtitle areas in the target subtitle area binary map. And the target subtitle area at the bottom is obtained from the one or more target subtitle areas, and the positioning information of the target video is extracted based on the target subtitle area. Thereby, some pixels that do not belong to the target subtitle text are effectively avoided from being included in the target subtitle area, and the accuracy of the target subtitle area is further improved. Here, the positioning information includes the starting point coordinates, the area length and the area width. In order to avoid the situation that the subtitle area in each image frame is obtained by the target subtitle area determined by the positioning information, the subtitle area intercepted has incomplete characters to affect the extraction of the target subtitle text, so the preset area extension size is determined based on the area length and the area width in the positioning information, so as to determine the starting coordinates of the target area, the length of the target area and the width of the target area based on the starting coordinates of the area, the length of the area, and the width of the preset area, and the target subtitle area determined by the starting coordinates of the target area, the length of the target area and the width of the target area can prevent the situation that incomplete characters appear when the subtitle area in each image frame is intercepted, thereby improving the accuracy of the target subtitle text extraction. The business server can also obtain multiple image frames of the target video based on a preset frame extraction frequency, and determine the subtitle area of ​​the above-mentioned multiple image frames through the above-mentioned target subtitle area to cut out the target sub-image of each image frame from the above-mentioned each image frame, and sort the above-mentioned multiple target sub-images according to the appearance order of the image frames to which each target sub-image belongs in the target video, and the initial subtitle text of the above-mentioned target video can be obtained from the sorted multiple target sub-images.The service server can detect the similarity of each initial subtitle clause in the above-mentioned initial subtitle text, and the similarity includes the edit distance and the length distance. The service server first determines the first initial subtitle clause among multiple initial subtitle clauses with an arrangement order as an alternative subtitle clause. For any initial subtitle clause after the first initial subtitle clause, if the difference between the edit distance and the length distance of the initial subtitle clause is not less than the preset threshold, then any of the above-mentioned initial subtitle clauses and its adjacent previous initial subtitle clause are determined as alternative subtitle clauses. Otherwise, the adjacent previous initial subtitle clause of any of the above-mentioned initial subtitle clauses is removed, and any of the above-mentioned initial subtitle clauses is determined as an alternative subtitle clause. Through the above-mentioned determination rules of alternative subtitle clauses, there can be no repeated alternative subtitle clauses in the final alternative subtitle clauses, that is, the above-mentioned alternative subtitle clauses can constitute the target subtitle text. The operation is simple, the subtitle extraction effect is good, and the applicability is strong.

[0089] The present application also provides a video subtitle extraction device, please refer to Fig. 9 , Fig. 9 : is a schematic diagram of the structure of a video subtitle extraction device provided in an embodiment of the present application. In the embodiment of the present application, the device can run the following modules:

[0090] An acquisition module 21 is used to acquire a target video;

[0091] A target subtitle region acquisition module 22, configured to determine a target subtitle region of the target video based on a plurality of image frames in the target video acquired by the acquisition module 21;

[0092] An initial subtitle text generating module 23 is used to determine the subtitle area of ​​each image frame in the target video based on the target subtitle area of ​​the target subtitle area generating module 22, and obtain the initial subtitle text of the target video based on the subtitle area of ​​each image frame, wherein the initial subtitle text includes a plurality of initial subtitle clauses in an arranged order;

[0093] A target subtitle text generating module 24 is used to obtain a plurality of target subtitle sentences having an arrangement order from the plurality of initial subtitle sentences based on the similarity between the adjacent initial subtitle sentences of the initial subtitle text generating module 23, wherein the characters between the adjacent target subtitle sentences are different from each other;

[0094] The target subtitle text generating module 24 is further configured to obtain the target subtitle text of the target video based on the plurality of target subtitle clauses having an arrangement order.

[0095] In some feasible implementations, the target subtitle area acquisition module 22 is further used for:

[0096] Acquire multiple image frames in the target video, and input the multiple image frames into a pixel classification model;

[0097] Outputting the probability that each pixel in each image frame belongs to the subtitle area based on the pixel classification model, and obtaining the subtitle positioning information of the target video based on the probability that each pixel in each image frame belongs to the subtitle area;

[0098] A target subtitle area of ​​the target video is determined based on the subtitle positioning information.

[0099] In some feasible implementations, the target subtitle area acquisition module 22 is further used for:

[0100] Based on the probability that each pixel point in each of the above image frames belongs to the subtitle area, obtaining the number of times each of the above pixel points is determined to be the subtitle area in each of the above image frames;

[0101] A target pixel is determined from the pixels based on the number of times the pixel is determined as a subtitle area in the image frames, and subtitle positioning information of the target video is acquired based on the target pixel.

[0102] In some feasible implementations, the target subtitle area acquisition module 22 is further used for:

[0103] Determine from the above-mentioned pixels the pixel points whose number of times of being determined as the subtitle area in the above-mentioned image frames is greater than or equal to the number threshold as target pixel points to obtain a plurality of target pixel points, wherein the plurality of target pixel points constitute one or more pixel point sets;

[0104] Extracting one or more target pixel point sets including a number of the target pixel points greater than a preset number based on the one or more pixel point sets;

[0105] A target pixel point set in a designated display area is selected from the one or more target pixel point sets, and subtitle positioning information of the target video is obtained based on the selected target pixel point set.

[0106] In some feasible implementations, the target subtitle area acquisition module 22 is further used for:

[0107] Determine the preset area extension size based on the area length and area width included in the subtitle positioning information, and determine the target area starting coordinates, target area length and target area width based on the area starting point coordinates, area length, area width and the preset area extension size;

[0108] An area consisting of the target area starting coordinates, the target area length and the target area width is determined as the target subtitle area.

[0109] In some feasible implementations, the initial subtitle text generation module 23 is further used to:

[0110] Acquire multiple image frames of the target video based on a preset frame extraction frequency, and determine the subtitle area of ​​each image frame from the multiple image frames based on the target subtitle area;

[0111] Based on the subtitle area of ​​each image frame, target sub-images of each image frame are cut out from each image frame to obtain a plurality of target sub-images, and the plurality of target sub-images are sorted according to the order of appearance of the image frames to which the target sub-images belong in the target video;

[0112] An image to be recognized is generated based on the sorted multiple target sub-images, and character recognition is performed on the image to be recognized to obtain an initial subtitle text of the target video.

[0113] In some feasible implementations, the target subtitle text generation module 24 is further used to:

[0114] Detecting the minimum target number of editing operations required to transform any initial subtitle sentence into its adjacent previous initial subtitle sentence, and obtaining the editing distance of any initial subtitle sentence based on the target number of editing operations;

[0115] Detecting the number of characters that differ between any initial subtitle sentence and its adjacent previous initial subtitle sentence, and obtaining the length distance of any initial subtitle sentence based on the number of characters that differ;

[0116] A target subtitle sentence is determined from the plurality of initial subtitle sentences based on the edit distance and the length distance of each of the initial subtitle sentences.

[0117] In some feasible implementations, the target subtitle text generation module 24 is further used to:

[0118] Determine the first initial subtitle clause among the multiple initial subtitle clauses as an alternative subtitle clause;

[0119] If the difference between the edit distance and the length distance of any initial subtitle clause after the first initial subtitle clause is not less than a preset threshold, the initial subtitle clause and the previous initial subtitle clause adjacent to the initial subtitle clause are both determined as the candidate subtitle clauses; otherwise, the previous initial subtitle clause adjacent to the initial subtitle clause is removed and the initial subtitle clause is determined as the candidate subtitle clause to determine the candidate subtitle clause from the multiple initial subtitle clauses; wherein the previous initial subtitle clause adjacent to the second initial subtitle clause in the multiple initial subtitle clauses is the first initial subtitle clause;

[0120] All the determined candidate subtitle sentences are determined as target subtitle sentences.

[0121] In an embodiment of the present application, the video subtitle extraction module can obtain a target subtitle area from the target video to obtain the position distribution of the target subtitle text in each image frame of the video, and obtain the initial subtitle text (including multiple initial subtitle clauses with an arrangement order) from each image frame of the target video through the target subtitle area. Based on the similarity between each adjacent clause in each initial subtitle clause, the target subtitle clause extraction is performed on the above multiple initial subtitle clauses with an arrangement order to obtain multiple target subtitle clauses with an arrangement order. In the above target subtitle clause extraction process, the initial subtitle clauses that are incomplete or repeated with the subtitles in the target video can be removed, so that the target subtitle text can be obtained through multiple target subtitle clauses. The problem of repeated subtitle extraction can be solved, the operation is simple, the subtitle extraction effect is good, and the applicability is strong.

[0122] In the embodiment of the present application, the above Fig. 9 Each module in the device shown can be separately or all combined into one or several other modules to constitute, or one (some) of the modules can also be split into multiple smaller modules in function to constitute, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above modules are divided based on logical functions. In practical applications, the function of a module can also be implemented by multiple modules, or the function of multiple modules can be implemented by one module. In other feasible implementations of the present application, the above device can also include other modules. In practical applications, these functions can also be implemented with the assistance of other modules, and can be implemented by the collaboration of multiple modules, which is not limited here.

[0123] See also Fig.10 , Fig.10 Schematic diagram of the structure of the computer device provided in the embodiment of the present application. Fig.10 As shown, the computer device 1000 may be the above Figure 2The business server in the corresponding embodiment. The computer device 1000 may include: a processor 1001, a network interface 1004 and a memory 1005. In addition, the computer device 1000 may also include: a user interface 1003, and at least one communication bus 1002. The communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), a keyboard (Keyboard), and the user interface 1003 may optionally include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1004 may be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 1005 may optionally be at least one storage device located away from the aforementioned processor 1001. Fig. 9 As shown, the memory 1005 as a computer-readable storage medium may include an operating system, a network communication module, a user interface module, and a device control application program.

[0124] The network interface 1004 in the computer device 1000 can also be connected to the above Figure 2 The terminal 200a in the corresponding embodiment is connected to the network, and the optional user interface 1003 may also include a display screen (Display) and a keyboard (Keyboard). Fig. 9 In the computer device 1000 shown in the figure, the network interface 1004 can provide a network communication function; the user interface 1003 is mainly used to provide an input interface for the user (or developer); and the processor 1001 can be used to call the device control application stored in the memory 1005 to implement the above Figure 2 The video subtitle extraction method in the corresponding embodiment.

[0125] It should be understood that the computer device 1000 described in the embodiment of the present application can execute the above Figure 2 The description of the video subtitle extraction method in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated here either.

[0126] In addition, it should be pointed out that the embodiment of the present application further provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program executed by the aforementioned statement expansion device, and the computer program includes program instructions. When the processor executes the program instructions, the aforementioned statement expansion device can be executed. Figure 2The description of the video subtitle extraction method in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated here. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application.

[0127] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the above-mentioned program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the above-mentioned storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0128] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A video subtitle extraction method, characterized in that: The method comprises: Acquire a target video, and determine a target subtitle area of ​​the target video based on a plurality of image frames in the target video; Determining a subtitle region of each image frame in the target video based on the target subtitle region, and obtaining an initial subtitle text of the target video based on the subtitle region of each image frame, wherein the initial subtitle text includes a plurality of initial subtitle clauses in an arrangement order; Detecting a minimum target number of editing operations required to transform any initial subtitle sentence into its adjacent previous initial subtitle sentence, and obtaining an editing distance of any initial subtitle sentence based on the target number of editing operations; Detecting the number of characters that differ between any initial subtitle sentence and its adjacent previous initial subtitle sentence, and obtaining the length distance of any initial subtitle sentence based on the number of characters that differ; Determine the first initial subtitle clause among the plurality of initial subtitle clauses as the candidate subtitle clause; If the difference between the edit distance and the length distance of any initial subtitle clause after the first initial subtitle clause is not less than a preset threshold, the initial subtitle clause and its adjacent previous initial subtitle clause are both determined as the candidate subtitle clauses; otherwise, the adjacent previous initial subtitle clause of any initial subtitle clause is removed and the initial subtitle clause is determined as the candidate subtitle clause; wherein the adjacent previous initial subtitle clause of the second initial subtitle clause among the multiple initial subtitle clauses is the first initial subtitle clause; Determine all the determined candidate subtitle clauses as target subtitle clauses, wherein characters between adjacent target subtitle clauses are different from each other; A target subtitle text of the target video is obtained based on the multiple target subtitle clauses having an arrangement order.

2. The method according to claim 1, characterized in that The step of determining a target subtitle area of ​​the target video based on a plurality of image frames in the target video comprises: Acquire multiple image frames in the target video, and input the multiple image frames into a pixel classification model to obtain a probability that each pixel in each image frame output by the pixel classification model belongs to a subtitle area; Based on the probability that each pixel point in each image frame belongs to the subtitle area, obtaining subtitle positioning information of the target video; A target subtitle area of ​​the target video is determined based on the subtitle positioning information.

3. The method according to claim 2, characterized in that The acquiring the subtitle positioning information of the target video based on the probability that each pixel point in each image frame belongs to the subtitle area includes: Based on the probability that each pixel point in each image frame belongs to the subtitle area, obtaining the number of times each pixel point is determined to be the subtitle area in each image frame; A target pixel is determined from the pixels based on the number of times the pixel is determined as a subtitle area in the image frames, and subtitle positioning information of the target video is acquired based on the target pixel.

4. The method according to claim 3, characterized in that: The step of determining a target pixel from each pixel based on the number of times each pixel is determined as a subtitle area in each image frame, and acquiring subtitle positioning information of the target video based on the target pixel, comprises: Determine from the pixels the pixel points whose number of times of being determined as the subtitle area in the image frames is greater than or equal to the number threshold as target pixel points to obtain a plurality of target pixel points, wherein the plurality of target pixel points constitute one or more pixel point sets; Selecting, from the one or more pixel point sets, one or more target pixel point sets including a number of target pixel points greater than a preset number; A target pixel point set in a designated display area is selected from the one or more target pixel point sets, and subtitle positioning information of the target video is acquired based on the selected target pixel point set.

5. The method according to claim 4, characterized in that The subtitle positioning information includes the coordinates of the starting point of the region, the length of the region, and the width of the region; and determining the target subtitle region of the target video based on the subtitle positioning information includes: Determine a preset area extension size based on the area length and area width included in the subtitle positioning information, and determine the target area starting coordinates, target area length and target area width based on the area starting point coordinates, area length, area width and the preset area extension size; An area consisting of the target area start coordinates, the target area length and the target area width is determined as the target subtitle area.

6. The method according to claim 1, characterized in that The step of determining the subtitle area of ​​each image frame in the target video based on the target subtitle area, and obtaining the initial subtitle text of the target video based on the subtitle area of ​​each image frame includes: Acquire multiple image frames of the target video based on a preset frame extraction frequency, and determine a subtitle area of ​​each image frame from the multiple image frames based on the target subtitle area; Based on the subtitle area of ​​each image frame, a target sub-image of each image frame is cut out from each image frame to obtain a plurality of target sub-images, and the plurality of target sub-images are sorted according to the appearance order of the image frames to which the target sub-images belong in the target video; An image to be recognized is generated based on the sorted multiple target sub-images, and character recognition is performed on the image to be recognized to obtain an initial subtitle text of the target video.

7. A computer device, characterized in that: include: Processor, memory, and network interface; The processor is connected to the memory and the network interface, wherein the network interface is used to provide a data communication function, the memory is used to store program code, and the processor is used to call the program code to execute the method described in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the method according to any one of claims 1 to 6.

Images