Text recognition method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a text recognition method and device, electronic equipment and a storage medium, and the method comprises the steps: recognizing a starting time point and an ending time point of a song for any song in a to-be-detected video, and enabling the to-be-detected video to comprise a textbox obtained through the recognition of a text line; extracting a plurality of video frames within a first preset duration range after the starting time point; respectively identifying the positions of the textboxes in the plurality of video frames, and overlapping the textboxes of the plurality of video frames according to the positions of the textboxes respectively corresponding to the video frames to obtain a lyric display area of the song; carrying out intersection and comparison calculation on the lyric display area and a textbox in a video frame between the starting time point and the ending time point of the song to obtain a lyric textbox containing lyrics; and obtaining lyrics of the song from the lyric textbox. According to the embodiment of the invention, the lyric recognition accuracy is improved.

Description

Technical Field

[0001] This invention relates to the field of Internet technology, and in particular to a text recognition method, apparatus, electronic device, and storage medium. Background Technology

[0002] In existing singing competition shows, a common setup is to display dialogue in a fixed area of ​​a video frame and lyrics in another fixed area. Therefore, when recognizing lyrics and dialogue, the process typically involves first identifying the fixed area displaying the dialogue, and then using that area to identify the dialogue. Next, in the remaining video frames that are determined not to contain dialogue, the fixed area displaying the lyrics is identified and designated as the lyrics region frame. At this point, the text lines in the remaining video frames whose intersection with the lyrics region frame falls within a certain range can be identified as lyrics.

[0003] However, with the diversification of video formats, singing competition shows, in pursuit of better program effects, create different lyric display styles for different songs, allowing lyrics to be displayed in different areas of the video frame. Therefore, using a fixed area to represent the lyric display position for all songs may lead to missed or misidentified lyrics. Summary of the Invention

[0004] The purpose of this invention is to provide a text recognition method, apparatus, electronic device, and storage medium to solve the problem that existing lyric recognition methods are relatively fixed and cannot be adaptively adjusted according to different lyric display styles, leading to missed or misidentified lyrics. The specific technical solution is as follows:

[0005] In a first aspect of this invention, a text recognition method is provided, which may include:

[0006] For any song in the video to be detected, identify the start and end times of the song, and the video to be detected contains text boxes obtained by identifying text lines;

[0007] Extract several video frames within a first preset duration range after the starting time point;

[0008] The positions of text boxes in several video frames are identified respectively, and the text boxes of several video frames are superimposed according to the positions of the text boxes corresponding to each video frame to obtain the lyrics display area of ​​the song.

[0009] The intersection-union ratio (IUU) of the lyrics display area and the text boxes in the video frames between the start and end times of the song is calculated to obtain the lyrics text box containing the lyrics.

[0010] Extract the lyrics of the song from the lyrics text box.

[0011] Optionally, identifying the start time point of any song in the video to be detected includes:

[0012] For any song in the video to be detected, identify the display time point of each lyric video frame displayed by the song, wherein the lyric video frame is a video frame containing information about the song title, singer, lyricist and composer.

[0013] The display time of the last frame of the lyrics video frame of the song is determined as the start time of the song.

[0014] Optionally, identifying the end time point of any song in the video to be detected further includes:

[0015] Identify the display time of the first frame of the lyrics video for the next song;

[0016] The display time interval between the display time of the last frame of the lyrics video frame of the song and the display time of the first frame of the lyrics video frame of the next song is obtained.

[0017] The end time of the song is identified by the display time interval.

[0018] Optionally, identifying the end time of the song through the display time interval includes:

[0019] If the display time interval is less than or equal to the first preset duration range, then the display time of the first frame of the lyrics video frame of the next song is determined as the end time of the song.

[0020] If the display time interval is greater than the first preset duration range, the start time of the song is delayed by the second preset duration and determined as the end time of the song.

[0021] Optionally, the step of superimposing the text boxes of several video frames according to the positions of the text boxes corresponding to each video frame to obtain the lyrics display area of ​​the song includes:

[0022] Based on the position of the text box corresponding to each video frame, extract the vertex coordinates of the text box corresponding to each video frame;

[0023] By using the vertex coordinates of the text boxes corresponding to each video frame, the text boxes of several video frames are overlaid on the same preset blank image.

[0024] The frequency of coverage of each region in the preset blank image by text boxes of several video frames is counted.

[0025] A heatmap of text boxes in several video frames is generated based on the coverage frequency, wherein the area with a higher coverage frequency of text boxes in the preset blank map corresponds to a higher heat value in the heatmap.

[0026] The area in the heat map where the heat value is greater than the target preset value is determined as the lyrics display area of ​​the song.

[0027] Optionally, after calculating the intersection-union ratio (IoU) of the lyrics display area with the text boxes in the video frames between the start and end times of the song to obtain the lyrics text boxes, the process includes:

[0028] Filter out the first video frame containing the lyrics text box from the video frames between the start and end times of the song;

[0029] Add a preset marker to the first video frame;

[0030] Filter out the second video frame that has not been marked with a preset tag from the video to be detected;

[0031] Extract the dialogue from the text box of the second video frame.

[0032] Optionally, extracting the dialogue from the text box of the second video frame includes:

[0033] Identify the position of the text box in the second video frame;

[0034] The text boxes of the second video frame are superimposed according to their positions to obtain the dialogue display area of ​​the video to be detected.

[0035] The text box of the second video frame is compared with the dialogue display area by the intersection-union calculation to obtain the dialogue text box containing the dialogue.

[0036] Extract the dialogue from the video to be detected from the dialogue text box.

[0037] In a second aspect of the present invention, a text recognition device is provided, the device comprising:

[0038] The first acquisition module is used to identify the start and end time points of any song in the video to be detected, wherein the video to be detected contains text boxes obtained by identifying text lines.

[0039] The second acquisition module is used to extract a number of video frames within a first preset time range after the starting time point;

[0040] The third acquisition module is used to identify the positions of text boxes in several video frames respectively, and to superimpose the text boxes of several video frames according to the positions of the text boxes corresponding to each video frame to obtain the lyrics display area of ​​the song.

[0041] The intersection-union ratio (IUU) calculation module is used to perform IUU calculation on the text boxes in the video frames between the lyrics display area and the start and end times of the song to obtain lyrics text boxes containing lyrics.

[0042] The fourth acquisition module is used to extract the lyrics of the song from the lyrics text box.

[0043] A third aspect of the present invention also provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus.

[0044] Memory, used to store computer programs;

[0045] The processor, when executing a program stored in memory, implements any of the text recognition methods described in the first aspect above.

[0046] In a fourth aspect of the present invention, a computer-readable storage medium is also provided, wherein instructions are stored therein, which, when executed on a computer, cause the computer to implement any of the text recognition methods of the first aspect described above.

[0047] This invention provides a text recognition method that identifies the positions of text boxes in several video frames and then superimposes these text boxes to obtain the lyrics display area of ​​a song. Since a higher number of superimposed text boxes indicates a higher frequency of text lines in that area, and the text lines in the video frames near the song's start point are typically lyrics, which are displayed in the lyrics display area, this superposition method eliminates interference from random factors and accurately captures the lyrics display area of ​​the song, providing a reliable basis for tracking lyrics in subsequent frames. Furthermore, because different songs can choose to display lyrics in different areas of the video frames, this invention... For example, this method extracts several video frames near the start of each song, and then identifies the lyrics display area based on these frames. Therefore, the identified lyrics display area adapts to different lyric display styles, avoiding the problems of missed or incorrect lyric identification caused by setting a fixed lyric recognition area. Finally, the intersection-union (IoU) calculation is performed between the lyrics display area and the text boxes in the video frames between the song's start and end times to obtain the lyrics text boxes containing the lyrics. The lyrics are then extracted from these text boxes, further avoiding misidentification of lyrics in the video frames between the song's start and end times, thus improving the accuracy of lyric recognition. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0049] Figure 1 A flowchart illustrating the steps of a text recognition method provided in an embodiment of the present invention;

[0050] Figure 2 This is a flowchart of step 101 in a text recognition method provided in an embodiment of the present invention;

[0051] Figure 3 A flowchart illustrating the steps of another text recognition method provided in an embodiment of the present invention;

[0052] Figure 4 A structural block diagram of an electronic device provided in an embodiment of the present invention;

[0053] Figure 5 This is a structural block diagram of another electronic device provided in an embodiment of the present invention. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will now be described with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application can be thoroughly understood and its scope can be fully conveyed to those skilled in the art.

[0055] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0056] In various embodiments of the present invention, it should be understood that the sequence number of each process described below does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0057] The text recognition method, apparatus, electronic device, system, and storage medium provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0058] In existing singing competition shows, a common setup is to display dialogue in a fixed area of ​​a video frame and lyrics in another fixed area, with the proportion of video frames displaying dialogue being greater than those displaying lyrics. Therefore, current lyric recognition algorithms typically begin with a fully automated process to detect and identify text lines in each video frame, obtaining corresponding detection boxes (also called text boxes). These detected text boxes are then fitted to determine the region with the highest frequency of occurrence, which is designated as the dialogue region box, and the dialogue is recognized from within this region. This process is repeated for the remaining video frames, identifying the region with the highest frequency of occurrence of text boxes as the lyric region box, and recognizing the lyrics from within this region. However, with the increasing variety of lyric display styles, the location of lyrics is no longer fixed. In this context, using a fixed region to recognize lyrics may result in the identification of non-lyric content or the omission of lyrics that should be recognized.

[0059] Based on the above problems, this invention proposes a new text recognition method, see [link to relevant documentation]. Figure 1This is a flowchart of a text recognition method provided in an embodiment of the present invention. The method may include:

[0060] Step 101: For any song in the video to be detected, identify the start and end times of the song. The video to be detected contains text boxes obtained from the identified text lines.

[0061] In this embodiment of the invention, the video to be detected can be obtained from videos of singing variety shows. The video to be detected may contain video frames from multiple songs, but must contain at least one song's video frames. The text lines included in the video to be detected may include lyrics, dialogue, inserted advertising text, and garbled strings after noise interference, etc.

[0062] Because different songs may display lyrics in different styles, the lyrics display area of ​​some songs may overlap with or intersect with the dialogue display area. Therefore, if the dialogue is identified first, the dialogue text box obtained based on the identified dialogue display area may contain part of the song's lyrics, affecting the accuracy of dialogue recognition. However, if the lyrics are identified first, even if there are dialogue display areas that overlap with or intersect with the lyrics display area, the influence of the dialogue text box can be avoided by identifying the song's start and end time points and selecting video frames within these two time points for lyrics recognition. Therefore, this embodiment of the invention does not identify dialogue first, but rather identifies lyrics first. Moreover, the recognition of lyrics is not based on a fixed area, but takes into account the different lyric characteristics of each song, identifying the corresponding lyrics display area once for each song. Therefore, this embodiment of the invention identifies lyrics based on each song. Furthermore, since the video to be detected contains not only songs but also dialogue, and to identify lyrics, it is first necessary to determine the range of lyrics display. Lyrics are the content of the song being sung. Therefore, this embodiment of the invention needs to first determine the song's start and end time points, and then identify lyrics within this range, which can minimize the number of video frames to be identified and save computing resources.

[0063] Song programs typically feature lyric displays. During a song performance, the lyric display is shown first, followed by the intro music, and then the actual performance begins. Therefore, the appearance of the lyric display video frame can be considered the start of the song. This invention utilizes lyric display video frames to determine the song's start time. The lyric display contains information such as the song title, singer, lyricist, and composer. For any song in the video to be detected, the lyric display algorithm can determine the location and display time of the lyric display video frame on the playback interface. Normally, to avoid the lyric display flashing by too quickly and being visually unreadable, the lyric display video frame is displayed in multiple consecutive frames. To avoid visual confusion, these lyric display video frames are positioned in fixed areas within the video frames and use a fixed text style when displayed in multiple consecutive frames. Because of these settings, when identifying the lyric display video frames of a particular song, multiple consecutive lyric display video frames will be detected. If the content is identical across multiple consecutive frames, it indicates that the song's lyrics display consists of only one page. If the content changes across multiple consecutive frames, it indicates that the song's lyrics display has multiple pages, meaning there is page turning involved. Regardless of the situation, the display time of the last lyric video frame is closer to the actual start time of the song. Therefore, this display time is determined as the song's start time, thus improving the accuracy of the start time.

[0064] Step 102: Extract several video frames within the first preset duration range after the starting time point.

[0065] This invention takes into account that the performance time of songs is usually over 2 minutes, and no lyrics are spoken during the performance. Therefore, the content displayed in the text lines of video frames within a certain time range (less than or equal to 2 minutes) after the start of the song can be assumed to be lyrics. Based on this rule, this invention sets a first preset duration, and the content in the text lines of video frames within this duration is assumed to be lyrics. Therefore, several video frames within this duration are extracted to facilitate the subsequent fitting of the location of the lyrics area in the current song based on the positions of the text boxes appearing in these video frames.

[0066] Step 103: Identify the positions of text boxes in several video frames, and based on the positions of the text boxes in each video frame, overlay the text boxes of several video frames to obtain the lyrics display area of ​​the song.

[0067] In determining the lyrics display area in this embodiment of the invention, since the extracted video frames are those within the first preset duration range after the song's start time, which is during the song's performance, no dialogue is expected. Therefore, the text boxes at this time are obtained by recognizing the lyrics in the video frames. However, to avoid the influence of other text elements (such as advertising text) that may occasionally appear in the background of the video frames, it is necessary to overlay all the text boxes in several video frames. When overlaying the text boxes, the vertex coordinates of the text boxes in each of the several video frames are first extracted, and all text boxes are mapped onto a preset blank image using these vertex coordinates. The size of the preset blank image is consistent with the size of the several video frames. Then, the coverage frequency of each area in the preset blank image covered by text boxes is counted to generate an overlay statistical image. This overlay statistical image is then converted into a heatmap. A higher heatmap value indicates a higher coverage frequency of text boxes in that area, and a higher frequency of the corresponding text. The heatmap value can be represented by color intensity; for example, areas with low heatmap values ​​are represented by cool colors (such as blue), and areas with high heatmap values ​​are represented by warm colors (such as red). Then, a target preset value is set to filter out areas with heat values ​​higher than a threshold, which are then identified as lyrics display areas. The specific steps include:

[0068] Based on the position of the text box corresponding to each video frame, extract the vertex coordinates of the text box corresponding to each video frame;

[0069] By using the vertex coordinates of the text boxes corresponding to each video frame, the text boxes of several video frames are overlaid on the same preset blank image.

[0070] The frequency of coverage of each region in the preset blank image by text boxes of several video frames is counted.

[0071] A heatmap of text boxes in video frames is generated based on the coverage frequency, wherein the area with a higher coverage frequency of text boxes in the preset blank map corresponds to a higher heat value in the heatmap.

[0072] The area in the heat map where the heat value is greater than the target preset value is determined as the lyrics display area of ​​the song.

[0073] The lyrics display area obtained through the above method can effectively filter out noisy areas and focus on the core area of ​​the lyrics text.

[0074] For example, several video frames contain the following text boxes: Text box 1: top left (10,10), bottom right (30,20); Text box 2: top left (15,15), bottom right (35,25); Text box 3: top left (50,50), bottom right (70,60); Text box 4: top left (55,55), bottom right (75,65); Text box 5: top left (10,10), bottom right (30,20). The result of overlaying the text boxes is as follows: the area (10,10) to (30,20) is covered by text boxes 1, 2 and 5 with a coverage frequency of 3; the area (15,15) to (35,25) is covered by text box 2 with an overlay frequency of 1; the area (50,50) to (70,60) is covered by text boxes 3 and 4 with a coverage frequency of 2; and the area (55,55) to (75,65) is covered by text box 4 with a coverage frequency of 1. Based on the overlay frequency, the heat value of the area (10,10) to (30,20) in the heat map generated is 3 (high), the heat value of the area (50,50) to (70,60) is 2 (medium), and the heat value of other areas is 1 or 0 (low). The target preset value is set to 2, so the area with a heat value greater than 2 is (10,10) to (30,20), which is determined to be the lyrics display area.

[0075] It should be understood that because the display style of lyrics is diverse, a single lyric may be displayed with a line break, or multiple lyric lines may be displayed at once. Thus, there may be multiple lyric display areas for a song. Correspondingly, there may be multiple areas in the heat map where the heat value is greater than the target preset value. In this embodiment of the invention, these multiple areas are determined to be lyric display areas.

[0076] Step 104: Perform an intersection-union calculation on the text boxes in the video frames between the lyrics display area and the start and end times of the song to obtain the lyrics text boxes containing the lyrics.

[0077] In this embodiment of the invention, after determining the lyrics display area, the approximate range of the lyrics display area is first determined by the start and end times of the song. Then, to filter out noise areas, the intersection-union ratio (IUGR) of the text boxes in the video frames between the lyrics display area and the start and end times of the song is calculated. Only when the IUGR reaches a certain threshold is the text box considered to contain lyrics. This threshold can be set to 0.5, 0.4, or 0.8, etc., and is not specifically limited in this invention. The IUGR is an indicator that measures the degree of overlap between two rectangular regions: IUGR = intersection area of ​​two rectangular regions ÷ union area of ​​two rectangular regions.

[0078] For example, suppose the lyrics display area is a rectangular area with coordinates (x1, y1, x2, y2) = (10, 10, 30, 20). Suppose there are three video frames containing text boxes with coordinates A (12, 12, 28, 18), B (40, 40, 60, 50), and C (15, 15, 25, 19). The cross-over ratio (COP) threshold is set to 0.4. Therefore, when calculating the intersection-union ratio (IUGR) of text box A and the lyrics display area, first determine that the intersection area of ​​text box A and the lyrics display area is (12, 12, 28, 18). The area of ​​their intersection is: (28-12)*(18-12)=16*6=96, and the area of ​​their union is: (30-10)*(20-10)+(28-12)*(18-12)-96=200+96-96=200. The IUGR is: 96 / 200=0.48. Similarly, when calculating the IUGR of text box B and the lyrics display area, first determine that the intersection area of ​​text box B and the lyrics display area is non-overlapping, so the area of ​​their intersection is 0. The area of ​​their union is: (30-10)*(20-10)+(60-40)*(50-40)=200+200=400. The IUGR is: 0 / 400=0. When calculating the intersection-union ratio (IUGR) of text box C and the lyrics display area, first determine the intersection area of ​​text box C and the lyrics display area as (15, 15, 25, 19). The area of ​​their intersection is: (25-15)*(19-15)=10*4=40, and the area of ​​their union is: (30-10)*(20-10)+(25-15)*(19-15)-40=200+40-40=200. The IUGR is: 40 / 200=0.2. Since the IUGR of text box A is 0.48>0.4, text box A is determined to be a lyrics text box. Since the IUGR of text box B is 0<0.4, and the IUGR of text box C is 0.2<0.4, text boxes B and C are determined to be non-lyric text boxes.

[0079] Step 105: Obtain the lyrics of the song from the lyrics text box.

[0080] In this embodiment of the invention, after determining the lyrics text box, the text lines within the lyrics text box are identified as lyrics. It should be noted that when two songs are far apart in time, the display time interval is greater than the first preset duration range. The end time of the song is determined based on the set second preset duration. It is highly likely that the text lines in some video frames at the end are no longer lyrics, but rather other text that occasionally falls into the lyrics area. In this case, further filtering and judgment are required. During the judgment, the process proceeds in reverse order of time, from back to front. First, the first video frame with a preliminarily determined lyrics text box is found. Then, the previous frame is judged repeatedly until no lyrics text box is determined in the previous frame. The number of consecutive video frames with lyrics text boxes is counted. If the number of consecutive frames exceeds a certain threshold, such as 3 frames, it is considered that all the above video frames contain lyrics, and the judgment stops. If the number of consecutive frames does not exceed a certain threshold, it is considered that the text lines appearing in these video frames may be very short lyrics or other random text that is mistakenly judged as lyrics. In this situation, the process needs to continue moving forward, checking the next video frame containing the preliminarily determined lyrics text box and obtaining its time point. The time interval between the time point of the video frame at the end of the previous check and this time point is calculated. If the two time intervals are relatively long (e.g., exceeding 10 seconds), the last video frame (i.e., all video frames from the previous check) is deleted, and the check of the last detection result is restarted to determine if it contains lyrics. If the time interval is relatively short, it is assumed that all video frames from the previous check contain lyrics, and the last lyrics detection result is retained. The check then stops.

[0081] This invention provides a text recognition method. For any song in a video to be detected, the method identifies the start and end times of the song. The video contains text boxes obtained from identified text lines. Then, it extracts several video frames within a first preset duration after the start time. These video frames near the song's start point contain the song's lyrics display characteristics. Since the lyrics display area of ​​a song is fixed, the positions of most text boxes in the video frames containing the song's lyrics display characteristics indicate the lyrics display area. Therefore, by identifying the positions of the text boxes in the video frames and superimposing them according to their respective positions, the song's lyrics display area is obtained. The more times the text boxes are superimposed, the more times the text lines appear in that area. Since the text lines in the video frames near the song's start point are usually lyrics, the lyrics will be displayed in the lyrics display area. By superimposing the lyrics, interference from accidental factors can be eliminated, accurately capturing the lyrics display area of ​​the song and providing a reliable basis for lyrics tracking in subsequent frames. Furthermore, since different songs can choose to display lyrics in different areas of the video frames, this embodiment of the invention extracts several video frames near the start point of each song, and then identifies the lyrics display area based on these video frames. Therefore, the identified lyrics display area will adaptively adjust according to different lyric display styles of the song, avoiding the problems of missed or misidentified lyrics caused by setting a fixed lyrics recognition area. Finally, the intersection-union (OCU) calculation is performed between the lyrics display area and the text boxes in the video frames between the start and end points of the song to obtain the lyrics text boxes containing the lyrics. The lyrics of the song are then obtained from the lyrics text boxes, further avoiding misidentification of lyrics in the video frames between the start and end points of the song, thus improving the accuracy of lyrics recognition.

[0082] See Figure 2 This is a flowchart of step 101 in a text recognition method provided in an embodiment of the present invention, specifically including:

[0083] In this embodiment of the invention, the end time of a song is determined based on the display time of the lyrics video frame of the next song. When the lyrics video frame of the next song begins to display, it indicates that the performance of the current song has ended. Therefore, the end time of the current song will be before the display of the lyrics video frame of the next song. When the time interval between two songs is close, it can be assumed that there are no interludes between the two songs. In this case, the end time of the current song can be determined directly based on the display time of the first frame of the lyrics video frame of the next song. If the time interval between two songs is far, a hyperparameter (second preset duration) is set, and the end time of the current song is determined based on the hyperparameter. Therefore, step 101 of determining the start time and end time of the current song includes:

[0084] Step 1011: For any song in the video to be detected, identify the display time point of each lyric video frame displayed by the song. The lyric video frame is a video frame that contains information about the song title, singer, lyricist and composer.

[0085] Step 1012: Determine the display time of the last frame of the lyrics video frame as the start time of the song.

[0086] Step 1013: Identify the display time of the first frame of the lyrics video frame for the next song.

[0087] Step 1014: Obtain the display time interval between the display time of the last frame of the lyrics video frame of the song and the display time of the first frame of the lyrics video frame of the next song.

[0088] Step 1015: Identify the end time of the song by displaying the time interval.

[0089] By inferring the start and end times of a song from the display times of the lyrics video frames, it can process videos containing multiple songs, automatically segment the time range of each song, and facilitate subsequent analysis of the lyrical characteristics of different songs based on different time ranges.

[0090] It should be noted that determining the end time of a song based on the display time interval falls into two categories. In one category, the display time interval is short, meaning the two songs are close in time. In this case, the end time of the current song is considered the start time of the next song, i.e., the display time of the first frame of the lyrics video of the next song. In the other category, the two songs are far apart in time. In this case, the actual duration of the song is determined based on the general song length or the lyrics information of this particular song, and a second preset duration is set. Then, the start time of the song is delayed by the second preset duration and determined as the end time of the song. The specific steps include:

[0091] If the display time interval is less than or equal to the first preset duration range, the display time of the first frame of the lyrics video frame of the next song will be determined as the end time of the song.

[0092] If the displayed time interval is greater than the first preset duration range, the start time of the song will be delayed by the second preset duration and determined as the end time of the song.

[0093] For example, suppose the first preset duration is 120 seconds, the second preset duration is 300 seconds, and the video to be tested contains three songs, with the lyric video frame display timestamps for each song as follows: First song: First lyric video frame: display time 00:00:10, last lyric video frame: display time 00:00:15. Second song: First lyric video frame: display time 00:02:00, last lyric video frame: display time 00:02:04. Third song: First lyric video frame: display time 00:08:00, last lyric video frame: display time 00:08:07. If the first song is being tested, its start time is the display time of the last lyric video frame, i.e., 00:00:15. If the second song is being detected, its start time is 00:02:00 because the last frame of its lyrics video is displayed at 00:02:04. If the third song is being detected, its start time is 00:08:07 because the last frame of its lyrics video is displayed at 00:08:07. When determining the song's end time, if the first song is being detected, the time interval between the first and second songs is calculated as 00:02:00 - 00:00:15 = 85 seconds. Since 85 seconds < 120 seconds, the end time of the first song is the display time of the first frame of the second song's lyrics video, which is 00:02:00. If the second song is being detected, the time interval between the second and third songs is first calculated as 00:08:00 - 00:02:04 = 356 seconds. Since 356 seconds > 120 seconds, the end time of the second song is the time point after the start time of the song is delayed by the second preset duration (300 seconds = 5 minutes), i.e., 00:02:04 + 00:05:00 = 00:07:04.

[0094] It should be noted that the lyric algorithm in this embodiment of the invention can identify all lyric video frames contained in the video to be detected, and can also determine the display area and display time of each lyric video frame on the playback interface. The latest display time is selected from the display times. If the latest display time is found in the display times of the lyric video frames of the song to be detected, then the song to be detected is determined to be the last song. Since there are no subsequent songs, the display time interval cannot be determined. In this case, the start time of the last song can be directly assumed to be delayed by a second preset duration and determined as the end time of the song. There is also a special case: when the display time interval between two songs is greater than the first preset duration range but less than the second preset duration range, simply delaying the start time of the song by the second preset duration and determining it as the end time of the song might result in the end time of the current song being after the start time of the next song, which is obviously unreasonable. Therefore, in this case, the display time of the first frame of the lyric video frame of the next song is determined as the end time of the song.

[0095] For example, if the first preset duration is 120 seconds and the second preset duration is 300 seconds, the first frame of the lyrics video for the current song displays the time at 00:00:10, and the last frame displays the time at 00:00:20. For the second song, the first frame of the lyrics video displays the time at 00:03:00. When calculating the end time of the current song, the display time interval between the current song and the second song is first calculated: 00:03:00 - 00:00:20 = 160S > 120S. If the start time of the song is delayed by the second preset duration (300S = 00:05:00), then the end time of the current song would be 00:00:20 + 00:05:00 = 00:05:20. 00:05:20 is after the display time of the first frame of the lyrics video of the second song, which is obviously unreasonable. Therefore, the end time of the current song is determined to be 00:03:00.

[0096] In addition, this embodiment of the invention first identifies the text lines in the video to be detected, obtaining the corresponding detection text boxes. A text box is typically a rectangular area defined by the coordinates of its four vertices (or the coordinates of its top-left and bottom-right corners). One or more text boxes may appear in a video frame, and each text box may contain the text lines from that video frame. For example, if a text box appears, it might contain the lyrics: "The brightest star in the night sky."

[0097] See Figure 3 This is a flowchart of another text recognition method provided in an embodiment of the present invention, specifically including:

[0098] Step 201: For any song in the video to be detected, identify the start and end times of the song. The video to be detected contains text boxes obtained from the identified text lines.

[0099] Step 202: Extract several video frames within the first preset duration range after the starting time point.

[0100] Step 203: Identify the positions of text boxes in several video frames, and based on the positions of the text boxes corresponding to each video frame, superimpose the text boxes of several video frames to obtain the lyrics display area of ​​the song.

[0101] Step 204: Perform an intersection-union calculation on the text boxes in the video frames between the lyrics display area and the start and end times of the song to obtain the lyrics text boxes containing the lyrics.

[0102] Step 205: Extract the lyrics of the song from the lyrics text box.

[0103] Steps 201-205 above will not be repeated here, referring to the previous discussion.

[0104] Step 206: Select the first video frame containing the lyrics text box from the video frames between the start and end times of the song.

[0105] In this embodiment of the invention, lyrics are identified in the above steps. Next, dialogue needs to be identified. When identifying dialogue, video frames containing lyric text frames need to be removed first to avoid interference. Here, the first video frame refers to all video frames containing lyric text frames corresponding to all songs in the video to be detected.

[0106] Step 207: Add a preset marker to the first video frame.

[0107] In this embodiment of the invention, a preset marker is added to the first video frame selected. For example, the marker is_lyrics_frame = True is added to the frame to facilitate subsequent differentiation. These markers are used to identify whether there is a lyrics text box in the video frame.

[0108] Step 208: Select the second video frame without the preset tag from the video to be detected.

[0109] In this embodiment of the invention, it is assumed that video frames with preset tags will no longer contain dialogue, while the remaining video frames without preset tags need to have their dialogue extracted. For example, the second video frame without the is_lyrics_frame tag is selected.

[0110] Step 209: Extract the dialogue from the text box of the video to be detected.

[0111] In this embodiment of the invention, when acquiring dialogue, the positions of all text boxes in the second video frame are first obtained, i.e., the coordinates of the text boxes. Then, all text boxes are superimposed, and the area with a superposition count higher than a certain threshold is determined as the dialogue display area. Next, the intersection-union ratio (IUGR) of each text box in the second video frame with the dialogue display area is calculated, and text boxes with IUGR greater than a certain threshold are determined as dialogue text boxes. Then, the dialogue can be extracted from the dialogue text boxes using optical character recognition (OCR) technology. The specific steps include:

[0112] Identify the position of the text box in the second video frame;

[0113] The text boxes of the second video frame are overlaid according to their positions to obtain the dialogue display area of ​​the video to be detected.

[0114] The intersection-union (IoU) ratio of the text box in the second video frame to the dialogue display area is calculated to obtain the dialogue text box containing the dialogue.

[0115] Extract the dialogue from the video to be tested from the dialogue text box.

[0116] By following the steps above, lyrics text boxes and dialogue text boxes can be efficiently filtered out from video frames, and the display areas for lyrics and dialogue can be accurately located.

[0117] See Figure 4 , Figure 4 This is a structural block diagram of a text recognition device provided in an embodiment of this application, such as... Figure 4 As shown, the device may include:

[0118] The first acquisition module 301 is used to identify the start and end times of any song in the video to be detected, where the video to be detected contains text boxes obtained from the identified text lines.

[0119] The second acquisition module 302 is used to extract several video frames within a first preset duration range after the start time point.

[0120] The third acquisition module 303 is used to identify the position of the text box in several video frames, and to superimpose the text boxes of several video frames according to the position of the text box corresponding to each video frame to obtain the lyrics display area of ​​the song.

[0121] The intersection-union ratio (IUGR) calculation module 304 is used to perform IUGR calculation on the text boxes in the video frames between the lyrics display area and the start and end times of the song to obtain the lyrics text boxes containing the lyrics.

[0122] The fourth acquisition module 305 is used to extract the lyrics of a song from the lyrics text box.

[0123] Optionally, the first acquisition module 301 specifically includes:

[0124] The first acquisition submodule is used to identify the display time point of each lyric video frame of any song in the video to be detected. The lyric video frame is a video frame that contains information about the song title, singer, lyricist and composer.

[0125] The first determining submodule is used to determine the display time of the last frame of the lyrics video frame of the song as the start time of the song.

[0126] The second acquisition submodule is used to identify the display time of the first frame of the lyrics video frame of the next song.

[0127] The third acquisition submodule is used to acquire the display time interval between the display time of the last frame of the lyrics video frame of a song and the display time of the first frame of the lyrics video frame of the next song.

[0128] The fourth acquisition submodule is used to identify the end time of a song by displaying time intervals.

[0129] Optionally, the fourth acquisition submodule specifically includes:

[0130] The first determination unit is used to determine the display time of the first frame of the lyrics video frame of the next song as the end time of the song if the display time interval is less than or equal to the first preset duration range.

[0131] The second determination unit is used to determine the end time of the song by postponing the start time point of the song by a second preset time point if the display time interval is greater than the first preset time range.

[0132] Optionally, the third acquisition module 303 specifically includes:

[0133] The coordinate extraction submodule is used to extract the vertex coordinates of the text boxes corresponding to each video frame based on their positions.

[0134] The overlay submodule is used to overlay the text boxes of several video frames onto the same preset blank image using the vertex coordinates of the text boxes corresponding to each video frame.

[0135] The statistics submodule is used to count the frequency of coverage of each region in a preset blank image by text boxes in several video frames.

[0136] The heatmap generation submodule is used to generate a heatmap of text boxes for several video frames based on the coverage frequency. The area with the higher the coverage frequency of the text box in the preset blank map corresponds to the higher the heat value in the heatmap.

[0137] The determination submodule is used to determine the area in the heat map where the heat value is greater than the target preset value as the lyrics display area of ​​the song.

[0138] Optionally, the text recognition device may also include:

[0139] The first filtering module is used to filter out the first video frame containing the lyrics text box from the video frames between the start and end times of the song.

[0140] The marker addition module is used to add preset markers to the first video frame.

[0141] The second filtering module is used to filter out second video frames that have not been marked with a preset tag from the video to be detected.

[0142] The fifth acquisition module is used to extract the dialogue of the video to be detected from the text box of the second video frame.

[0143] Optionally, the fifth acquisition module specifically includes:

[0144] The fifth acquisition submodule is used to identify the position of the text box in the second video frame.

[0145] The text box overlay submodule is used to overlay the text boxes of the second video frame according to their positions to obtain the dialogue display area of ​​the video to be detected.

[0146] The intersection-union (IUU) calculation submodule is used to perform IUU calculation on the text box of the second video frame and the dialogue display area to obtain the dialogue text box containing the dialogue.

[0147] The sixth submodule is used to extract the dialogue from the dialogue text box of the video to be detected.

[0148] This invention also provides an electronic device. Figure 5 This is a structural block diagram of an electronic device provided in an embodiment of the present invention, such as... Figure 5 As shown, it includes a processor 401, a communication interface 402, a memory 403, and a communication bus 404, wherein the processor 401, the communication interface 402, and the memory 403 communicate with each other through the communication bus 404.

[0149] Memory 403 is used to store computer programs;

[0150] When processor 401 executes the program stored in memory 403, it performs the following steps:

[0151] For any song in the video to be detected, identify the start and end times of the song, and the video to be detected contains text boxes obtained by identifying text lines;

[0152] Extract several video frames within a first preset duration range after the starting time point;

[0153] The positions of text boxes in several video frames are identified, and the text boxes of several video frames are superimposed according to the positions of the text boxes corresponding to each video frame to obtain the lyrics display area of ​​the song.

[0154] The intersection-union ratio (IUU) of the lyrics display area and the text boxes in the video frames between the start and end times of the song is calculated to obtain the lyrics text box containing the lyrics.

[0155] Extract the lyrics of the song from the lyrics text box.

[0156] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0157] The communication interface is used for communication between the aforementioned terminal and other devices.

[0158] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0159] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0160] In another embodiment of the present invention, a computer-readable storage medium is also provided, which stores instructions that, when executed on a computer, cause the computer to perform any of the text recognition methods described in the above embodiments.

[0161] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the text recognition methods described in the above embodiments.

[0162] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or third-party database to another website, computer, server, or third-party database via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or third-party database that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid state disks (SSDs)).

[0163] It should be noted that, in this document, relational terms such as "first" and "first" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0164] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0165] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A text recognition method, characterized in that, include: For any song in the video to be detected, identify the start and end times of the song, and the video to be detected contains text boxes obtained by identifying text lines; Extract several video frames within a first preset duration range after the starting time point; The positions of text boxes in several video frames are identified respectively, and the text boxes of several video frames are superimposed according to the positions of the text boxes corresponding to each video frame to obtain the lyrics display area of ​​the song. The intersection-union ratio (IUU) of the lyrics display area and the text boxes in the video frames between the start and end times of the song is calculated to obtain the lyrics text box containing the lyrics. Extract the lyrics of the song from the lyrics text box.

2. The method according to claim 1, characterized in that, The step of identifying the start time point of any song in the video to be detected includes: For any song in the video to be detected, identify the display time point of each lyric video frame displayed by the song, wherein the lyric video frame is a video frame containing information about the song title, singer, lyricist and composer. The display time of the last frame of the lyrics video frame of the song is determined as the start time of the song.

3. The method according to claim 1 or 2, characterized in that, The step of identifying the end time point of any song in the video to be detected further includes: Identify the display time of the first frame of the lyrics video for the next song; The display time interval between the display time of the last frame of the lyrics video frame of the song and the display time of the first frame of the lyrics video frame of the next song is obtained. The end time of the song is identified by the display time interval.

4. The method according to claim 3, characterized in that, The step of identifying the end time of the song through the display time interval includes: If the display time interval is less than or equal to the first preset duration range, then the display time of the first frame of the lyrics video frame of the next song is determined as the end time of the song. If the display time interval is greater than the first preset duration range, the start time of the song is delayed by the second preset duration and determined as the end time of the song.

5. The method according to claim 1, characterized in that, The step of superimposing the text boxes of several video frames according to their respective positions to obtain the lyrics display area of ​​the song includes: Based on the position of the text box corresponding to each video frame, extract the vertex coordinates of the text box corresponding to each video frame; By using the vertex coordinates of the text boxes corresponding to each video frame, the text boxes of several video frames are overlaid on the same preset blank image. The frequency of coverage of each region in the preset blank image by text boxes of several video frames is counted. A heatmap of text boxes for several video frames is generated based on the coverage frequency, wherein the area with the higher coverage frequency of text boxes in the preset blank map corresponds to a higher heat value in the heatmap. The area in the heat map where the heat value is greater than the target preset value is determined as the lyrics display area of ​​the song.

6. The method according to claim 1, characterized in that, After calculating the intersection-union ratio (IoU) of the lyrics display area with the text boxes in the video frames between the start and end times of the song to obtain the lyrics text boxes, the process includes: Filter out the first video frame containing the lyrics text box from the video frames between the start and end times of the song; Add a preset marker to the first video frame; Filter out the second video frame that has not been marked with a preset tag from the video to be detected; Extract the dialogue from the text box of the second video frame.

7. The method according to claim 6, characterized in that, Extracting the dialogue from the text box of the second video frame includes: Identify the position of the text box in the second video frame; The text boxes of the second video frame are superimposed according to their positions to obtain the dialogue display area of ​​the video to be detected. The text box of the second video frame is compared with the dialogue display area by the intersection-union calculation to obtain the dialogue text box containing the dialogue. Extract the dialogue from the video to be detected from the dialogue text box.

8. A text recognition device, characterized in that, include: The first acquisition module is used to identify the start and end time points of any song in the video to be detected, wherein the video to be detected contains text boxes obtained by identifying text lines. The second acquisition module is used to extract a number of video frames within a first preset time range after the starting time point; The third acquisition module is used to identify the positions of text boxes in several video frames respectively, and to superimpose the text boxes of several video frames according to the positions of the text boxes corresponding to each video frame to obtain the lyrics display area of ​​the song. The intersection-union ratio (IUU) calculation module is used to perform IUU calculation on the text boxes in the video frames between the lyrics display area and the start and end times of the song to obtain lyrics text boxes containing lyrics. The fourth acquisition module is used to extract the lyrics of the song from the lyrics text box.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When executing a program stored in memory, the processor implements the steps of the text recognition method according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the text recognition method as described in any one of claims 1-7.

Images