Lyrics time extraction method based on word accuracy and computer readable storage medium

Abstract

The application discloses a song lyrics time extraction method based on word precision and a computer readable storage medium, reads each video frame in a music video; judges whether a text image corresponding to each video frame has color change, and when the color change exists, acquires a color change position; according to the color change position of each video frame, determines a singing start frame and a singing end frame of each sentence of lyrics; respectively performs text recognition on the singing start frame of each sentence of lyrics, and determines left and right boundaries of each word in each sentence of lyrics; according to the color change position of each video frame between the singing start frame and the singing end frame of each sentence of lyrics and the left and right boundaries of each word in each sentence of lyrics, determines a start video frame and an end video frame of each word in each sentence of lyrics; according to a frame rate and frame numbers of the start video frame and the end video frame of each word, determines a start time and an end time of each word. The application can accurately acquire start and end times of each word in lyrics.

Description

Technical Field

[0001] This invention relates to the field of video image processing technology, and in particular to a method for extracting lyrics time based on word precision and a computer-readable storage medium. Background Technology

[0002] Current techniques for extracting song lyrics and timing information typically utilize the song's audio data, employing technologies such as speech recognition models to identify lyrics from the audio and extract their timing information. While these methods are relatively mature, they are highly susceptible to factors such as homophones, near-homophones, prolonged vowels, song tempo, and accompaniment music, leading to the identification of incorrect lyrics and incorrect start and end times. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a method for extracting lyrics time based on word precision and a computer-readable storage medium, which can accurately obtain the start and end time of each word in the lyrics.

[0004] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a lyrics time extraction method based on character precision, comprising:

[0005] Read each video frame from the music video;

[0006] Text images in each video frame are obtained using a text detection algorithm;

[0007] Based on each video frame and the text image corresponding to the previous video frame, determine whether there is a color change in the text image corresponding to each video frame, and if there is a color change, obtain the position of the color change.

[0008] Based on the position of color change in each video frame, determine the start and end frames of each line of lyrics.

[0009] The text recognition algorithm is used to perform text recognition on the starting frame of each line of lyrics to obtain the text information of each line of lyrics and determine the left and right boundaries of each character in each line of lyrics.

[0010] Based on the color change positions of each video frame between the start and end frames of each line of lyrics and the left and right boundaries of each word in each line of lyrics, determine the start and end video frames of each word in each line of lyrics.

[0011] The start and end times of each word are determined based on the frame rate of the music video and the frame numbers of the start and end video frames of each word.

[0012] The present invention also proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0013] The beneficial effects of this invention are as follows: By extracting lyrics text information using optical character recognition technology, it is unaffected by the accompaniment or tempo of the song, and it avoids recognition errors caused by homophones or near-homophones. Furthermore, it can accurately capture the stop times of longer lyrics. This invention can accurately obtain the start and end times of each character in the lyrics. Attached Figure Description

[0014] Figure 1 This is a flowchart of a lyrics time extraction method based on character precision according to an embodiment of the present invention;

[0015] Figure 2 This is a flowchart of the method according to Embodiment 1 of the present invention. Detailed Implementation

[0016] To explain the technical content, objectives, and effects of the present invention in detail, the following description is provided in conjunction with the embodiments and accompanying drawings.

[0017] Please see Figure 1 A method for extracting lyrics time based on character precision, comprising:

[0018] Read each video frame from the music video;

[0019] Text images in each video frame are obtained using a text detection algorithm;

[0020] Based on each video frame and the text image corresponding to the previous video frame, determine whether there is a color change in the text image corresponding to each video frame, and if there is a color change, obtain the position of the color change.

[0021] Based on the position of color change in each video frame, determine the start and end frames of each line of lyrics.

[0022] The text recognition algorithm is used to perform text recognition on the starting frame of each line of lyrics to obtain the text information of each line of lyrics and determine the left and right boundaries of each character in each line of lyrics.

[0023] Based on the color change positions of each video frame between the start and end frames of each line of lyrics and the left and right boundaries of each word in each line of lyrics, determine the start and end video frames of each word in each line of lyrics.

[0024] The start and end times of each word are determined based on the frame rate of the music video and the frame numbers of the start and end video frames of each word.

[0025] As can be seen from the above description, the beneficial effects of the present invention are: it can accurately obtain the start and end time of each word in the lyrics, without being affected by the accompaniment of the song or the speed of the song, and without the problem of recognition errors caused by homophones, near-homophones, etc., and can also accurately capture the stop time of lyrics with a long duration.

[0026] Further, the text images in each video frame are obtained through a text detection algorithm; based on the text images corresponding to each video frame and its previous video frame, it is determined whether there is a color change in the text images corresponding to each video frame, and when a color change exists, the location of the color change is obtained, including:

[0027] Iterate through each video frame in the music video and sequentially obtain a video frame as the current video frame.

[0028] The current video frame is detected by a text detection algorithm to obtain the text bounding box in the current video frame, and the image in the text bounding box is extracted to obtain the text image corresponding to the current video frame.

[0029] The text images corresponding to the current video frame and the previous video frame are converted to grayscale, and the background in the text images corresponding to the current video frame and the previous video frame is removed by a background removal algorithm to obtain grayscale text images corresponding to the current video frame and the previous video frame.

[0030] Subtract the grayscale text image corresponding to the current video frame and the previous video frame to obtain the inter-frame difference map.

[0031] The inter-frame difference map is transformed into a horizontal difference distribution array by using a vertical projection method. The i-th value in the horizontal difference distribution array is the sum of the pixel values ​​in the i-th column of the inter-frame difference map, i∈[1,N], and N is the length of the inter-frame difference map.

[0032] Calculate the mean and standard deviation based on the values ​​in the horizontal difference distribution array, and obtain the maximum value among the values;

[0033] Calculate the first judgment variable based on the standard deviation and the height of the text image corresponding to the current video frame;

[0034] Calculate the second judgment variable based on the maximum and average values;

[0035] If the first judgment variable is greater than a preset first threshold and the second judgment variable is greater than a preset second threshold, then it is determined that the text image corresponding to the current video frame has a color change;

[0036] Based on each value in the horizontal difference distribution array and the average value, calculate the third judgment variable corresponding to each value;

[0037] If the third judgment variable corresponding to the i-th value in the horizontal difference distribution array is greater than the preset third threshold, then the i-th column of pixels in the text image corresponding to the current video frame is taken as the color change position.

[0038] As described above, when a text image undergoes a color change, the data distribution characteristics of the horizontal difference distribution array exhibit a unimodal convexity and a bipolar distribution, meaning that some continuous values ​​are much higher than other values ​​in the array. Based on this characteristic, two judgment variables related to the horizontal difference distribution array are calculated as the basis for judging whether a color change has occurred. Similarly, based on this characteristic, a third judgment variable corresponding to each value is calculated as the basis for judging whether it is the location of a color change.

[0039] Furthermore, the formula for calculating the first judgment variable is K1 = x std / H, the first threshold is 1; the calculation formula for the second judgment variable is K2=(x max -x mean ) / x max The second threshold is 0.94;

[0040] Where K1 and K2 are the first and second judgment variables, respectively, x std Let H be the standard deviation, H be the height of the text image corresponding to the current video frame, and x be the standard deviation. max For the maximum value, x mean This refers to the average value.

[0041] Furthermore, the calculation formula for the third judgment variable is K3 = (x i -x mean ) / x i The third threshold is 0.8;

[0042] Where K3 is the third judgment variable, x mean For the average value, x i This represents the i-th value in the horizontal difference distribution array.

[0043] Furthermore, the step of detecting the text bounding box in the current video frame using a text detection algorithm specifically involves:

[0044] The text detection algorithm is used to detect the lower half of the current video frame to obtain the text bounding box in the current video frame.

[0045] As described above, since lyrics subtitles in music videos generally appear in the lower half of the video frame, text detection can be performed only on the lower half of the current video frame to reduce interference from background text and non-lyric subtitles in the music video.

[0046] Further, the step of performing grayscale processing on the text images corresponding to the current video frame and the previous video frame, and removing the background from the text images corresponding to the current video frame and the previous video frame using a background removal algorithm to obtain grayscale text images corresponding to the current video frame and the previous video frame, specifically involves:

[0047] The text images corresponding to the current video frame and the previous video frame are converted to grayscale to obtain the grayscale images within the boxes corresponding to the current video frame and the previous video frame.

[0048] Based on the preset binarization threshold, the grayscale images within the bounding box corresponding to the current video frame and the previous video frame are binarized to obtain the binarized images within the bounding box corresponding to the current video frame and the previous video frame.

[0049] The connected component analysis function is used to obtain the connected components in the binarized images of the current video frame and the previous video frame, and the pixel values ​​of the pixels in the connected components with an area smaller than a preset area threshold are set to 0.

[0050] Add the binarized images of the current video frame and the corresponding bounding box of the previous video frame, and then binarize the added image to generate the mask corresponding to the current video frame.

[0051] Multiply the mask corresponding to the current video frame by the grayscale image within the bounding box corresponding to the current video frame and the previous video frame respectively to obtain the grayscale text image corresponding to the current video frame and the previous video frame.

[0052] As described above, by eliminating the background in the text image, the changes in the background are prevented from affecting the detection of color changes in the text image. Compared with using a single frame of binary image as a mask, a mask generated based on two adjacent video frames can retain more areas of color change in the text image.

[0053] Further, the step of calculating the third judgment variable corresponding to each value based on the values ​​in the horizontal difference distribution array and the average value specifically involves:

[0054] Based on the color change position of the text image corresponding to the previous video frame of the current video frame, determine the candidate range of the color change position of the text image corresponding to the current video frame, and obtain the continuous subarray corresponding to the candidate range of the color change position in the horizontal difference distribution array.

[0055] Based on the values ​​in the corresponding continuous subarray and the average value, calculate the third judgment variable corresponding to each value in the corresponding continuous subarray.

[0056] Furthermore, the step of determining the candidate range of color change positions of the text image corresponding to the current video frame based on the color change positions of the text image corresponding to the previous video frame specifically involves:

[0057] If the color change position of the text image corresponding to the previous video frame of the current video frame is located between the left and right boundaries of a character, or between the right boundary of the character and the left boundary of the next character, then the candidate range of the color change position of the text image corresponding to the current video frame is determined based on the left boundary of the character and the right boundary of the next character.

[0058] As described above, the detection range of color change locations may be affected by residual background interference, which may lead to the inability to identify color changes or the acquisition of incorrect color change locations. Therefore, it is necessary to narrow the detection range of color change locations to the character level. At the same time, since the color changes of lyrics are sequential, if the text image corresponding to the previous video frame has a color change, the candidate range of color change locations in the current video frame can be determined based on the color change location of the previous video frame.

[0059] Furthermore, the step of performing text recognition on the starting frame of each line of lyrics using a text recognition algorithm to obtain the text information of each line of lyrics and determining the left and right boundaries of each character in each line of lyrics includes:

[0060] The text information of the lyrics is obtained by performing text recognition on the text image corresponding to the starting frame of a lyric's performance using a text recognition algorithm.

[0061] The grayscale text image corresponding to the starting frame of the lyric song is binarized to obtain the binary text image corresponding to the starting frame of the lyric song.

[0062] The binary text image is transformed into a horizontal pixel distribution array by using a vertical projection method. The i-th value in the horizontal pixel distribution array is the sum of the pixel values ​​in the i-th column of the binary text image, i∈[1,M], where M is the length of the binary text image.

[0063] Obtain at least one non-zero continuous subarray from the horizontal pixel distribution array, wherein the non-zero continuous subarray contains at least two non-zero values ​​and the interval between adjacent non-zero values ​​is less than a preset first number of pixels;

[0064] Filter the non-zero contiguous subarrays whose length is less than the preset number of second pixels;

[0065] Based on the pixel column corresponding to each value in each non-zero continuous subarray, determine each initial word boundary that corresponds one-to-one with each non-zero continuous subarray.

[0066] Determine the initial character width corresponding to each initial character boundary, and calculate the standard deviation of the width based on each initial character width;

[0067] Based on the width standard deviation, determine whether the width difference of each initial character boundary is large;

[0068] If not, then determine the left and right boundaries of each character in the lyric based on the boundaries of each initial character;

[0069] If so, the initial character boundary is re-segmented to obtain a new boundary;

[0070] Determine whether the number of new boundaries corresponds to the number of characters in the text information;

[0071] If so, then determine the left and right boundaries of each word in the lyrics of the sentence based on the new boundary.

[0072] If not, then based on the number of characters in the text information, the text image corresponding to the starting frame of the lyric is uniformly segmented to obtain the left and right boundaries of each character in the lyric.

[0073] Furthermore, the re-segmentation of the initial character boundary to obtain a new boundary specifically involves:

[0074] Calculate the average width based on each initial character width;

[0075] The initial character boundaries with an initial character width greater than or equal to the average width are taken as the first initial character boundaries to obtain the long boundary group, and the initial character boundaries with an initial character width less than the average width are taken as the second initial character boundaries to obtain the short boundary group;

[0076] Calculate the standard deviation of the initial character width of each first initial character boundary to obtain the first character width standard deviation, and calculate the standard deviation of the initial character width of each second initial character boundary to obtain the second character width standard deviation;

[0077] If both the first character width standard deviation and the second character width standard deviation are greater than or equal to the preset fourth threshold, then the text image corresponding to the starting frame of the lyric is uniformly segmented according to the number of characters in the text information to obtain the left and right boundaries of each character in the lyric.

[0078] If the first character width standard deviation is greater than or equal to the preset fourth threshold and the second character width standard deviation is less than the preset fourth threshold, then the long boundary group is designated as an untrusted group and the short boundary group is designated as a trustworthy group.

[0079] If the first character width standard deviation is less than the preset fourth threshold and the second character width standard deviation is greater than or equal to the preset fourth threshold, then the long boundary group is regarded as a reliable group and the short boundary group is regarded as an unreliable group.

[0080] If both the first character width standard deviation and the second character width standard deviation are less than the preset fourth threshold, and the number of first initial character boundaries is greater than or equal to the number of second initial character boundaries, then the long boundary group is regarded as a reliable group, and the short boundary group is regarded as an unreliable group.

[0081] If both the first character width standard deviation and the second character width standard deviation are less than the preset fourth threshold, and the number of first initial character boundaries is less than the number of second initial character boundaries, then the long boundary group is designated as an untrusted group, and the short boundary group is designated as a trustworthy group.

[0082] If the long boundary group is an unreliable group, then according to the maximum value of the initial word width of each second initial word boundary, each first initial word boundary in the long boundary group is divided, and the divided boundary and each second initial word boundary in the short boundary group are used as new boundaries.

[0083] If the short boundary group is an untrusted group, then traverse the short boundary group and sequentially obtain a second initial word boundary from the short boundary group as the current second initial word boundary;

[0084] Determine whether the current second initial character boundary is the first or last initial character boundary among all initial character boundaries;

[0085] If so, then merge the current second initial character boundary with its adjacent initial character boundary;

[0086] If not, obtain the initial word width of the two initial word boundaries adjacent to the current second initial word boundary, and calculate the difference between the initial word widths of the two adjacent initial word boundaries;

[0087] Determine whether the difference is greater than the preset number of third pixels;

[0088] If so, then the current second initial character boundary is merged with the initial character boundary of the two adjacent initial character boundaries whose initial character width is smaller;

[0089] If not, calculate the distance between the current second initial character boundary and its two adjacent initial character boundaries respectively, and merge the current second initial character boundary with the initial character boundary that is closer to it;

[0090] After traversing the short boundary group, the merged boundary and each first initial word boundary in the long boundary group are used as the new boundary.

[0091] Further, the step of dividing each first initial character boundary in the long boundary group according to the maximum value of the initial character width of each second initial character boundary, and taking the divided boundary and each second initial character boundary in the short boundary group as the new boundary, specifically involves:

[0092] The maximum value of the initial character width of each second initial character boundary is taken as the correct character width;

[0093] Divide the initial width of a first initial character boundary by the correct character width to obtain the quotient and remainder corresponding to the first initial character boundary, and use the quotient as the maximum number of characters that the first initial character boundary can contain;

[0094] Divide the remainder by the difference between the quotient and the minus one to obtain the character spacing corresponding to the first initial character boundary.

[0095] Based on the correct character width and the character spacing, the first initial character boundary is segmented to obtain the segmented boundary of the first initial character boundary;

[0096] The boundaries after dividing each first initial character boundary in the long boundary group and each second initial character boundary in the short boundary group are taken as new boundaries.

[0097] The present invention also proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0098] Example 1

[0099] Please refer to Figure 2 Embodiment 1 of the present invention is: a method for extracting lyrics timing based on character precision, which can be applied to extracting lyrics timing from music videos. For example... Figure 2 As shown, it includes the following steps:

[0100] S1: Read each video frame in the music video, that is, read all the video frames of the MV.

[0101] S2: Traverse each video frame in the playback order, that is, sequentially obtain a video frame from each video frame as the current video frame, until all video frames have been traversed.

[0102] S3: Obtain the text image in the current video frame using a text detection algorithm.

[0103] That is, the text detection algorithm is used to detect the current video frame, obtain the text bounding box in the current video frame, and then the image in the text bounding box is extracted to obtain the text image corresponding to the current video frame.

[0104] Furthermore, since lyrics subtitles in music videos generally appear in the lower half of the video frame, this step can perform text detection only on the lower half of the current video frame to reduce interference from background text and non-lyric subtitles in the music video.

[0105] S4: Using the change frame detection algorithm, determine whether there is a color change in the text image corresponding to the current video frame. If yes, proceed to step S5. If no, continue to traverse each video frame and obtain the next video frame of the current video frame as the current video frame, that is, return to execute step S2.

[0106] Specifically, this step includes the following steps:

[0107] S401: Perform grayscale processing on the text image corresponding to the current video frame and the previous video frame to obtain the grayscale image within the frame corresponding to the current video frame and the previous video frame.

[0108] S402: Based on the preset binarization threshold, perform binarization processing on the grayscale images within the bounding box corresponding to the current video frame and the previous video frame to obtain the binarized images within the bounding box corresponding to the current video frame and the previous video frame.

[0109] In this embodiment, an adaptive threshold is used for binarization. The threshold is calculated using a Gaussian weighted sum of local neighborhood blocks (ADAPTIVE_THRESH_GAUSSIAN_C, where pixels around (x, y) in the region are weighted according to their distance from the center point using a Gaussian function, and then a constant C is subtracted). The preferred value for the local neighborhood block is 7, and the preferred value for the offset adjustment (i.e., the constant C) is 18, which means that the pixel values ​​of the pixels in the 7-neighborhood are weighted and then subtracted by 18.

[0110] S403: By using the connected component analysis function, obtain the connected components in the bounding binarized images of the current video frame and the previous video frame, and set the pixel values ​​of the pixels in the connected components with an area less than a preset area threshold to 0, thereby obtaining the filtered bounding binarized images of the current video frame and the previous video frame.

[0111] In this embodiment, for each pixel, only the four adjacent pixels above, below, left, and right are analyzed for connected components; the preset area threshold is 7 pixels.

[0112] By filtering out connected components with excessively small areas, residual background noise in the image is reduced.

[0113] Furthermore, since the filtered binarized image of the previous video frame has been obtained after traversing the previous video frame, in steps S401-403, only the current video frame can be processed to obtain the filtered binarized image of the current video frame.

[0114] S404: Add the filtered current video frame and the binarized image within the bounding box corresponding to the previous video frame, and perform binarization on the added image to generate the mask corresponding to the current video frame.

[0115] Specifically, after addition, the pixel values ​​of pixels with a value greater than 0 are set to 1, while pixels with a value of 0 remain unchanged. Compared to using a single-frame binary image as a mask, the mask in this embodiment can preserve more areas of color variation in the text image.

[0116] S405: Multiply the mask corresponding to the current video frame by the grayscale image within the box corresponding to the current video frame and the previous video frame respectively to obtain the grayscale text image corresponding to the current video frame and the previous video frame.

[0117] The mask corresponding to the current video frame is multiplied by the grayscale image within the bounding box corresponding to the current video frame to obtain the grayscale text image corresponding to the current video frame; the mask corresponding to the current video frame is multiplied by the grayscale image within the bounding box corresponding to the previous video frame to obtain the grayscale text image corresponding to the previous video frame.

[0118] Furthermore, in this embodiment, although during the traversal of the previous video frame, the grayscale text image corresponding to the previous video frame is obtained by multiplying the mask and the grayscale image within the box corresponding to the previous video frame, when traversing the current video frame, the grayscale text image corresponding to the previous video frame is obtained by multiplying the mask and the grayscale image within the box corresponding to the previous video frame again. In the subsequent traversal of the current video frame, the grayscale text image corresponding to the previous video frame refers to the grayscale text image obtained by multiplying again.

[0119] S406: Subtract the grayscale text image corresponding to the current video frame and the previous video frame to obtain the inter-frame difference map.

[0120] S407: The inter-frame difference map is converted into a horizontal difference distribution array by the vertical projection method.

[0121] Specifically, the inter-frame difference map is transformed into a one-dimensional horizontal difference distribution array X = {x1, x2, ..., x} using a vertical projection method. N}, where N is the length of the inter-frame difference map, and x is the i-th value in the horizontal difference distribution array X. i It is the sum of the pixel values ​​in the i-th column of the inter-frame difference map, where i∈[1,N].

[0122] S408: Calculate the first judgment variable and the second judgment variable based on the horizontal difference distribution array.

[0123] Specifically, the first judgment variable K1 = x std / H, the second judgment variable K2 = (x max -x mean ) / x max , where x std Let H be the standard deviation of each value in the horizontal difference distribution array, H be the height of the text image corresponding to the current video frame, and x be the standard deviation of each value in the array. max x is the maximum value of each value in the horizontal difference distribution array. mean It is the average value of each value in the horizontal difference distribution array.

[0124] S409: Determine whether the first judgment variable is greater than a preset first threshold and whether the second judgment variable is greater than a preset second threshold. If yes, proceed to step S410; otherwise, proceed to step S411.

[0125] S410: Determine that the text image corresponding to the current video frame has a color change.

[0126] S411: Determine that the text image corresponding to the current video frame does not have a color change.

[0127] In this embodiment, the first threshold is 1 and the second threshold is 0.94. That is, if K1 > 1 and K2 > 0.94, it is considered that the text image corresponding to the current video frame has a color change.

[0128] The principle behind using the first judgment variable K1 as the basis for judging color changes is as follows:

[0129] Because the data distribution characteristics of the horizontal difference distribution array exhibit a unimodal, bipolar distribution when the text image shows color changes—that is, some continuous values ​​are much higher than other values ​​in the array—the standard deviation x of the horizontal difference distribution array is... std It must be a relatively large value. Furthermore, since the font height of the lyrics varies across different songs, it affects the standard deviation of the horizontal difference distribution array calculated based on the different music videos. Therefore, the standard deviation is divided by the height of the text image to reduce this effect.

[0130] The principle behind using the second judgment variable K2 as the basis for judging color changes is as follows:

[0131] Suppose there exists an array Y = {y1, y2, ..., y...} k ,y k+1 ,……,y nThe data distribution exhibits a clear bipolarity. Assume y1 to y... k For a batch of data with relatively small values, the mean is p1, y k+1 to y n Given a set of data with relatively large values ​​and a mean of p2, the mean of this array is:

[0132] y mean =((nk)p2+kp1) / n=p2-k(p2-p1) / n;

[0133] So, p2-y mean =p2-(p2-k(p2-p1) / n)=k(p2-p1) / n;

[0134] Since p2 will be much larger than p1 when the text image undergoes a color change, it can be assumed that:

[0135] p2-y mean =kp2 / n;

[0136] The derivation yields:

[0137] (p2-y mean ) / p2=k / n;

[0138] When the lyrics have no color change or a large color change, the number of data points with small values ​​is very small, meaning n is much larger than k. When the lyrics have a color change, meaning only a small portion of the data points have large values, then we can assume n ≈ k. Therefore, we can assume that when the lyrics have no color change, p2-y mean / p2 is close to 0, and when the lyrics change color, p2-y mean / p2 is close to 1. For ease of calculation, the maximum value y in the array is used. max Replace p2 in the formula and set q = y max -p2, from which we can deduce:

[0139] 1>(y max -y mean ) / y max =1-y mean / (p2+q)>1-y mean / p2=(p2-y mean ) / p2

[0140] 1>(y max -y mean ) / y max >(p2-y mean ) / p2=k / n;

[0141] When the lyrics change color, (p2-y mean ) / p2 is close to 1, (p2-ymean ) / p2 and (y max -y mean ) / y max The difference between them is very small, almost equivalent, therefore, (y) can be... max -y mean ) / y max A value close to 1 is considered a necessary but not sufficient condition for the lyrics to change color.

[0142] S5: Get the position of color change in the current video frame.

[0143] Specifically, based on each value in the horizontal difference distribution array and the average value, a third judgment variable corresponding to each value is calculated; if the third judgment variable corresponding to the i-th value in the horizontal difference distribution array is greater than a preset third threshold, then the i-th column of pixels in the text image corresponding to the current video frame is taken as the color change position.

[0144] In this embodiment, the third judgment variable K corresponds to the i-th value in the horizontal difference distribution array. 3i =(x i -x mean ) / x i The third threshold is 0.8. That is, if K 3i If the value is greater than 0.8, it means that the i-th column of pixels in the text image corresponding to the current video frame has changed color.

[0145] Furthermore, within a video frame, there may be multiple consecutive columns of pixels that exhibit color changes; these consecutive columns of pixels are considered as the locations of the color changes.

[0146] The principle behind using the third judgment variable K3 as the basis for judging color changes is as follows:

[0147] Because the data distribution characteristics of the horizontal difference distribution array exhibit a unimodal, bipolar distribution when the text image shows color changes, the horizontal difference of the pixel column with color changes is much higher than that of the pixel column without color changes, and the number of such pixels is much smaller than that of the pixel column without color changes. Therefore, the horizontal difference x of the pixel column with color changes is... i It should be much larger than the mean x of the horizontal difference distribution array X. mean That is (x i -x mean ) / x i The value is also close to 1.

[0148] Furthermore, since the acquisition of color change locations may be affected by residual background interference, leading to either failure to recognize color changes or incorrect acquisition of color change locations, it is necessary to narrow the detection range of color change locations to the character level. Simultaneously, since the color changes in lyrics are sequential, if the text image corresponding to the previous video frame exhibits a color change, the candidate range for the color change location in the current video frame can be determined based on the color change location of the previous video frame. When calculating the third judgment variable, only the values ​​in the continuous subarray corresponding to this candidate range in the horizontal difference distribution array are calculated to determine the third judgment variable and thus the color change location.

[0149] Specifically, if the color change position of the previous video frame is between the left and right boundaries of a certain character in the current lyric, then the range between the left boundary of that character and the right boundary of the next character is taken as the candidate range for the color change position of the current video frame; if the color change position of the previous video frame is within the character spacing of two adjacent characters in the current lyric, that is, between the right boundary of the preceding character and the left boundary of the following character in the two adjacent characters, then the range between the left boundary of the preceding character and the right boundary of the following character in the two adjacent characters is taken as the candidate range for the color change position of the current video frame.

[0150] S6: Based on the position of the color change in the current video frame, determine whether the current video frame is the starting frame of a lyric. If yes, take the lyric as the current lyric and proceed to step S7. If no, proceed to step S8.

[0151] Specifically, if the distance between the color change position of the current video frame and the left boundary of the text image corresponding to the current video frame is less than a preset distance threshold (in this embodiment, the distance threshold is 2 pixels), then the current video frame is considered to be the starting frame of a certain lyric.

[0152] Furthermore, since the color change position of the current video frame may contain multiple consecutive pixel columns, the above calculation is the distance between the leftmost position of the color change position of the current video frame and the left boundary of the text image corresponding to the current video frame.

[0153] S7: Use a text recognition algorithm to perform text recognition on the starting frame of the current lyric, obtain the text information of the current lyric, and determine the left and right boundaries of each word in the current lyric.

[0154] Since the current video frame is the starting frame for the current lyric, this step involves processing the current video frame.

[0155] Specifically, this step includes the following steps:

[0156] S701: The text recognition algorithm is used to perform text recognition on the starting frame of the current lyric, that is, to perform text recognition on the text image corresponding to the current video frame, so as to obtain the text information of the current lyric.

[0157] S702: Binarize the grayscale text image corresponding to the starting frame of the current lyric's performance to obtain the binary text image corresponding to the starting frame of the current lyric's performance.

[0158] That is, the grayscale text image corresponding to the current video frame obtained in step S405 is binarized. The binarization threshold here is an adaptive threshold, and the threshold is calculated by Gaussian weighted sum of local neighborhood blocks. The preferred value of the local neighborhood block is 3, and the preferred value of the offset adjustment amount (i.e., constant C) is 9. That is, the pixel values ​​of the pixels in the 3-neighborhood are weighted and then subtracted by 9.

[0159] S703: The binary text image is converted into a horizontal pixel distribution array by a vertical projection method.

[0160] Specifically, the binary text image is transformed into a one-dimensional horizontal pixel distribution array Z = {z1, z2, ..., z} using the vertical projection method. M}, where M is the length of the binary text image, and z is the i-th value in the horizontal pixel distribution array Z. i It is the sum of the pixel values ​​in the i-th column of the binary text image, that is, the sum of the pixel values ​​of all pixels in the i-th pixel column, i∈[1,M].

[0161] S704: Obtain at least one non-zero continuous subarray from the horizontal pixel distribution array, wherein the non-zero continuous subarray contains at least two non-zero values ​​and the interval between adjacent non-zero values ​​is less than a preset number of first pixels.

[0162] In the horizontal pixel distribution array, the non-zero contiguous subarrays correspond to the pixel columns in the text image, which roughly represent the range of each character in the lyrics. However, due to the font structure of some characters (such as characters with a left-right structure), their corresponding non-zero pixel columns may be separated. Therefore, when the interval between two non-zero pixel columns is small, they are considered to be contiguous and may correspond to the same character.

[0163] In this embodiment, the number of first pixels is 3. This step involves connecting columns of non-zero pixels with a spacing of less than 3 pixels to reduce separation caused by the font structure.

[0164] S705: Filter the non-zero continuous subarray whose length is less than the preset number of second pixels.

[0165] In this embodiment, the number of second pixels is 2. That is, consecutive non-zero pixel columns with a length of less than 2 pixels are deleted to reduce the impact of residual background.

[0166] S706: Determine each initial word boundary corresponding to each non-zero continuous subarray based on the pixel column corresponding to each value in each non-zero continuous subarray.

[0167] Since the values ​​in the horizontal pixel distribution array correspond one-to-one with the pixel columns in the text image, the corresponding pixel column can be determined based on the sequential position of the values ​​in the non-zero contiguous subarray within the horizontal pixel distribution array. Each non-zero contiguous subarray corresponds to an initial word boundary.

[0168] S707: Determine the initial character width corresponding to each initial character boundary, and calculate the width standard deviation based on each initial character width.

[0169] S708: Based on the width standard deviation, determine whether the width difference of each initial character boundary is large. If yes, proceed to step S710; otherwise, proceed to step S709.

[0170] This means using the standard deviation of width to measure the difference in width between each initial character. If the difference is large, it indicates that there is a problem with the segmentation and it needs to be re-segmented; otherwise, the segmentation is considered to be without problems.

[0171] S709: Determine the left and right boundaries of each character in the current sentence lyrics based on the boundaries of each initial character.

[0172] S710: The initial character boundary is re-segmented to obtain a new boundary.

[0173] S710: Determine whether the number of new boundaries corresponds to the number of characters in the text information. If yes, proceed to step S711; otherwise, proceed to step S712.

[0174] S711: Determine the left and right boundaries of each character in the lyrics of the current sentence based on the new boundary.

[0175] S712: Based on the number of characters in the text information, uniformly segment the text image corresponding to the starting frame of the current lyric to obtain the left and right boundaries of each character in the current lyric.

[0176] That is, determine whether the number of new boundaries obtained by re-segmentation is consistent with the number of characters in the text information identified in step S701. If so, it is considered that there is no problem with re-segmentation; otherwise, it is considered that there is still a problem. At this time, the text image can be uniformly segmented along its length direction according to the number of characters in the text information to obtain the character boundaries.

[0177] Furthermore, for step S710, the specific steps are as follows:

[0178] First, calculate the average width of each initial character width, and then divide all initial character boundaries into two groups based on the average width: the long boundary group and the short boundary group. In the long boundary group, the initial character width of the first initial character boundary is greater than or equal to the average width, while in the short boundary group, the initial character width of the second initial character boundary is less than the average width.

[0179] Then, the standard deviations of the long boundary group and the short boundary group are calculated separately. The standard deviation calculated based on the initial character width of each first initial character boundary is used as the first character width standard deviation, and the standard deviation calculated based on the initial character width of each second initial character boundary is used as the second character width standard deviation.

[0180] Next, based on the standard deviations of the long boundary group and the short boundary group, it is determined whether the long boundary group and the short boundary group are reliable.

[0181] Specifically, if the first character width standard deviation is greater than or equal to the preset fourth threshold and the second character width standard deviation is less than the preset fourth threshold, then the long boundary group is designated as an untrusted group and the short boundary group is designated as a trustworthy group; if the first character width standard deviation is less than the preset fourth threshold and the second character width standard deviation is greater than or equal to the preset fourth threshold, then the long boundary group is designated as a trustworthy group and the short boundary group is designated as an untrustworthy group.

[0182] In this embodiment, the fourth threshold is 5. That is, if the standard deviation of one group is greater than or equal to 5 and the standard deviation of another group is less than 5, then the word boundary segmentation of the group with the smaller standard deviation can be considered reliable, and the word boundary segmentation of the group with the larger standard deviation is unreliable.

[0183] If both the first and second character width standard deviations are less than a preset fourth threshold, and the number of first initial character boundaries is greater than or equal to the number of second initial character boundaries, then the long boundary group is considered a reliable group, and the short boundary group is considered an unreliable group. Conversely, if both the first and second character width standard deviations are less than the preset fourth threshold, and the number of first initial character boundaries is less than the number of second initial character boundaries, then the long boundary group is considered an unreliable group, and the short boundary group is considered a reliable group. In other words, if the standard deviations of both groups are less than 5, then the number of initial character boundaries in the two groups is compared, and the character boundary segmentation of the group with more initial character boundaries is considered reliable, while the character boundary segmentation of the group with fewer initial character boundaries is considered unreliable.

[0184] If both the first and second character width standard deviations are greater than or equal to a preset fourth threshold, then the text image corresponding to the starting frame of the lyric is uniformly segmented according to the number of characters in the text information to obtain the left and right boundaries of each character in the lyric. In other words, if both standard deviations are greater than 5, it is considered difficult to find a corresponding pattern, and the text image is directly uniformly segmented along its length direction according to the number of characters obtained from text recognition to obtain the character boundaries.

[0185] Finally, the initial word boundaries in the untrusted boundary group are re-segmented.

[0186] Specifically, when the long boundary group is an untrusted group, the first initial word boundary in each long boundary group is re-segmented.

[0187] Taking a first initial character boundary as an example, firstly, the maximum initial character width of each second initial character boundary in the short boundary group is taken as the correct character width. Then, the initial character width of the first initial character boundary is divided by the correct character width to obtain a quotient and a remainder. The quotient represents the maximum number of characters that the first initial character boundary can contain. Since there may be whitespace between characters, it is necessary to calculate the spacing between each character within the first initial character boundary. In this embodiment, the remainder obtained from the above calculation is considered as the sum of all character spacings. Since the number of character spacings is one less than the number of characters, dividing the remainder by the difference between the quotient and one gives the character spacing between adjacent characters within the first initial character boundary. In this embodiment, the character spacing corresponding to different first initial character boundaries is generally different.

[0188] Next, based on the correct character width and the character spacing corresponding to the first initial character boundary, the first initial character boundary is segmented to obtain the segmented boundary of the first initial character boundary; that is, the left boundary of the first initial character boundary is taken as the left boundary of the first character contained in the first initial character boundary, the correct character width is added to the position of the left boundary of the first initial character boundary to obtain the right boundary of the first character, the character spacing is added to the right boundary of the first character to obtain the left boundary of the second character, and so on, until the right boundary of the first initial character boundary is the right boundary of the last character contained in the first initial character boundary.

[0189] Finally, after re-segmenting all the first initial word boundaries in the long boundary group, the boundaries after segmentation of each first initial word boundary in the long boundary group and each second initial word boundary in the short boundary group are taken as the new boundaries.

[0190] When the short boundary group is an unreliable group, the short boundary group is traversed, and a second initial character boundary is sequentially obtained from the short boundary group as the current second initial character boundary. If the current second initial character boundary is the first or last initial character boundary among all initial character boundaries, it is directly merged with its adjacent initial character boundaries. If the current second initial character boundary is not the first or last initial character boundary, the initial character width of its two adjacent initial character boundaries is checked. If the initial character width of one of the initial character boundaries is greater than the initial character width of the other initial character boundary by a preset number of third pixels (in this embodiment, the number of third pixels is 5), the current second initial character boundary is merged with the initial character boundary with the smaller initial character width among the two adjacent initial character boundaries. Otherwise, the distance between the current second initial character boundary and its two adjacent initial character boundaries is compared, and the current second initial character boundary is merged with the initial character boundary that is closer to it. After traversing the short boundary group, the merged boundary and each first initial character boundary in the long boundary group are used as the new boundary.

[0191] Furthermore, the starting frame of the current lyric can also be considered as the starting video frame of the first word in the current lyric. Therefore, if the current video frame is the starting frame of the current lyric, the start time of the first word in the current lyric can be determined based on the frame rate of the music video and the frame number of the current video frame.

[0192] S8: Based on the position of the color change in the current video frame, determine whether the current video frame is the end frame of the current lyric. If yes, proceed to step S11. If no, that is, the current video frame has a color change, but is neither the start frame nor the end frame of the song, it means that the current video frame is located between the start frame and the end frame of the current lyric. Proceed to step S9.

[0193] Specifically, if the color change position of the current video frame coincides with the right boundary of the last word in the lyrics of the current sentence, then the current video frame is considered to be the end frame of the current sentence of lyrics.

[0194] S9: Based on the color change position of the current video frame and the left and right boundaries of each character in the current lyric, determine whether the current video frame is the starting or ending video frame of a character in the current lyric. That is, determine whether the color change position of the current video frame coincides with the left and right boundaries of a character in the current lyric. If yes, proceed to step S10. If no, it means that the color change position of the current video frame is located in the middle of a character in the current lyric. Then continue to traverse each video frame, obtain the next video frame of the current video frame as the current video frame, and return to execute step S2.

[0195] In other words, if the color change position of the current video frame is the left boundary of a certain word in the current lyric, then the current video frame is considered to be the starting video frame of that word; if the color change position of the current video frame is the right boundary of a certain word in the current lyric, then the current video frame is considered to be the ending video frame of that word.

[0196] Furthermore, since the color change position of the current video frame may contain multiple consecutive pixel columns, in this embodiment, the rightmost position of the color change position of the current video frame is compared with the left and right boundaries of each word in the lyrics of the current sentence for judgment.

[0197] S10: Determine the start or end time of the word based on the frame rate of the music video and the frame number of the current video frame.

[0198] Specifically, based on the frame rate of the music video, the duration of each video frame can be obtained. Then, combined with the frame number of the current video frame, the time point of the current video frame in the music video can be calculated, and this time point is used as the start or end time of the word.

[0199] Then continue to traverse each video frame, obtain the next video frame of the current video frame as the current video frame, and return to execute step S2.

[0200] S11: Determine the end time of the last word in the lyrics of the current sentence based on the frame rate of the music video and the frame number of the current video frame.

[0201] Since the video frame at the end of the last word of the current lyric is the same as the singing end of the current lyric, if the current video frame is the singing end of the current lyric, then the current video frame is the video frame at the end of the last word of the current lyric. In this case, the end time of the last word in the current lyric can be calculated based on the frame rate and the frame number of the current video frame.

[0202] Then, continue traversing each video frame, obtaining the next video frame as the current video frame, and returning to execute step S2. At this point, the starting frame for the next line of lyrics is searched in the subsequent video frames. Since the overall display position of the lyrics in the music video is generally unchanged, the starting frame for the next line of lyrics can be further detected based on the bounding box of the text image corresponding to the ending frame of the current line of lyrics.

[0203] This embodiment utilizes the characteristics of lyrics appearing line by line and exhibiting color changes in music videos (MVs). It employs a frame-by-frame traversal method to extract the lyrics text and time from the MV line by line and word by word. By using optical character recognition (OCR) technology to extract the lyrics text information, it is unaffected by the song's accompaniment or tempo, and avoids recognition errors caused by homophones or near-homophones. Furthermore, it can accurately capture the pause times of longer lyrics.

[0204] Example 2

[0205] This embodiment is a computer-readable storage medium corresponding to the above embodiments, on which a computer program is stored. When the program is executed by a processor, it implements the various steps of the word-precision-based lyrics time extraction method in the above embodiments and can achieve the same technical effect, which will not be repeated here.

[0206] In summary, the present invention provides a word-precision-based lyrics timing extraction method and a computer-readable storage medium. By using optical character recognition technology to extract lyrics text information, it is unaffected by the accompaniment or tempo of the song, and avoids recognition errors caused by homophones or near-homophones. Furthermore, it can accurately capture the stop times of longer lyrics. This invention can accurately obtain the start and end times of each word in the lyrics.

[0207] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent modifications made based on the content of the present invention specification and drawings, or direct or indirect applications in related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for extracting lyrics time based on character precision, characterized in that, include: Read each video frame from the music video; Text images in each video frame are obtained using a text detection algorithm; Based on each video frame and the text image corresponding to the previous video frame, determine whether there is a color change in the text image corresponding to each video frame, and if there is a color change, obtain the position of the color change. Based on the position of color change in each video frame, determine the start and end frames of each line of lyrics. The text recognition algorithm is used to perform text recognition on the starting frame of each line of lyrics to obtain the text information of each line of lyrics and determine the left and right boundaries of each character in each line of lyrics. Based on the color change positions of each video frame between the start and end frames of each line of lyrics and the left and right boundaries of each word in each line of lyrics, determine the start and end video frames of each word in each line of lyrics. The start and end times of each word are determined based on the frame rate of the music video and the frame numbers of the start and end video frames of each word. The text image in each video frame is obtained through a text detection algorithm; based on the text image corresponding to each video frame and its previous video frame, it is determined whether there is a color change in the text image corresponding to each video frame, and when a color change exists, the location of the color change is obtained, including: Iterate through each video frame in the music video and sequentially obtain a video frame as the current video frame. The current video frame is detected by a text detection algorithm to obtain the text bounding box in the current video frame, and the image in the text bounding box is extracted to obtain the text image corresponding to the current video frame. The text images corresponding to the current video frame and the previous video frame are converted to grayscale, and the background in the text images corresponding to the current video frame and the previous video frame is removed by a background removal algorithm to obtain grayscale text images corresponding to the current video frame and the previous video frame. Subtract the grayscale text image corresponding to the current video frame and the previous video frame to obtain the inter-frame difference map. The inter-frame difference map is transformed into a horizontal difference distribution array by using a vertical projection method. The i-th value in the horizontal difference distribution array is the sum of the pixel values ​​in the i-th column of the inter-frame difference map, i∈[1,N], and N is the length of the inter-frame difference map. Calculate the mean and standard deviation based on the values ​​in the horizontal difference distribution array, and obtain the maximum value among the values; Calculate the first judgment variable based on the standard deviation and the height of the text image corresponding to the current video frame; Calculate the second judgment variable based on the maximum and average values; If the first judgment variable is greater than a preset first threshold and the second judgment variable is greater than a preset second threshold, then it is determined that the text image corresponding to the current video frame has a color change; Based on each value in the horizontal difference distribution array and the average value, calculate the third judgment variable corresponding to each value; If the third judgment variable corresponding to the i-th value in the horizontal difference distribution array is greater than the preset third threshold, then the i-th column of pixels in the text image corresponding to the current video frame is taken as the color change position.

2. The lyrics time extraction method based on character precision according to claim 1, characterized in that, The formula for calculating the first judgment variable is K1=x std / H, the first threshold is 1; the calculation formula for the second judgment variable is K2=(x max -x mean ) / x max The second threshold is 0.94; Where K1 and K2 are the first and second judgment variables, respectively, x std Let H be the standard deviation, H be the height of the text image corresponding to the current video frame, and x be the standard deviation. max For the maximum value, x mean This refers to the average value.

3. The lyric timing extraction method based on character precision according to claim 1, characterized in that, The formula for calculating the third judgment variable is K3=(x i -x mean ) / x i The third threshold is 0.8; Where K3 is the third judgment variable, x mean For the average value, x i This represents the i-th value in the horizontal difference distribution array.

4. The lyric timing extraction method based on character precision according to claim 1, characterized in that, The step of detecting the text bounding boxes in the current video frame using a text detection algorithm specifically involves: The text detection algorithm is used to detect the lower half of the current video frame to obtain the text bounding box in the current video frame.

5. The lyric timing extraction method based on character precision according to claim 1, characterized in that, The process of converting the text images corresponding to the current video frame and the previous video frame to grayscale and then removing the background from the text images corresponding to the current video frame and the previous video frame using a background removal algorithm to obtain grayscale text images corresponding to the current video frame and the previous video frame is as follows: The text images corresponding to the current video frame and the previous video frame are converted to grayscale to obtain the grayscale images within the boxes corresponding to the current video frame and the previous video frame. Based on the preset binarization threshold, the grayscale images within the bounding box corresponding to the current video frame and the previous video frame are binarized to obtain the binarized images within the bounding box corresponding to the current video frame and the previous video frame. The connected component analysis function is used to obtain the connected components in the binarized images of the current video frame and the previous video frame, and the pixel values ​​of the pixels in the connected components with an area smaller than a preset area threshold are set to 0. Add the binarized images of the current video frame and the corresponding bounding box of the previous video frame, and then binarize the added image to generate the mask corresponding to the current video frame. Multiply the mask corresponding to the current video frame by the grayscale image within the bounding box corresponding to the current video frame and the previous video frame respectively to obtain the grayscale text image corresponding to the current video frame and the previous video frame.

6. The lyrics time extraction method based on character precision according to claim 1, characterized in that, The step of calculating the third judgment variable corresponding to each value based on the values ​​in the horizontal difference distribution array and the average value is as follows: Based on the color change position of the text image corresponding to the previous video frame of the current video frame, determine the candidate range of the color change position of the text image corresponding to the current video frame, and obtain the continuous subarray corresponding to the candidate range of the color change position in the horizontal difference distribution array. Based on the values ​​in the corresponding continuous subarray and the average value, calculate the third judgment variable corresponding to each value in the corresponding continuous subarray.

7. The lyric timing extraction method based on character precision according to claim 6, characterized in that, The step of determining the candidate range of color change positions of the text image corresponding to the current video frame based on the color change position of the text image corresponding to the previous video frame is as follows: If the color change position of the text image corresponding to the previous video frame of the current video frame is located between the left and right boundaries of a character, or between the right boundary of the character and the left boundary of the next character, then the candidate range of the color change position of the text image corresponding to the current video frame is determined based on the left boundary of the character and the right boundary of the next character.

8. The lyric timing extraction method based on character precision according to claim 1, characterized in that, The step involves using a text recognition algorithm to perform text recognition on the starting frame of each line of lyrics, obtaining the text information of each line of lyrics, and determining the left and right boundaries of each character in each line of lyrics, including: The text information of the lyrics is obtained by performing text recognition on the text image corresponding to the starting frame of a lyric's performance using a text recognition algorithm. The grayscale text image corresponding to the starting frame of the lyric song is binarized to obtain the binary text image corresponding to the starting frame of the lyric song. The binary text image is transformed into a horizontal pixel distribution array by using a vertical projection method. The i-th value in the horizontal pixel distribution array is the sum of the pixel values ​​in the i-th column of the binary text image, i∈[1,M], where M is the length of the binary text image. Obtain at least one non-zero continuous subarray from the horizontal pixel distribution array, wherein the non-zero continuous subarray contains at least two non-zero values ​​and the interval between adjacent non-zero values ​​is less than a preset first number of pixels; Filter the non-zero contiguous subarrays whose length is less than the preset number of second pixels; Based on the pixel column corresponding to each value in each non-zero continuous subarray, determine each initial word boundary that corresponds one-to-one with each non-zero continuous subarray. Determine the initial character width corresponding to each initial character boundary, and calculate the standard deviation of the width based on each initial character width; Based on the width standard deviation, determine whether the width difference of each initial character boundary is large; If not, then determine the left and right boundaries of each character in the lyric based on the boundaries of each initial character; If so, the initial character boundary is re-segmented to obtain a new boundary; Determine whether the number of new boundaries corresponds to the number of characters in the text information; If so, then determine the left and right boundaries of each word in the lyrics of the sentence based on the new boundary. If not, then based on the number of characters in the text information, the text image corresponding to the starting frame of the lyric is uniformly segmented to obtain the left and right boundaries of each character in the lyric.

9. The lyric timing extraction method based on character precision according to claim 8, characterized in that, The specific steps for re-segmenting the initial character boundary to obtain the new boundary are as follows: Calculate the average width based on each initial character width; The initial character boundaries with an initial character width greater than or equal to the average width are taken as the first initial character boundaries to obtain the long boundary group, and the initial character boundaries with an initial character width less than the average width are taken as the second initial character boundaries to obtain the short boundary group; Calculate the standard deviation of the initial character width of each first initial character boundary to obtain the first character width standard deviation, and calculate the standard deviation of the initial character width of each second initial character boundary to obtain the second character width standard deviation; If both the first character width standard deviation and the second character width standard deviation are greater than or equal to the preset fourth threshold, then the text image corresponding to the starting frame of the lyric is uniformly segmented according to the number of characters in the text information to obtain the left and right boundaries of each character in the lyric. If the first character width standard deviation is greater than or equal to the preset fourth threshold and the second character width standard deviation is less than the preset fourth threshold, then the long boundary group is designated as an untrusted group and the short boundary group is designated as a trustworthy group. If the first character width standard deviation is less than the preset fourth threshold and the second character width standard deviation is greater than or equal to the preset fourth threshold, then the long boundary group is regarded as a reliable group and the short boundary group is regarded as an unreliable group. If both the first character width standard deviation and the second character width standard deviation are less than the preset fourth threshold, and the number of first initial character boundaries is greater than or equal to the number of second initial character boundaries, then the long boundary group is regarded as a reliable group, and the short boundary group is regarded as an unreliable group. If both the first character width standard deviation and the second character width standard deviation are less than the preset fourth threshold, and the number of first initial character boundaries is less than the number of second initial character boundaries, then the long boundary group is designated as an untrusted group, and the short boundary group is designated as a trustworthy group. If the long boundary group is an unreliable group, then according to the maximum value of the initial word width of each second initial word boundary, each first initial word boundary in the long boundary group is divided, and the divided boundary and each second initial word boundary in the short boundary group are used as new boundaries. If the short boundary group is an untrusted group, then traverse the short boundary group and sequentially obtain a second initial word boundary from the short boundary group as the current second initial word boundary; Determine whether the current second initial character boundary is the first or last initial character boundary among all initial character boundaries; If so, then merge the current second initial character boundary with its adjacent initial character boundary; If not, obtain the initial word width of the two initial word boundaries adjacent to the current second initial word boundary, and calculate the difference between the initial word widths of the two adjacent initial word boundaries; Determine whether the difference is greater than the preset number of third pixels; If so, then the current second initial character boundary is merged with the initial character boundary of the two adjacent initial character boundaries whose initial character width is smaller; If not, calculate the distance between the current second initial character boundary and its two adjacent initial character boundaries respectively, and merge the current second initial character boundary with the initial character boundary that is closer to it; After traversing the short boundary group, the merged boundary and each first initial word boundary in the long boundary group are used as the new boundary.

10. The lyric timing extraction method based on character precision according to claim 9, characterized in that, The step involves dividing each first initial character boundary in the long boundary group according to the maximum value of the initial character width of each second initial character boundary, and using the resulting boundary and each second initial character boundary in the short boundary group as new boundaries. The maximum value of the initial character width of each second initial character boundary is taken as the correct character width; Divide the initial width of a first initial character boundary by the correct character width to obtain the quotient and remainder corresponding to the first initial character boundary, and use the quotient as the maximum number of characters that the first initial character boundary can contain; Divide the remainder by the difference between the quotient and the minus one to obtain the character spacing corresponding to the first initial character boundary. Based on the correct character width and the character spacing, the first initial character boundary is segmented to obtain the segmented boundary of the first initial character boundary; The boundaries after dividing each first initial character boundary in the long boundary group and each second initial character boundary in the short boundary group are taken as new boundaries.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-10.

Images