Video line extraction method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a video line extraction method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a to-be-processed video and a video type of the to-be-processed video; performing frame extraction processing on the to-be-processed video to obtain a plurality of frame extraction images of the to-be-processed video; determining a reference line text box of the plurality of frame extraction images according to the text box in each frame extraction image; determining a special line text box of the plurality of frame extraction images in the text box according to the video type and the reference line text box; and extracting lines located in the reference line text box and the special line text box from the plurality of frame extraction images. According to the embodiment of the invention, the reference line text box and the special line text box are combined, so that line omission can be avoided, and the recall rate of the video lines is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of text detection, and in particular to a method, device, electronic equipment and storage medium for extracting video lines. Background Art

[0002] In the process of extracting dialogues from long videos, heat map statistics will be performed on the text boxes detected in the frame-sampling images of the long video to filter the dialogues and obtain the effective dialogue text area, and then the dialogues will be extracted through text recognition.

[0003] In the prior art, when extracting dialogues from a video, a heat map is performed on the detected text boxes, and the dialogue text area is obtained by performing position regression on the area with the highest heat in the heat map, and the text area is used as the dialogue area of ​​the video.

[0004] However, the heat map statistics method may not be applicable to all videos. For example, in videos with irregular dialogue layout, if the heat map statistics method is still used to extract the video dialogue, the dialogue content in the irregular area will be omitted, resulting in insufficient recall rate of the dialogue. Summary of the invention

[0005] The purpose of the embodiments of the present invention is to provide a method, device, electronic device and storage medium for extracting video lines to improve the recall rate of video lines. The specific technical solution is as follows:

[0006] In a first aspect of the present invention, a method for extracting lines from a video is provided, comprising:

[0007] Obtain a video to be processed and a video type of the video to be processed;

[0008] Performing frame extraction processing on the video to be processed to obtain multiple frame extracted images of the video to be processed;

[0009] Determining the reference dialogue text frame of the plurality of frame-by-frame images according to the text frame in each of the frame-by-frame images;

[0010] Determining, in the text box, a special text box of the plurality of frame-sampling images according to the video type and the reference text box of the text box;

[0011] The lines located in the reference line text box and the special line text box are extracted from the multiple frame-sampling images.

[0012] In a second aspect of the present invention, a video dialogue extraction device is provided, comprising:

[0013] An acquisition module, used for acquiring a video to be processed and a video type of the video to be processed;

[0014] A frame extraction module is used to perform frame extraction processing on the video to be processed to obtain multiple frame extraction images of the video to be processed;

[0015] A reference dialogue frame determination module, configured to determine reference dialogue text frames of the plurality of frame-by-frame images according to a text frame in each of the frame-by-frame images;

[0016] A special dialogue frame determination module, used for determining the special dialogue text frame of the plurality of frame-sampling images in the text frame according to the video type and the reference dialogue text frame;

[0017] The dialogue extraction module is used to extract the dialogue located in the reference dialogue text box and the special dialogue text box from the multiple frame-sampling images.

[0018] In another aspect of the present invention, there is provided an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0019] Memory, used to store computer programs;

[0020] The processor is used to implement any of the above-mentioned steps of the video dialogue extraction method when executing the program stored in the memory.

[0021] In another aspect of the present invention, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, the computer executes any of the above-mentioned methods for extracting video lines.

[0022] In another aspect of the present invention, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute any of the above-mentioned methods for extracting video lines.

[0023] The video dialogue extraction method, device, electronic device and storage medium provided by the embodiments of the present invention, after determining the baseline dialogue text box of multiple framed images according to the text box in each framed image, further determine the special dialogue text box of multiple framed images in the text box according to the video type and the baseline dialogue text box, and then extract the lines located in the baseline dialogue text box and the special dialogue text box. In this way, the special layout of the lines is fully considered on the basis of the baseline dialogue text box to obtain the special dialogue text box. By combining the baseline dialogue text box and the special dialogue text box, the omission of the lines can be avoided and the recall rate of the video lines can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0025] Figure 1 is a flow chart of a video dialogue extraction method provided by an embodiment of the present invention;

[0026] Figure 2a is an example diagram of a line of reference lines in an embodiment of the present invention;

[0027] Figure 2b is an example diagram of a first special dialogue text box in an embodiment of the present invention;

[0028] Figure 2c is an example diagram of a second special dialogue text box in an embodiment of the present invention;

[0029] Figure 3a is an example diagram of two lines of reference dialogue in an embodiment of the present invention;

[0030] Figure 3b is an example diagram of a special dialogue text box in the case of two lines of reference dialogue in an embodiment of the present invention;

[0031] Figure 4a is an example diagram of reference lines in a variety show type video in an embodiment of the present invention;

[0032] Figure 4b This is an example diagram of a special dialogue text box in a variety show type video in an embodiment of the present invention;

[0033] Figure 5 It is a structural schematic diagram of a video dialogue extraction device provided by an embodiment of the present invention;

[0034] Figure 6 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0036] The embodiment of the present invention determines the benchmark line text box in the video to be processed through heat map statistics, and then determines the special line text box with a complex layout based on the video type of the video to be processed and takes the benchmark line text box as a benchmark, and then extracts the lines based on the benchmark line text box and the special line text box, which can improve the recall rate of the lines.

[0037] Figure 1 is a flow chart of a video dialogue extraction method provided by an embodiment of the present invention. Figure 1As shown, the video line extraction method includes the following steps:

[0038] Step 110: Obtain a video to be processed and a video type of the video to be processed.

[0039] The video type may include, for example, a film or TV series type or a variety show type.

[0040] The layout of the lines in the videos to be processed of different video types may be different, so obtaining the video type here can provide a basis for subsequently determining the special position of the lines (special line text box).

[0041] Step 120: perform frame extraction processing on the video to be processed to obtain multiple frame extracted images of the video to be processed.

[0042] The video to be processed may be subjected to frame extraction processing at a certain frame extraction rate to obtain a plurality of frame extracted images in the video to be processed.

[0043] Step 130 , determining the reference dialogue text frame of the plurality of frame-by-frame images according to the text frame in each of the frame-by-frame images.

[0044] The reference dialogue text box is the position where the dialogue appears most frequently in the video to be processed.

[0045] Perform text detection on each framed image to obtain the text box in each framed image. Perform heat map statistics on the text boxes in all framed images, and obtain the position with the highest frequency of text boxes in the framed images as the reference line text box. The reference line text box is the position information of the lines under the conventional layout in the video to be processed, and the position information is the line position information applicable to the entire video to be processed. Based on the reference line text box, the specific situation of each framed image containing lines can be determined. For example, through the reference line text box, it can be preliminarily determined whether there are lines in each framed image and the position of the lines in the framed image.

[0046] Exemplarily, according to the text box in each framed image, the reference dialogue text box in multiple framed images is determined, which may optionally include: creating a first two-dimensional array, the initial value of each element in the first two-dimensional array is 0, and the size of the first two-dimensional array is the same as the size of the framed image; counting the text boxes in each framed image in turn, when the text box appears, adding 1 to the original value of the element at the corresponding position of the text box in the first two-dimensional array; obtaining the largest value in the first two-dimensional array as the first highest heat value; retaining the element values ​​whose element values ​​are greater than a preset ratio of the first highest heat value, and setting the remaining elements to 0 to obtain a first reserved area; determining the minimum circumscribed rectangle of the first reserved area to obtain the reference dialogue text box. The preset ratio may be, for example, 50%.

[0047] Step 140: Determine, in the text box, a special text box for the plurality of extracted frame images according to the video type and the reference text box for the text box.

[0048] Different video types may have different layouts of other potential lines. Therefore, based on the video type of the video to be processed, the reference line text box may be used as a reference to determine all potential line positions in a complex layout around the reference line text box to obtain a special line text box. Exemplarily, the text boxes in each frame-drawing image may be matched based on the line layout information corresponding to the video type to determine the special line text box of multiple frame-drawing images. Among them, for the film and television drama type, in the case of a line of reference lines, the line layout information may include, for example: there are special lines (such as dialogue lines) that are close in height to the reference lines and are arranged left and right, and / or there is a line of special lines above or below the reference lines. For the film and television drama type, in the case of two lines of reference lines, the line layout information may include, for example: there is a line of special lines (such as Chinese-English parallel lines, when the English is longer, there will be an extra line of English lines) above or below the reference lines.

[0049] Step 150: extracting the lines located in the reference line text box and the special line text box from the plurality of frame-sampling images.

[0050] After determining the baseline dialogue text box and the special dialogue text box in the video to be processed, the lines located in the baseline dialogue text box and the special dialogue text box can be extracted from multiple frame-by-frame images respectively, and the lines extracted from each frame-by-frame image can be combined into the video lines of the video to be processed according to the time sequence of each frame-by-frame image.

[0051] The video dialogue extraction method provided in the present embodiment determines the baseline dialogue text box of multiple framed images according to the text box in each framed image, and then further determines the special dialogue text box of multiple framed images in the text box according to the video type and the baseline dialogue text box, and then extracts the lines located in the baseline dialogue text box and the special dialogue text box. In this way, the special layout of the lines is fully considered on the basis of the baseline dialogue text box to obtain the special dialogue text box. By combining the baseline dialogue text box and the special dialogue text box, the omission of lines can be avoided and the recall rate of video lines can be improved.

[0052] On the basis of the above technical solution, the special line text boxes of the multiple frame-dropped images are determined in the text box according to the video type and the benchmark line text box, that is, the above step 140, including: when the video type is a film and television drama type, the benchmark line type of the video to be processed is determined according to the benchmark line text box; according to the benchmark line type, the remaining text boxes in the text box located outside the benchmark line text box are counted, and the special line text boxes of the multiple frame-dropped images are determined according to the statistical results.

[0053] The reference lines type may include one line of reference lines or two lines of reference lines. One line of reference lines means that there is one line of reference lines in a video. Two lines of reference lines means that there are two lines of reference lines in a video, such as Chinese and English parallel lines.

[0054] When the video type is a film or TV drama type, determine whether the baseline dialogue text box in the frame-extracted image includes one text box or two text boxes above and below. If the baseline dialogue text box includes one text box, determine that the baseline dialogue type of the video to be processed is one line of baseline dialogue. If the baseline dialogue text box includes two text boxes above and below, determine that the baseline dialogue type of the video to be processed is two lines of baseline dialogue.

[0055] Different base speech types have different special layouts of speech, so the method of determining special speech text boxes is also different. After determining the base speech type of the video to be processed, the text box statistics method corresponding to the base speech type can be used to count the remaining speech text boxes outside the base speech text box in all text boxes, and determine the special speech text boxes of multiple frame images based on the statistical results.

[0056] By counting the remaining dialogue text boxes outside the benchmark dialogue text box in a corresponding manner based on the benchmark dialogue type of the video to be processed when the video type is a film and television drama type, and determining the special dialogue text box based on the statistical results, the accuracy of determining the special dialogue text box can be improved, and the recall rate of the dialogue can be further improved.

[0057] In one embodiment of the present invention, the reference line type includes a line of reference lines; the special line text box includes a first special line text box and / or a second special line text box;

[0058] The method of counting the remaining text boxes in the text box outside the reference text box according to the reference line type, and determining the special line text boxes of the plurality of frame-sampling images according to the statistical result, comprises:

[0059] According to the reference speech text frame, a first special speech text frame is determined from the remaining text frames, wherein in the same extracted frame image, two first special speech text frames are arranged on the left and right, the height of the first special speech text frame is within a preset multiple range of the height of the reference speech text frame, and the center point height of the first special speech text frame is within the height range of the reference speech text frame;

[0060] Acquire other text boxes outside the first special lines text box from the remaining text boxes;

[0061] According to the reference speech text box, a second special speech text box is determined from the other text boxes, wherein the second special speech text box and the reference speech text box are included in the same frame-drawing image, the height of the second special speech text box is within a preset multiple range of the height of the reference speech text box, the absolute value of the difference between the height of the center point of the second special speech text box and the height of the center point of the reference speech text box is within a preset height range, and the difference between the horizontal coordinate of the center point of the second special speech text box and the horizontal coordinate of the center point of the reference speech text box is within a preset difference range.

[0062] Various irregular dialogue layout situations can be collected in advance to determine all potential dialogue layout templates, so that based on the dialogue layout template, it is possible to purposefully determine whether a video has an irregular dialogue layout situation. In the determined dialogue layout template, for a line of reference dialogue, there are two sections of dialogue arranged on the left and right, or there is another line of dialogue above or below the reference dialogue.

[0063] Figure 2a is an example diagram of a line of reference lines in an embodiment of the present invention, Figure 2b is an example diagram of the first special dialogue text box in an embodiment of the present invention, Figure 2c is an example diagram of the second special dialogue text box in an embodiment of the present invention. Figure 2a and Figure 2b As shown, Figure 2a This is a line of standard dialogue, which is the layout of the lines that appear more frequently. Figure 2b It is an occasional line layout situation in the case of a line of basic lines, with two lines distributed on the left and right, such as lines of dialogue. Figure 2a and Figure 2c As shown, Figure 2a This is a line of standard dialogue, which is the layout of the lines that appear more frequently. Figure 2b It is a situation where two lines of lines are occasionally arranged in the case of one line of basic lines, that is, there is another line of lines above or below one line of basic lines.

[0064] like Figure 2a and Figure 2b As shown, the y-axis position information of the first special line text box is close to the y-axis position information of the reference line text box, while the x-axis position information of the first special line text box is different from the x-axis position information of the reference line text box. According to the reference line text box, a text box that meets the following conditions can be screened out from the remaining text boxes: the center point height is within the height range of the reference line text box, the height of the text box is within the preset multiple range of the height of the reference line text box, and there are two text boxes arranged on the left and right in the same frame image, and the screened text boxes are subjected to special graph statistics to determine the first special line text box.

[0065] like Figure 2a and Figure 2c As shown, the y-axis position information of the second special line text box and the y-axis position of the reference line text box (gt_rects) are in a top-to-bottom layout, and the x-axis position information of the second special line text box is close to the x-axis position of the reference line text box. According to the reference line text box, statistics can be performed on all text boxes except the reference line text box and the first special line text box, and text boxes that meet the following conditions can be screened out from other text boxes: the height of the text box is within a preset multiple range of the height of the reference line text box, the absolute value of the difference between the center point height and the center point height of the reference line text box is within a preset height range, the difference between the center point horizontal coordinate and the center point horizontal coordinate of the reference line text box is within a preset difference range, and the text box is located above or below the reference line text box in the same frame image, and the screened text boxes are statistically analyzed to determine the second special line text box.

[0066] By taking into account the special lines of different layouts in the case of one line of baseline lines, the recall rate of lines can be further improved.

[0067] On the basis of the above technical solution, the method of determining a first special speech text box from the remaining text boxes according to the reference speech text box includes: determining a first height of the reference speech text box, determining a second height of each of the remaining text boxes, and determining a first center point height of each of the remaining text boxes; obtaining, from all the remaining text boxes, remaining text boxes whose second height is within a preset multiple of the first height and whose first center point height is within the height range of the reference speech text box as first target text boxes; generating first heat map rectangular boxes corresponding to each of the first target text boxes; if two first heat map rectangular boxes arranged on the left and right are included in the same frame image, the center point height of the first heat map rectangular box is within the height range of the reference speech text box, and the number of remaining text boxes within the first heat map rectangular box is greater than or equal to a first preset ratio of the number of text boxes within the reference speech text box, determining that the first heat map rectangular box is a first special speech text box.

[0068] The preset multiple range is a range that indicates that the second height is close to the first height, for example, it may be 0.8 to 1.2.

[0069] Calculate the first height of the reference dialogue text box, and calculate the second height and the first center point height of each remaining text box; since the remaining text boxes may include non-didactic text boxes, it is necessary to screen the remaining text boxes to screen out the remaining dialogue text boxes whose second height is within the preset multiple range of the first height and whose first center point height is within the height range of the reference dialogue text box from all the remaining text boxes, and use the screened text boxes as the first target text boxes; generate the first heat map rectangular box corresponding to each first target text box; compare the generated first heat map rectangular box with the reference dialogue text box, if two first heat map rectangular boxes arranged on the left and right are included in the same frame image, the y-axis position information (center point height) of the first heat map rectangular box is close to the y-axis position information of the reference dialogue text box, and the number of remaining text boxes located in the first heat map rectangular box is greater than or equal to the first preset ratio of the number of text boxes located in the reference dialogue text box, determine that the first heat map rectangular box is the first special dialogue text box. Among them, the first preset ratio is a small value, such as 5%, etc., which can be obtained by statistics based on the video of the existing text box of this type.

[0070] Exemplarily, the first heat map rectangular frame can be generated in the following manner: create a second two-dimensional array, the initial value of each element in the second two-dimensional array is 0, and the size of the second two-dimensional array is the same as the size of the frame-drawing image; count the first target text box in each frame-drawing image in turn, and when the first target text box appears, add 1 to the element at the corresponding position of the first target text box in the second two-dimensional array based on the original value; obtain the largest value in the second two-dimensional array as the second highest heat value; retain the element value whose element value is greater than the preset proportion of the second highest heat value, set the remaining elements to 0, and obtain the second reserved area; determine the minimum circumscribed rectangle of the second reserved area, and obtain the first heat map rectangular frame. The preset proportion can be, for example, 50%.

[0071] By screening out the first target text box from the remaining text boxes, and then generating a first heat map rectangular box based on the first target text box, if the first heat map rectangular box meets the conditions of the first special lines text box, the first heat map rectangular box is determined to be the first special lines text box. The first special lines text box obtained in this way is more accurate and can further improve the recall rate of the lines.

[0072] On the basis of the above technical solution, the second special speech text box is determined from the other text boxes according to the reference speech text box, including: determining the third height of each of the other text boxes, determining the second center point height of each of the other text boxes, and determining the third center point height of the reference speech text box; obtaining other text boxes whose third height is within a preset multiple range of the first height and whose difference between the third center point height and the second center point height is within a preset height range from all the other text boxes as second target text boxes; generating second heat map rectangular boxes corresponding to each second target text box; if in the same frame image, the difference between the center point height of the second heat map rectangular box and the third center point height is within the preset height range, and the difference between the center point horizontal coordinate of the second heat map rectangular box and the center point horizontal coordinate of the reference speech text box is within the preset difference range, and the number of other text boxes located in the second heat map rectangular box is greater than or equal to a first preset ratio of the number of text boxes located in the reference speech text box, the second heat map rectangular box is determined to be a second special speech text box.

[0073] Among them, the second special dialogue text box to be determined should be distributed in an up-down relationship with the reference dialogue text box, so the preset height range is related to the first height of the reference dialogue text box, and the preset height range H can be expressed as an αh range, where h is the first height of the reference dialogue text box, and α can be positive or negative. When α is positive, α has a value range of [1.0, 1.5], and when α is negative, α has a value range of [-1.0, -1.5]. When the second special dialogue text box to be determined is located above the reference dialogue text box, the preset height range is a negative range; when the second special dialogue text box to be determined is located below the reference dialogue text box, the preset height range is a positive range.

[0074] Calculate the third height and the second center point height of each other text box, determine the third center point height of the benchmark dialogue text box, select other text boxes whose third height is within the preset multiple range of the first height and whose difference between the third center point height and the second center point height is within the preset height range from all other text boxes, and use the selected other text boxes as second target text boxes; generate second heat map rectangular boxes corresponding to each second target text box; compare the generated second heat map rectangular box with the benchmark dialogue text box, if in the same frame image, the difference between the center point height of the second heat map rectangular box and the third center point height is within the preset height range, and the difference between the horizontal coordinate of the center point of the second heat map rectangular box and the horizontal coordinate of the center point of the benchmark dialogue text box is within the preset difference range, that is, the second heat map rectangular box is located above or below the benchmark dialogue text box and is close to the height of the benchmark dialogue text box, and the number of other text boxes located in the second heat map rectangular box is greater than or equal to the first preset ratio of the number of text boxes located in the benchmark dialogue text box, determine that the second heat map rectangular box is a second special dialogue text box. The first preset ratio is a smaller value, such as 5%, etc., and can be obtained by counting videos of existing text boxes of this type.

[0075] Exemplarily, the second heat map rectangular frame can be generated in the following manner: create a third two-dimensional array, the initial value of each element in the third two-dimensional array is 0, and the size of the third two-dimensional array is the same as the size of the frame-drawing image; count the second target text boxes in each frame-drawing image in turn, and when the second target text box appears, add 1 to the element at the corresponding position of the second target text box in the third two-dimensional array based on the original value; obtain the largest value in the third two-dimensional array as the third highest heat value; retain the element value whose element value is greater than the preset ratio of the third highest heat value, set the remaining elements to 0, and obtain the third reserved area; determine the minimum circumscribed rectangle of the third reserved area to obtain the second heat map rectangular frame. Among them, the preset ratio can be, for example, 50%.

[0076] By filtering out the second target text box from other text boxes, and then generating a second heat map rectangular box based on the second target text box, if the second heat map rectangular box meets the conditions of the second special lines text box, the second heat map rectangular box is determined to be the second special lines text box. The second special lines text box obtained in this way is more accurate and can further improve the recall rate of the lines.

[0077] In another embodiment of the present invention, the reference line type includes two lines of reference lines, the reference line text box includes a first line of reference line text box and a second line of reference line text box, and in the same frame image, the first line of reference line text box is located above the second line of reference line text box;

[0078] The method of counting the remaining text boxes in the text box outside the reference text box according to the reference line type, and determining the special line text boxes of the plurality of frame-sampling images according to the statistical result, comprises:

[0079] According to the second row of reference dialogue text boxes, in the same frame-sampling image, the remaining text boxes located below the second row of reference dialogue text boxes are counted, and the special dialogue text boxes of the plurality of frame-sampling images are determined according to the statistical results; or,

[0080] According to the first row of reference dialogue text boxes, statistics are taken on the remaining text boxes located above the first row of reference dialogue text boxes in the same frame-sampling image, and special dialogue text boxes of the plurality of frame-sampling images are determined according to the statistical results.

[0081] In the speech layout template determined based on the pre-collected speech layout situations, for the case of two lines of reference speech, three lines of speech may occasionally appear, that is, there is another line of speech above or below the two lines of reference speech.

[0082] Figure 3a is an example diagram of two lines of reference lines in an embodiment of the present invention, Figure 3b 2 is an example diagram of a special dialogue text box in the case of two lines of reference dialogue in an embodiment of the present invention. Figure 3a and Figure 3b As shown, Figure 3a It is a two-line baseline dialogue, which is a dialogue layout that appears more frequently in a video with a two-line dialogue layout. Figure 3b It is a three-line layout situation that occasionally occurs in the case of two lines of basic lines, that is, there is another line of lines above or below the two lines of basic lines.

[0083] like Figure 3a and Figure 3bAs shown, the y-axis position information of the special line text box and the y-axis position of the base line text box are in a top-to-bottom layout, and the x-axis position information of the special line text box is close to the x-axis position information of the base line text box. For example, the three-line line layout may appear in Chinese-English parallel lines. If the English lines are longer, there will be one more line of lines than the base line, and the extra line of lines is close to the text box height of the English lines in the base line text box and is located above or below the base line. At this time, the special line text box can be determined based on the second line of the English line text box or the first line of the base line text box.

[0084] In an optional embodiment, the special line text box is located below the second row of baseline line text boxes. In this case, the second row of baseline line text boxes are used as a reference, and the remaining text boxes below the second row of baseline line text boxes are counted in the same framed image. The remaining text boxes that are close in height to the second row of baseline line text boxes and located below the second row of baseline line text boxes are counted, and the special line text boxes of multiple framed images are determined based on the statistical results.

[0085] In another optional embodiment, the special line text box is located above the first row of baseline line text boxes. In this case, the first row of baseline line text boxes is used as a reference, and the remaining text boxes located above the first row of baseline line text boxes are counted in the same framed image. The remaining text boxes that are close in height to the first row of baseline line text boxes and located above the first row of baseline line text boxes are counted, and the special line text boxes of multiple framed images are determined based on the statistical results.

[0086] By fully considering the special lines located below or above the baseline lines in the case of two baseline lines, the recall rate of the lines can be further improved.

[0087] In an optional implementation, the step of counting the remaining text boxes located below the second line of reference text boxes in the same frame-sampling image based on the second line of reference text boxes, and determining the special text boxes of the multiple frame-sampling images based on the statistical results, includes:

[0088] Determine a fourth height and a fourth center point height of the second line reference line text box, and determine a fifth height and a fifth center point height of each of the remaining text boxes;

[0089] Acquire, from the remaining text boxes, the remaining text boxes whose fifth height is within the preset multiple range of the fourth height and whose difference between the fifth center point height and the fourth center point height is within the preset height range, as the third target text box;

[0090] Generate a third heat map rectangular frame corresponding to each of the third target text frames;

[0091] If in the same frame image, the difference between the center point height of the third heat map rectangular box and the fourth center point height is within the preset height range, and the difference between the horizontal coordinate of the center point of the third heat map rectangular box and the horizontal coordinate of the center point of the second row of reference dialogue text boxes is within the preset difference range, and the number of remaining text boxes within the third heat map rectangular box is greater than or equal to a first preset ratio of the number of text boxes within the reference dialogue text box, the three heat map rectangular boxes are determined to be the special dialogue text boxes.

[0092] Among them, the special dialogue text box to be determined should be distributed in an up-down relationship with the second-line reference dialogue text box, so the preset height range is related to the first height of the second-line reference dialogue text box, and the preset height range H can be expressed as an αh range, where h is the fourth height of the second-line reference dialogue text box, and the α value range can be [1.0, 1.5]. The preset multiple range is a range that represents the value of the fifth height close to the fourth height, for example, it can be 0.8 to 1.2, etc.

[0093] Calculate the fourth height and the fourth center point height of the second line of the reference text box, and calculate the fifth height and the fifth center point height of each remaining text box; select the remaining text boxes whose fifth height is within the preset multiple range of the fourth height (the fifth height is close to the fourth height) and whose difference between the fifth center point height and the fourth center point height is within the preset height range (located below the second line of the reference text box), and use the selected remaining text boxes as third target text boxes; generate third heat map rectangular boxes corresponding to each third target text box; compare the generated third heat map rectangular boxes with the second line of the reference text box; This frame is compared. If, in the same frame image, the difference between the height of the center point of the third heat map rectangular frame and the height of the fourth center point is within the preset height range, and the difference between the horizontal coordinate of the center point of the third heat map rectangular frame and the horizontal coordinate of the center point of the second line of the reference line text frame is within the preset difference range, that is, the third heat map rectangular frame is located below the second line of the reference line text frame and is close to the height of the second line of the reference line text frame, and the number of remaining text frames in the third heat map rectangular frame is greater than or equal to the first preset ratio of the number of text frames in the reference line text frame, the third heat map rectangular frame is determined to be a special line text frame. Among them, the first preset ratio is a smaller value, such as 5%, etc., which can be obtained by statistics based on videos with existing text frames of this type.

[0094] Exemplarily, the third heat map rectangular frame can be generated in the following manner: create a fourth two-dimensional array, the initial value of each element in the fourth two-dimensional array is 0, and the size of the fourth two-dimensional array is the same as the size of the frame-drawing image; count the third target text boxes in each frame-drawing image in turn, and when the third target text box appears, add 1 to the original value of the element at the corresponding position of the third target text box in the fourth two-dimensional array; obtain the largest value in the fourth two-dimensional array as the fourth highest heat value; retain the element value whose element value is greater than the preset ratio of the fourth highest heat value, set the remaining elements to 0, and obtain the fourth reserved area; determine the minimum circumscribed rectangle of the fourth reserved area, and obtain the third heat map rectangular frame. Among them, the preset ratio can be, for example, 50%.

[0095] By filtering out the third target text box from the remaining text boxes, and then generating a third heat map rectangular box based on the third target text box, if the third heat map rectangular box meets the condition of being a special line text box located below the second row of baseline line text boxes, the third heat map rectangular box is determined to be a special line text box. The special line text box obtained in this way is more accurate and can further improve the recall rate of the lines.

[0096] In another optional implementation, the method of counting the remaining text boxes located above the first line of reference text boxes in the same frame-sampling image based on the first line of reference text boxes, and determining the special text boxes of the multiple frame-sampling images based on the statistical results, includes:

[0097] Determine the sixth height and the sixth center point height of the first line of reference lines text box, and determine the fifth height and the fifth center point height of each of the remaining text boxes;

[0098] Acquire, from the remaining text boxes, the remaining text boxes whose fifth height is within the preset multiple range of the sixth height and whose difference between the sixth center point height and the fifth center point height is within the preset height range, as the fourth target text box;

[0099] Generate a fourth heat map rectangular frame corresponding to each of the fourth target text frames;

[0100] If in the same frame image, the difference between the center point height of the fourth heat map rectangular box and the sixth center point height is within the preset height range, and the difference between the horizontal coordinate of the center point of the fourth heat map rectangular box and the horizontal coordinate of the center point of the first row of reference dialogue text boxes is within the preset difference range, and the number of remaining text boxes within the fourth heat map rectangular box is greater than or equal to a first preset ratio of the number of text boxes within the reference dialogue text box, the fourth heat map rectangular box is determined to be the special dialogue text box.

[0101] Calculate the sixth height and the sixth center point height of the first line of the reference text box, and calculate the fifth height and the fifth center point height of each remaining text box; select the remaining text boxes whose fifth height is within the preset multiple range of the sixth height (the fifth height is close to the sixth height) and whose difference between the sixth center point height and the fifth center point height is within the preset height range (located above the first line of the reference text box), and use the selected remaining text boxes as fourth target text boxes; generate fourth heat map rectangular boxes corresponding to each fourth target text box; compare the generated fourth heat map rectangular boxes with the first line of the reference text box; This frame is compared. If, in the same frame image, the difference between the height of the center point of the fourth heat map rectangular frame and the height of the sixth center point is within the preset height range, and the difference between the horizontal coordinate of the center point of the fourth heat map rectangular frame and the horizontal coordinate of the center point of the first line of the reference line text frame is within the preset difference range, that is, the fourth heat map rectangular frame is located above the first line of the reference line text frame and is close to the height of the first line of the reference line text frame, and the number of remaining text frames in the fourth heat map rectangular frame is greater than or equal to the first preset ratio of the number of text frames in the reference line text frame, the fourth heat map rectangular frame is determined to be a special line text frame. Among them, the first preset ratio is a smaller value, such as 5%, etc., which can be obtained by statistics based on videos with existing text frames of this type.

[0102] Exemplarily, the fourth heat map rectangular frame can be generated in the following manner: create a fifth two-dimensional array, the initial value of each element in the fifth two-dimensional array is 0, and the size of the fifth two-dimensional array is the same as the size of the frame-drawing image; count the fourth target text box in each frame-drawing image in turn, and when the fourth target text box appears, add 1 to the element at the corresponding position of the fourth target text box in the fifth two-dimensional array on the basis of the original value; obtain the largest value in the fifth two-dimensional array as the fifth highest heat value; retain the element value whose element value is greater than the preset ratio of the fifth highest heat value, set the remaining elements to 0, and obtain the fifth reserved area; determine the minimum circumscribed rectangle of the fifth reserved area, and obtain the fourth heat map rectangular frame. Among them, the preset ratio can be, for example, 50%.

[0103] By screening out the fourth target text box from the remaining text boxes, and then generating a fourth heat map rectangular box based on the fourth target text box, if the fourth heat map rectangular box meets the condition of being a special line text box located above the first row of baseline line text boxes, the fourth heat map rectangular box is determined to be a special line text box. The special line text box obtained in this way is more accurate and can further improve the recall rate of the lines.

[0104] In one embodiment of the present invention, the special line text box of the multiple frame-by-frame images is determined in the text box according to the video type and the baseline line text box, that is, the above-mentioned step 140, including: when the video type is a variety show type, obtaining the remaining text boxes outside the baseline line text box from the text box; generating a fifth hot map rectangular box for each of the remaining text boxes; if the number of remaining text boxes in the fifth hot map rectangular box in each of the frame-by-frame images is greater than or equal to a second preset ratio of the number of text boxes in the baseline line text box, determining that the fifth hot map rectangular box is the special line text box.

[0105] Figure 4a is an example diagram of reference lines in a variety show type video in an embodiment of the present invention, Figure 4b is an example diagram of a special dialogue text box in a variety show type video in an embodiment of the present invention. Figure 4a and Figure 4b As shown, in variety show videos, such as common music variety shows, the lines exist in one position, namely Figure 4a The base line text box shown, the lyrics appear in another position, namely Figure 4b The special lines text box shown.

[0106] When the video type is a variety show type, all the remaining text boxes outside the reference line text box are counted to generate the fifth special map rectangular box of each remaining text box, and if the number of the remaining text boxes in the fifth heat map rectangular box in all the extracted frame images is greater than or equal to the second preset ratio of the number of text boxes in the reference line text box, the fifth heat map rectangular box is determined to be a special line text box. The second preset ratio is greater than the first preset ratio, for example, the second preset ratio can be 20%.

[0107] Exemplarily, the fifth heat map rectangular frame can be generated in the following manner: create a sixth two-dimensional array, the initial value of each element in the sixth two-dimensional array is 0, and the size of the sixth two-dimensional array is the same as the size of the frame-drawing image; count the remaining text boxes in each frame-drawing image in turn, and when the remaining text boxes appear, add 1 to the original value of the element at the corresponding position of the remaining text box in the sixth two-dimensional array; obtain the largest value in the sixth two-dimensional array as the sixth highest heat value; retain the element value whose element value is greater than the preset ratio of the sixth highest heat value, set the remaining elements to 0, and obtain the sixth reserved area; determine the minimum circumscribed rectangle of the sixth reserved area, and obtain the fifth heat map rectangular frame. Among them, the preset ratio can be, for example, 50%.

[0108] When the video type is a variety show type, since different types of lines (such as general lines and lyrics) have different positions, special line text boxes can be determined by performing heat map statistics on all remaining text boxes. Text boxes for different types of lines can be obtained, which can further improve the recall rate of lines in variety show type videos.

[0109] Figure 5 is a structural diagram of a video dialogue extraction device provided by an embodiment of the present invention. Figure 5 As shown, the video lines extraction device includes:

[0110] An acquisition module 510 is used to acquire a video to be processed and a video type of the video to be processed;

[0111] A frame extraction module 520 is used to perform frame extraction processing on the video to be processed to obtain multiple frame extracted images of the video to be processed;

[0112] A reference dialogue frame determination module 530 is used to determine the reference dialogue text frame of the plurality of frame-by-frame images according to the text frame in each frame-by-frame image;

[0113] A special dialogue frame determination module 540 is used to determine the special dialogue text frame of the plurality of frame-sampling images in the text frame according to the video type and the reference dialogue text frame;

[0114] The dialogue extraction module 550 is used to extract the dialogue located in the reference dialogue text box and the special dialogue text box from the plurality of frame-sampling images.

[0115] Optionally, the special dialogue frame determination module includes:

[0116] A dialogue type determination unit, configured to determine a reference dialogue type of the video to be processed according to the reference dialogue text box when the video type is a film or TV drama type;

[0117] The first special dialogue frame determining unit is used to count the remaining text frames in the text frame that are outside the reference dialogue text frame according to the reference dialogue type, and determine the special dialogue text frames of the multiple frame-sampling images according to the statistical results.

[0118] Optionally, the reference line type includes a line of reference lines; the special line text box includes a first special line text box and / or a second special line text box;

[0119] The first special dialogue frame determining unit includes:

[0120] A first special speech frame determining subunit is used to determine a first special speech text frame from the remaining text frames according to the reference speech text frame, wherein in the same frame image, two first special speech text frames are arranged on the left and right, the height of the first special speech text frame is within a preset multiple range of the height of the reference speech text frame, and the center point height of the first special speech text frame is within the height range of the reference speech text frame;

[0121] Other text box acquisition subunit, used for acquiring other text boxes outside the first special line text box from the remaining text boxes;

[0122] A second special dialogue frame determination subunit is used to determine a second special dialogue text frame from the other text frames based on the reference dialogue text frame, wherein the second special dialogue text frame and the reference dialogue text frame are included in the same frame image, the height of the second special dialogue text frame is within a preset multiple range of the height of the reference dialogue text frame, the absolute value of the difference between the height of the center point of the second special dialogue text frame and the height of the center point of the reference dialogue text frame is within a preset height range, and the difference between the horizontal coordinate of the center point of the second special dialogue text frame and the horizontal coordinate of the center point of the reference dialogue text frame is within a preset difference range.

[0123] Optionally, the first special dialogue frame determining subunit is specifically used for:

[0124] Determine a first height of the reference dialogue text box, determine a second height of each of the remaining text boxes, and determine a first center point height of each of the remaining text boxes;

[0125] From all the remaining text boxes, the remaining text boxes whose second height is within a preset multiple of the first height and whose first center point height is within a height range of the reference dialogue text box are obtained as first target text boxes;

[0126] Generate a first heat map rectangular frame corresponding to each of the first target text frames;

[0127] If the same frame image includes two first heat map rectangular frames arranged on the left and right, the height of the center point of the first heat map rectangular frame is within the height range of the benchmark dialogue text frame, and the number of remaining text frames within the first heat map rectangular frame is greater than or equal to a first preset ratio of the number of text frames within the benchmark dialogue text frame, the first heat map rectangular frame is determined to be a first special dialogue text frame.

[0128] Optionally, the second special dialogue frame determination subunit is specifically used for:

[0129] Determine the third height of each of the other text boxes, determine the second center point height of each of the other text boxes, and determine the third center point height of the reference dialogue text box;

[0130] Acquire, from all the other text boxes, other text boxes whose third height is within a preset multiple range of the first height and whose difference between the third center point height and the second center point height is within a preset height range, as second target text boxes;

[0131] Generate a second heat map rectangular frame corresponding to each of the second target text frames;

[0132] If in the same frame image, the difference between the center point height of the second heat map rectangular box and the third center point height is within the preset height range, and the difference between the horizontal coordinate of the center point of the second heat map rectangular box and the horizontal coordinate of the center point of the benchmark dialogue text box is within the preset difference range, and the number of other text boxes within the second heat map rectangular box is greater than or equal to a first preset ratio of the number of text boxes within the benchmark dialogue text box, the second heat map rectangular box is determined to be a second special dialogue text box.

[0133] Optionally, the reference line type includes two lines of reference lines, the reference line text box includes a first line of reference line text box and a second line of reference line text box, and in the same frame-snap image, the first line of reference line text box is located above the second line of reference line text box;

[0134] The first special dialogue frame determining unit includes:

[0135] a third special dialogue frame determination subunit, configured to count the remaining text frames located below the second line of reference dialogue text frames in the same frame-sampling image according to the second line of reference dialogue text frames, and determine the special dialogue text frames of the plurality of frame-sampling images according to the statistical results; or

[0136] The fourth special dialogue frame determination subunit is used to count the remaining text frames located above the first row of reference dialogue text frames in the same frame-sampling image based on the first row of reference dialogue text frames, and determine the special dialogue text frames of the multiple frame-sampling images based on the statistical results.

[0137] Optionally, the third special dialogue frame determination subunit is specifically used for:

[0138] Determine a fourth height and a fourth center point height of the second line reference line text box, and determine a fifth height and a fifth center point height of each of the remaining text boxes;

[0139] Acquire, from the remaining text boxes, the remaining text boxes whose fifth height is within the preset multiple range of the fourth height and whose difference between the fifth center point height and the fourth center point height is within the preset height range, as the third target text box;

[0140] Generate a third heat map rectangular frame corresponding to each of the third target text frames;

[0141] If in the same frame image, the difference between the center point height of the third heat map rectangular box and the fourth center point height is within the preset height range, and the difference between the horizontal coordinate of the center point of the third heat map rectangular box and the horizontal coordinate of the center point of the second row of reference dialogue text boxes is within the preset difference range, and the number of remaining text boxes within the third heat map rectangular box is greater than or equal to a first preset ratio of the number of text boxes within the reference dialogue text box, the three heat map rectangular boxes are determined to be the special dialogue text boxes.

[0142] Optionally, the fourth special dialogue frame determination subunit is specifically used for:

[0143] Determine the sixth height and the sixth center point height of the first line of reference lines text box, and determine the fifth height and the fifth center point height of each of the remaining text boxes;

[0144] Acquire, from the remaining text boxes, the remaining text boxes whose fifth height is within the preset multiple range of the sixth height and whose difference between the sixth center point height and the fifth center point height is within the preset height range, as the fourth target text box;

[0145] Generate a fourth heat map rectangular frame corresponding to each of the fourth target text frames;

[0146] If in the same frame image, the difference between the center point height of the fourth heat map rectangular box and the sixth center point height is within the preset height range, and the difference between the horizontal coordinate of the center point of the fourth heat map rectangular box and the horizontal coordinate of the center point of the first row of reference dialogue text boxes is within the preset difference range, and the number of remaining text boxes within the fourth heat map rectangular box is greater than or equal to a first preset ratio of the number of text boxes within the reference dialogue text box, the fourth heat map rectangular box is determined to be the special dialogue text box.

[0147] Optionally, the special dialogue frame determination module includes:

[0148] A remaining text box acquisition unit, configured to acquire remaining text boxes outside the reference line text box from the text box when the video type is a variety show type;

[0149] A heat map rectangular frame generating unit, used for generating a fifth heat map rectangular frame of each of the remaining text frames;

[0150] The second special dialogue frame determination unit is used to determine that the fifth heat map rectangular frame is the special dialogue text frame if the number of remaining text frames in each of the extracted frame images within the fifth heat map rectangular frame is greater than or equal to a second preset ratio of the number of text frames within the reference dialogue text frame.

[0151] The video dialogue extraction device provided in the present embodiment determines the base dialogue text box of multiple framed images according to the text box in each framed image, and then further determines the special dialogue text box of multiple framed images in the text box according to the video type and the base dialogue text box, and then extracts the lines located in the base dialogue text box and the special dialogue text box. In this way, the special layout of the lines is fully considered on the basis of the base dialogue text box to obtain the special dialogue text box. By combining the base dialogue text box and the special dialogue text box, the omission of lines can be avoided and the recall rate of video lines can be improved.

[0152] The embodiment of the present invention further provides an electronic device, such as Figure 6 As shown, it includes a processor 601, a communication interface 602, a memory 603 and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604.

[0153] Memory 603, used for storing computer programs;

[0154] The processor 601 is used to execute the program stored in the memory 603 to implement the following steps:

[0155] Obtain a video to be processed and a video type of the video to be processed;

[0156] Performing frame extraction processing on the video to be processed to obtain multiple frame extracted images of the video to be processed;

[0157] Determining the reference dialogue text frame of the plurality of frame-by-frame images according to the text frame in each of the frame-by-frame images;

[0158] Determining, in the text box, a special text box of the plurality of frame-sampling images according to the video type and the reference text box of the text box;

[0159] The lines located in the reference line text box and the special line text box are extracted from the multiple frame-sampling images.

[0160] Optionally, determining the special dialogue text box of the plurality of frame-sampling images in the text box according to the video type and the reference dialogue text box includes:

[0161] When the video type is a film or TV drama type, determining a reference dialogue type of the video to be processed according to the reference dialogue text box;

[0162] According to the reference line type, statistics are collected on the remaining text boxes in the text box that are outside the reference line text box, and special line text boxes in the plurality of frame-sampling images are determined according to the statistical results.

[0163] Optionally, the reference line type includes a line of reference lines; the special line text box includes a first special line text box and / or a second special line text box;

[0164] The method of counting the remaining text boxes in the text box outside the reference text box according to the reference line type, and determining the special line text boxes of the plurality of frame-sampling images according to the statistical result, comprises:

[0165] According to the reference speech text frame, a first special speech text frame is determined from the remaining text frames, wherein in the same extracted frame image, two first special speech text frames are arranged on the left and right, the height of the first special speech text frame is within a preset multiple range of the height of the reference speech text frame, and the center point height of the first special speech text frame is within the height range of the reference speech text frame;

[0166] Acquire other text boxes outside the first special lines text box from the remaining text boxes;

[0167] According to the reference speech text box, a second special speech text box is determined from the other text boxes, wherein the second special speech text box and the reference speech text box are included in the same frame-drawing image, the height of the second special speech text box is within a preset multiple range of the height of the reference speech text box, the absolute value of the difference between the height of the center point of the second special speech text box and the height of the center point of the reference speech text box is within a preset height range, and the difference between the horizontal coordinate of the center point of the second special speech text box and the horizontal coordinate of the center point of the reference speech text box is within a preset difference range.

[0168] Optionally, determining the first special speech text box from the remaining text boxes according to the reference speech text box includes:

[0169] Determine a first height of the reference dialogue text box, determine a second height of each of the remaining text boxes, and determine a first center point height of each of the remaining text boxes;

[0170] From all the remaining text boxes, the remaining text boxes whose second height is within a preset multiple of the first height and whose first center point height is within a height range of the reference dialogue text box are obtained as first target text boxes;

[0171] Generate a first heat map rectangular frame corresponding to each of the first target text frames;

[0172] If the same frame image includes two first heat map rectangular frames arranged on the left and right, the height of the center point of the first heat map rectangular frame is within the height range of the benchmark dialogue text frame, and the number of remaining text frames within the first heat map rectangular frame is greater than or equal to a first preset ratio of the number of text frames within the benchmark dialogue text frame, the first heat map rectangular frame is determined to be a first special dialogue text frame.

[0173] Optionally, determining a second special dialogue text box from the other text boxes according to the reference dialogue text box includes:

[0174] Determine the third height of each of the other text boxes, determine the second center point height of each of the other text boxes, and determine the third center point height of the reference dialogue text box;

[0175] Acquire, from all the other text boxes, other text boxes whose third height is within a preset multiple range of the first height and whose difference between the third center point height and the second center point height is within a preset height range, as second target text boxes;

[0176] Generate a second heat map rectangular frame corresponding to each of the second target text frames;

[0177] If in the same frame image, the difference between the center point height of the second heat map rectangular box and the third center point height is within the preset height range, and the difference between the horizontal coordinate of the center point of the second heat map rectangular box and the horizontal coordinate of the center point of the benchmark dialogue text box is within the preset difference range, and the number of other text boxes within the second heat map rectangular box is greater than or equal to a first preset ratio of the number of text boxes within the benchmark dialogue text box, the second heat map rectangular box is determined to be a second special dialogue text box.

[0178] Optionally, the reference line type includes two lines of reference lines, the reference line text box includes a first line of reference line text box and a second line of reference line text box, and in the same frame-snap image, the first line of reference line text box is located above the second line of reference line text box;

[0179] The method of counting the remaining text boxes in the text box outside the reference text box according to the reference line type, and determining the special line text boxes of the plurality of frame-sampling images according to the statistical result, comprises:

[0180] According to the second row of reference dialogue text boxes, in the same frame-sampling image, the remaining text boxes located below the second row of reference dialogue text boxes are counted, and the special dialogue text boxes of the plurality of frame-sampling images are determined according to the statistical results; or,

[0181] According to the first row of reference dialogue text boxes, statistics are taken on the remaining text boxes located above the first row of reference dialogue text boxes in the same frame-sampling image, and special dialogue text boxes of the plurality of frame-sampling images are determined according to the statistical results.

[0182] Optionally, the step of counting the remaining text boxes located below the second line of reference text boxes in the same frame-sampling image according to the second line of reference text boxes, and determining the special text boxes of the multiple frame-sampling images according to the statistical results, comprises:

[0183] Determine a fourth height and a fourth center point height of the second line reference line text box, and determine a fifth height and a fifth center point height of each of the remaining text boxes;

[0184] Acquire, from the remaining text boxes, the remaining text boxes whose fifth height is within the preset multiple range of the fourth height and whose difference between the fifth center point height and the fourth center point height is within the preset height range, as the third target text box;

[0185] Generate a third heat map rectangular frame corresponding to each of the third target text frames;

[0186] If in the same frame image, the difference between the center point height of the third heat map rectangular box and the fourth center point height is within the preset height range, and the difference between the horizontal coordinate of the center point of the third heat map rectangular box and the horizontal coordinate of the center point of the second row of reference dialogue text boxes is within the preset difference range, and the number of remaining text boxes within the third heat map rectangular box is greater than or equal to a first preset ratio of the number of text boxes within the reference dialogue text box, the three heat map rectangular boxes are determined to be the special dialogue text boxes.

[0187] Optionally, the step of counting the remaining text boxes located above the first line of reference text boxes in the same frame-sampling image based on the first line of reference text boxes, and determining the special text boxes of the multiple frame-sampling images based on the statistical results, comprises:

[0188] Determine the sixth height and the sixth center point height of the first line of reference lines text box, and determine the fifth height and the fifth center point height of each of the remaining text boxes;

[0189] Acquire, from the remaining text boxes, the remaining text boxes whose fifth height is within the preset multiple range of the sixth height and whose difference between the sixth center point height and the fifth center point height is within the preset height range, as the fourth target text box;

[0190] Generate a fourth heat map rectangular frame corresponding to each of the fourth target text frames;

[0191] If in the same frame image, the difference between the center point height of the fourth heat map rectangular box and the sixth center point height is within the preset height range, and the difference between the horizontal coordinate of the center point of the fourth heat map rectangular box and the horizontal coordinate of the center point of the first row of reference dialogue text boxes is within the preset difference range, and the number of remaining text boxes within the fourth heat map rectangular box is greater than or equal to a first preset ratio of the number of text boxes within the reference dialogue text box, the fourth heat map rectangular box is determined to be the special dialogue text box.

[0192] Optionally, determining the special dialogue text box of the plurality of frame-sampling images in the text box according to the video type and the reference dialogue text box includes:

[0193] When the video type is a variety show type, obtaining the remaining text boxes outside the reference line text box from the text box;

[0194] Generating a fifth heat map rectangular frame for each of the remaining text frames;

[0195] If the number of remaining text boxes in the fifth heat map rectangular box in each of the extracted frame images is greater than or equal to a second preset ratio of the number of text boxes in the reference line text box, the fifth heat map rectangular box is determined to be the special line text box.

[0196] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0197] The communication interface is used for communication between the above electronic device and other devices.

[0198] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0199] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0200] In another embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium stores instructions. When the instructions are executed on a computer, the computer executes the video dialogue extraction method described in any of the above embodiments.

[0201] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes the video dialogue extraction method described in any one of the above embodiments.

[0202] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0203] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0204] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0205] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A video dialogue extraction method, characterized in that: include: Obtain a video to be processed and the video type of the video to be processed; Performing frame extraction processing on the video to be processed to obtain multiple frame extracted images of the video to be processed; Determining the reference dialogue text frame of the plurality of frame-by-frame images according to the text frame in each of the frame-by-frame images; Determining, in the text box, a special text box of the plurality of frame-sampling images according to the video type and the reference text box of the text box; The lines located in the reference line text box and the special line text box are extracted from the multiple frame-sampling images.

2. The method according to claim 1, characterized in that The determining, in the text box, the special text box of the plurality of frame-sampling images according to the video type and the reference text box of the text box comprises: When the video type is a film or TV drama type, determining a reference dialogue type of the video to be processed according to the reference dialogue text box; According to the reference line type, statistics are collected on the remaining text boxes in the text box that are outside the reference line text box, and special line text boxes in the plurality of frame-sampling images are determined according to the statistical results.

3. The method according to claim 2, characterized in that The reference line type includes a line of reference lines; the special line text box includes a first special line text box and / or a second special line text box; The method of counting the remaining text boxes in the text box outside the reference text box according to the reference line type, and determining the special line text boxes of the plurality of frame-sampling images according to the statistical result, comprises: According to the reference speech text frame, a first special speech text frame is determined from the remaining text frames, wherein in the same extracted frame image, two first special speech text frames are arranged on the left and right, the height of the first special speech text frame is within a preset multiple range of the height of the reference speech text frame, and the center point height of the first special speech text frame is within the height range of the reference speech text frame; Acquire other text boxes outside the first special lines text box from the remaining text boxes; According to the reference speech text box, a second special speech text box is determined from the other text boxes, wherein the second special speech text box and the reference speech text box are included in the same frame-drawing image, the height of the second special speech text box is within a preset multiple range of the height of the reference speech text box, the absolute value of the difference between the height of the center point of the second special speech text box and the height of the center point of the reference speech text box is within a preset height range, and the difference between the horizontal coordinate of the center point of the second special speech text box and the horizontal coordinate of the center point of the reference speech text box is within a preset difference range.

4. The method according to claim 3, characterized in that The step of determining a first special speech text frame from the remaining text frames according to the reference speech text frame includes: Determine a first height of the reference dialogue text box, determine a second height of each of the remaining text boxes, and determine a first center point height of each of the remaining text boxes; From all the remaining text boxes, the remaining text boxes whose second height is within a preset multiple of the first height and whose first center point height is within a height range of the reference dialogue text box are obtained as first target text boxes; Generate a first heat map rectangular frame corresponding to each of the first target text frames; If the same frame image includes two first heat map rectangular frames arranged on the left and right, the height of the center point of the first heat map rectangular frame is within the height range of the benchmark dialogue text frame, and the number of remaining text frames within the first heat map rectangular frame is greater than or equal to a first preset ratio of the number of text frames within the benchmark dialogue text frame, the first heat map rectangular frame is determined to be a first special dialogue text frame.

5. The method according to claim 3, characterized in that: The step of determining a second special speech text frame from the other text frames according to the reference speech text frame includes: Determine the third height of each of the other text boxes, determine the second center point height of each of the other text boxes, and determine the third center point height of the reference dialogue text box; Acquire, from all the other text boxes, other text boxes whose third height is within a preset multiple range of the first height and whose difference between the third center point height and the second center point height is within a preset height range, as second target text boxes; Generate a second heat map rectangular frame corresponding to each of the second target text frames; If in the same frame image, the difference between the center point height of the second heat map rectangular box and the third center point height is within the preset height range, and the difference between the horizontal coordinate of the center point of the second heat map rectangular box and the horizontal coordinate of the center point of the benchmark dialogue text box is within the preset difference range, and the number of other text boxes within the second heat map rectangular box is greater than or equal to a first preset ratio of the number of text boxes within the benchmark dialogue text box, the second heat map rectangular box is determined to be a second special dialogue text box.

6. The method according to claim 2, characterized in that The reference line type includes two lines of reference lines, the reference line text box includes a first line of reference line text box and a second line of reference line text box, and in the same frame image, the first line of reference line text box is located above the second line of reference line text box; The method of counting the remaining text boxes in the text box outside the reference text box according to the reference line type, and determining the special line text boxes of the plurality of frame-sampling images according to the statistical result, comprises: According to the second row of reference dialogue text boxes, counting the remaining text boxes located below the second row of reference dialogue text boxes in the same frame-sampling image, and determining the special dialogue text boxes of the plurality of frame-sampling images according to the counting result; or, According to the first row of reference dialogue text boxes, statistics are taken on the remaining text boxes located above the first row of reference dialogue text boxes in the same frame-sampling image, and special dialogue text boxes of the plurality of frame-sampling images are determined according to the statistical results.

7. The method according to claim 6, characterized in that The method of counting the remaining text boxes located below the second line of reference text boxes in the same frame-sampling image according to the second line of reference text boxes, and determining the special text boxes of the multiple frame-sampling images according to the statistical results, comprises: Determine a fourth height and a fourth center point height of the second line reference line text box, and determine a fifth height and a fifth center point height of each of the remaining text boxes; Acquire, from the remaining text boxes, the remaining text boxes whose fifth height is within the preset multiple range of the fourth height and whose difference between the fifth center point height and the fourth center point height is within the preset height range, as the third target text box; Generate a third heat map rectangular frame corresponding to each of the third target text frames; If in the same frame image, the difference between the center point height of the third heat map rectangular box and the fourth center point height is within the preset height range, and the difference between the horizontal coordinate of the center point of the third heat map rectangular box and the horizontal coordinate of the center point of the second row of reference dialogue text boxes is within the preset difference range, and the number of remaining text boxes within the third heat map rectangular box is greater than or equal to a first preset ratio of the number of text boxes within the reference dialogue text box, the three heat map rectangular boxes are determined to be the special dialogue text boxes.

8. The method according to claim 6, characterized in that The method of counting the remaining text boxes located above the first line of reference text boxes in the same frame-sampling image according to the first line of reference text boxes, and determining the special text boxes of the multiple frame-sampling images according to the statistical results, comprises: Determine the sixth height and the sixth center point height of the first line of reference lines text box, and determine the fifth height and the fifth center point height of each of the remaining text boxes; Acquire, from the remaining text boxes, the remaining text boxes whose fifth height is within the preset multiple range of the sixth height and whose difference between the sixth center point height and the fifth center point height is within the preset height range, as the fourth target text box; Generate a fourth heat map rectangular frame corresponding to each of the fourth target text frames; If in the same frame image, the difference between the center point height of the fourth heat map rectangular box and the sixth center point height is within the preset height range, and the difference between the horizontal coordinate of the center point of the fourth heat map rectangular box and the horizontal coordinate of the center point of the first row of reference dialogue text boxes is within the preset difference range, and the number of remaining text boxes within the fourth heat map rectangular box is greater than or equal to a first preset ratio of the number of text boxes within the reference dialogue text box, the fourth heat map rectangular box is determined to be the special dialogue text box.

9. The method according to claim 1, characterized in that: The step of determining the special dialogue text box of the plurality of frame-sampling images in the text box according to the video type and the reference dialogue text box includes: When the video type is a variety show type, obtaining the remaining text boxes outside the reference line text box from the text box; Generating a fifth heat map rectangular frame for each of the remaining text frames; If the number of remaining text boxes in the fifth heat map rectangular box in each of the extracted frame images is greater than or equal to a second preset ratio of the number of text boxes in the reference line text box, the fifth heat map rectangular box is determined to be the special line text box.

10. A video dialogue extraction device, characterized in that: include: An acquisition module, used for acquiring a video to be processed and a video type of the video to be processed; A frame extraction module is used to perform frame extraction processing on the video to be processed to obtain multiple frame extraction images of the video to be processed; A reference dialogue frame determination module, configured to determine reference dialogue text frames of the plurality of frame-by-frame images according to a text frame in each of the frame-by-frame images; A special dialogue frame determination module, used for determining the special dialogue text frame of the plurality of frame-sampling images in the text frame according to the video type and the reference dialogue text frame; The dialogue extraction module is used to extract the dialogue located in the reference dialogue text box and the special dialogue text box from the multiple frame-sampling images.

11. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 9 when executing a program stored in a memory.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images