Text type classification method, device, electronic device and storage medium

Abstract

The embodiment of the present invention provides a method and device for classifying text types, wherein the method comprises: obtaining a text mask image to be processed, the text mask image comprising at least one text region; determining a non-mask region of each text region for each text region, and dividing each text region into a first text mask region and a second text mask region based on each non-mask region; determining the text type of the first text mask region based on the positional relationship between the non-mask region and the first text mask region; determining the text type of the second text mask region based on the positional relationship between the non-mask region, the second text mask region and the text mask image. The embodiment of the present invention implements the classification of text types within a text region, and can solve the technical problem that after existing text filtering, the text cannot be distinguished as a line text and a name text, and achieves the effect of distinguishing each text type after text filtering.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a text type classification method and device, as well as an electronic device and a computer-readable storage medium. Background Art

[0002] With the rapid development of the Internet and computer technology, there are a large number of video contents on traditional TV stations and the Internet. In order to help users understand the video content, corresponding texts can usually be set for the video content, for example, displaying dialogue texts, name texts, and non-dialogue texts in variety shows.

[0003] When there are a lot of texts in the video content, the texts may interfere with each other. For example, non-dialogue texts in variety shows will interfere with dialogue texts and name texts. Therefore, it is necessary to filter the texts in the video content to display more valuable texts. After text filtering, dialogue texts and name texts can be retained, but related technologies cannot distinguish between dialogue texts and name texts. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide a method and device for classifying text types, as well as an electronic device and a computer-readable storage medium, which solve the problem of how to classify text types. The specific technical solution is as follows:

[0005] In the first aspect of the implementation of the present invention, a text type classification method is first provided, comprising: obtaining a text mask image to be processed, the text mask image comprising at least one text area; for each of the text areas, determining a non-mask area of ​​each of the text areas, and based on each of the non-mask areas, dividing each of the text areas into a first text mask area and a second text mask area; for each of the text areas, determining a text type of the first text mask area based on a positional relationship between the non-mask area and the first text mask area; for each of the text areas, determining a text type of the second text mask area based on a positional relationship between the non-mask area, the second text mask area and the text mask image.

[0006] Optionally, determining the non-mask area of ​​each of the text regions includes: acquiring pixel values ​​of each pixel in each of the text regions; and taking an area whose pixel values ​​satisfy a preset pixel condition as the non-mask area of ​​the corresponding text region.

[0007] Optionally, taking the area whose pixel values ​​satisfy a preset pixel condition as the non-mask area of ​​the corresponding text area includes: taking the area whose average pixel value is less than a preset pixel threshold as the non-mask area.

[0008] Optionally, determining the text type of the first text mask area based on the positional relationship between the non-mask area and the first text mask area includes: obtaining a first horizontal coordinate of the non-mask area and a second horizontal coordinate of the right border of the first text mask area; if the value of the second horizontal coordinate is smaller than the value of the first horizontal coordinate, classifying the first text mask area as the first text type; or, if the value of the second horizontal coordinate is greater than the value of the first horizontal coordinate, and the absolute value of the difference between the values ​​of the first horizontal coordinate and the second horizontal coordinate is smaller than a preset value, classifying the first text mask area as the first text type.

[0009] Optionally, determining the text type of the second text mask area based on the positional relationship between the non-mask area, the second text mask area and the text mask image includes: obtaining a first horizontal coordinate of the non-mask area, a third horizontal coordinate of the left border of the second text mask area and a fourth horizontal coordinate of the left border of the text mask image; if the value of the third horizontal coordinate is greater than the value of the first horizontal coordinate, classifying the second text mask area as the second text type; or, if the absolute value of the difference between the value of the third horizontal coordinate and the value of the fourth horizontal coordinate is greater than the absolute value of the difference between the value of the third horizontal coordinate and the value of the first horizontal coordinate, classifying the second text mask area as the second text type.

[0010] Optionally, the method further includes: obtaining a first horizontal coordinate of the non-mask area, a second horizontal coordinate of the right border of the first text mask area, a third horizontal coordinate of the left border of the second text mask area, and a fourth horizontal coordinate of the left border of the text mask image; if the first horizontal coordinate, the second horizontal coordinate, the third horizontal coordinate, and the fourth horizontal coordinate satisfy at least one of preset comparison conditions, classifying the first text mask area as the second text type, and classifying the second text mask area as the first text type; the preset comparison conditions include: the value of the second horizontal coordinate is greater than or equal to the value of the first horizontal coordinate, the value of the second horizontal coordinate is less than or equal to the value of the first horizontal coordinate, the absolute value of the difference between the value of the first horizontal coordinate and the value of the second horizontal coordinate is greater than or equal to a preset value, the value of the third horizontal coordinate is less than or equal to the value of the first horizontal coordinate, the absolute value of the difference between the value of the third horizontal coordinate and the value of the fourth horizontal coordinate is less than or equal to the absolute value of the difference between the value of the third horizontal coordinate and the value of the first horizontal coordinate.

[0011] Optionally, the obtaining of the text mask image to be processed includes: accumulating the pixel value of each pixel in the frame image to which each text area belongs to obtain a pixel value accumulation result; and normalizing and binarizing the pixel value accumulation result to obtain the text mask image.

[0012] In the second aspect of the implementation of the present invention, a text type classification device is also provided, including: an image acquisition module, used to acquire a text mask image to be processed, the text mask image containing at least one text area; a region segmentation module, used to determine the non-mask area of ​​each text area for each text area, and based on each non-mask area, divide each text area into a first text mask area and a second text mask area; a type classification module, used to determine the text type of the first text mask area for each text area based on the positional relationship between the non-mask area and the first text mask area; the type classification module is also used to determine the text type of the second text mask area for each text area based on the positional relationship between the non-mask area, the second text mask area and the text mask image.

[0013] Optionally, the region segmentation module includes: a pixel value acquisition module, used to obtain the pixel value of each pixel point in each of the text areas; and a non-mask area determination module, used to take the area whose pixel value meets the preset pixel condition as the non-mask area corresponding to the text area.

[0014] Optionally, the non-mask area determination module is used to take an area where the average value of the pixel values ​​is less than a preset pixel threshold as the non-mask area.

[0015] Optionally, the type classification module includes: a coordinate acquisition module, used to acquire a first horizontal coordinate of the non-mask area and a second horizontal coordinate of the right border of the first text mask area; a text type classification module, used to classify the first text mask area as the first text type if the value of the second horizontal coordinate is smaller than the value of the first horizontal coordinate; or, if the value of the second horizontal coordinate is greater than the value of the first horizontal coordinate, and the absolute value of the difference between the values ​​of the first horizontal coordinate and the second horizontal coordinate is smaller than a preset value, classify the first text mask area as the first text type.

[0016] Optionally, the coordinate acquisition module is also used to acquire a third horizontal coordinate of the left border of the second text mask area and a fourth horizontal coordinate of the left border of the text mask image; the text type classification module is also used to classify the second text mask area as the second text type if the value of the third horizontal coordinate is greater than the value of the first horizontal coordinate; or classify the second text mask area as the second text type if the absolute value of the difference between the value of the third horizontal coordinate and the value of the fourth horizontal coordinate is greater than the absolute value of the difference between the value of the third horizontal coordinate and the value of the first horizontal coordinate.

[0017] Optionally, the text type classification module is also used to classify the first text mask area as the second text type, and classify the second text mask area as the first text type if the first horizontal coordinate, the second horizontal coordinate, the third horizontal coordinate and the fourth horizontal coordinate meet at least one of preset comparison conditions; the preset comparison conditions include: the value of the second horizontal coordinate is greater than or equal to the value of the first horizontal coordinate, the value of the second horizontal coordinate is less than or equal to the value of the first horizontal coordinate, the absolute value of the difference between the value of the first horizontal coordinate and the value of the second horizontal coordinate is greater than or equal to a preset value, the value of the third horizontal coordinate is less than or equal to the value of the first horizontal coordinate, the absolute value of the difference between the value of the third horizontal coordinate and the value of the fourth horizontal coordinate is less than or equal to the absolute value of the difference between the value of the third horizontal coordinate and the value of the first horizontal coordinate.

[0018] Optionally, the image acquisition module includes: a pixel accumulation module, used to accumulate the pixel values ​​of each pixel point in the frame image to which each text area belongs to obtain a pixel value accumulation result; a preprocessing module, used to normalize and binarize the pixel value accumulation result to obtain the text mask image.

[0019] In another aspect of the present invention, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, the computer executes any of the above-mentioned text type classification methods.

[0020] In yet another aspect of the present invention, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute any of the above-mentioned text type classification methods.

[0021] The text type classification scheme provided by the embodiment of the present invention is to determine the non-mask area of ​​each text area by adopting each text area in the text mask image, and to divide each text area into a first text mask area and a second text mask area based on each non-mask area. Then, for each text area, the text type of the first text mask area is determined based on the positional relationship between the non-mask area and the first text mask area. Moreover, for each text area, the technical means of determining the text type of the second text mask area based on the positional relationship between the non-mask area, the second text mask area and the text mask image. In the case where the text area includes two text mask areas, that is, the text area is divided into the first text mask area and the second text mask area, the positional relationship between the non-mask area, the first text mask area, the second text mask area and the text mask image of the text area is used to respectively determine the text type of the first text mask area and the text type of the second text mask area, thereby realizing the classification of the text type in the text area, and solving the technical problem that the text cannot be distinguished as a line text and a name text after the existing text filtering, and achieving the effect that each text type can be distinguished after the text filtering. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0023] Figure 1 The present invention is a flowchart of a text type classification method according to an embodiment of the present invention.

[0024] Figure 2 The present invention is a flowchart of a method for filtering a text area according to an embodiment of the present invention.

[0025] Figure 3 The present invention is a flowchart of a method for classifying name text types and dialogue text types in variety show video data according to an embodiment of the present invention.

[0026] Figure 4 The figure is a schematic diagram of the structure of a text type classification device according to an embodiment of the present invention.

[0027] Figure 5 The figure is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention.

[0028] Figure 6 The present invention is a schematic diagram of the working process of a system for recognizing subtitles of lines in audio and video files according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0030] The embodiment of the present invention proposes a text type classification scheme, which is mainly used to classify name areas and dialogue areas in a text area. First, it is determined whether there is a segmentation column in the text area. If there is a segmentation column in the text area, it means that the text area may contain name text, and it is necessary to further identify the segmentation line from the text area. Then, the name area and dialogue area in the text area are classified according to the positional relationship between the segmentation line, the boundary of the text area, and the boundary of the text mask image.

[0031] like Figure 1 As shown, a flowchart of a method for classifying text types according to an embodiment of the present invention is shown. The method for classifying text types can be applied to a terminal or a server. The method for classifying text types can specifically include the following steps.

[0032] Step 101: Obtain a text mask image to be processed.

[0033] In an embodiment of the present invention, the text mask image may be an image after filtering the text area in the text detection image. Wherein, the text detection image may be derived from video data or audio and video data. In practical applications, the text detection image may be a frame image of video data or audio and video data. Moreover, the text detection image may contain a text area. The text area may be understood as the location where the text content in a frame image is located. In one embodiment, the text detection image may contain one text area or multiple text areas. The text area in the text detection image may be understood as the text area before filtering, that is, the text area in the text detection image may contain lines text content, name text content, non-lines text content, etc. The text area in the text mask image may be understood as the text area after filtering, that is, the text area in the text mask image may contain lines text content and / or name text content, and does not contain non-lines text content.

[0034] Step 102 : for each text region, determine a non-mask region of each text region, and based on each non-mask region, segment each text region into a first text mask region and a second text mask region.

[0035] In an embodiment of the present invention, when a text region includes a segmentation column, it can be considered that the text region includes a name region. The segmentation column of a text region can be understood as the boundary of different types of text content in the text region. In one embodiment, the segmentation column of a text region can be represented by a horizontal axis, that is, the segmentation column of the text region is used to roughly determine the column boundaries of different types of text content in the text region. If a segmentation column exists in a text region, it means that there may be name text content in the text region. It is necessary to further identify the segmentation line, and use the segmentation line to determine whether the text region contains name text content and / or line text content.

[0036] The non-mask area in the embodiment of the present invention may be a segmentation line or include a segmentation line. The non-mask area is used as a segmentation boundary to divide the text area into two, namely, a first text mask area and a second text mask area. Then, the text type of the first text mask area and the text type of the second text mask area are determined respectively.

[0037] Step 103: for each text region, based on the positional relationship between the non-mask region and the first text mask region, determine the text type of the first text mask region.

[0038] In an embodiment of the present invention, the positional relationship between the non-mask area and the first text mask area can be determined according to the boundary of the non-mask area and the boundary of the first text mask area, and then the text type of the first text mask area can be determined according to the positional relationship. In practical applications, the text type of the first text mask area can be classified as a name text area or a line text area frame.

[0039] Step 104 : for each text region, determine the text type of the second text mask region based on the positional relationship among the non-mask region, the second text mask region and the text mask image.

[0040] In an embodiment of the present invention, the positional relationship between the non-mask area and the second text mask area, as well as the positional relationship between the non-mask area, the second text mask area and the text mask image can be determined according to the boundary of the non-mask area, the boundary of the second text mask area and the boundary of the text mask image, and then the text type of the second text mask area can be determined according to the positional relationship. In practical applications, the text type of the second text mask area can be classified as a name text area or a line text area frame.

[0041] It should be noted that the step execution relationship between step 103 and step 104 can be a parallel execution relationship or a sequential execution relationship. The sequential execution relationship can be that step 103 is executed first, and then step 104 is executed, or that step 104 is executed first, and then step 103 is executed.

[0042] The text type classification scheme provided by the embodiment of the present invention is to determine the non-mask area of ​​each text area by adopting each text area in the text mask image, and to divide each text area into a first text mask area and a second text mask area based on each non-mask area. Then, for each text area, the text type of the first text mask area is determined based on the positional relationship between the non-mask area and the first text mask area. Moreover, for each text area, the technical means of determining the text type of the second text mask area based on the positional relationship between the non-mask area, the second text mask area and the text mask image. In the case where the text area includes two text mask areas, that is, the text area is divided into the first text mask area and the second text mask area, the positional relationship between the non-mask area, the first text mask area, the second text mask area and the text mask image of the text area is used to respectively determine the text type of the first text mask area and the text type of the second text mask area, thereby realizing the classification of the text type in the text area, and solving the technical problem that the text cannot be distinguished as a line text and a name text after the existing text filtering, and achieving the effect that each text type can be distinguished after the text filtering.

[0043] In an exemplary embodiment of the present invention, an implementation method for determining the non-mask area of ​​each text area is to obtain the pixel value of each pixel point in each text area, and use the area where the pixel value meets the preset pixel condition as the non-mask area of ​​the corresponding text area frame. In practical applications, the area where the average value of the pixel value is less than the preset pixel threshold can be used as the non-mask area. For example, in the text area T1, there is an area Tmz where the average value of the pixel value is less than the preset pixel threshold, and the area Tmz is used as the non-mask area. Further, the middle column Tmzl of the area Tmz can be used as a dividing line. The above-mentioned preset pixel threshold can be set to 255, and the embodiment of the present invention does not specifically limit the value and unit of the preset pixel threshold.

[0044] In an exemplary embodiment of the present invention, based on the positional relationship between the non-mask area and the first text mask area, an implementation method of determining the text type of the first text mask area is to obtain the first horizontal coordinate of the non-mask area and the second horizontal coordinate of the right boundary of the first text mask area, and if the value of the second horizontal coordinate is less than the value of the first horizontal coordinate, the first text mask area is classified as the first text type. Alternatively, if the value of the second horizontal coordinate is greater than the value of the first horizontal coordinate, and the absolute value of the difference between the value of the first horizontal coordinate and the value of the second horizontal coordinate is less than a preset value, the first text mask area is classified as the first text type. That is, according to the first horizontal coordinate of the non-mask area and the second horizontal coordinate of the right boundary of the first text mask area, the positional relationship between the non-mask area and the right boundary of the first text mask area is determined. When the value of the second horizontal coordinate is less than the value of the first horizontal coordinate, it means that the right boundary of the first text mask area is located on the left side of the non-mask area, and the first text mask area can be classified as the first text type. When the value of the second horizontal coordinate is greater than the value of the first horizontal coordinate, it means that the right boundary of the first text mask area is located on the right side of the non-mask area. Moreover, the absolute value of the difference between the numerical value of the first horizontal coordinate and the numerical value of the second horizontal coordinate is less than the preset numerical value, indicating that the distance between the non-mask area and the right border of the first text mask area is less than the preset numerical value. When the right border of the first text mask area is located on the right side of the non-mask area, and the distance between the non-mask area and the right border of the first text mask area is less than the preset numerical value, the first text mask area can be classified as the first text type. The above-mentioned preset numerical value can be set to 10 pixel units, and the embodiment of the present invention does not specifically limit the preset numerical value.

[0045] In an exemplary embodiment of the present invention, based on the positional relationship between the non-mask area, the second text mask area and the text mask image, an implementation method for determining the text type of the second text mask area is to obtain the first horizontal coordinate of the non-mask area, the third horizontal coordinate of the left boundary of the second text mask area and the fourth horizontal coordinate of the left boundary of the text mask image, and if the value of the third horizontal coordinate is greater than the value of the first horizontal coordinate, the second text mask area is classified as the second text type. Alternatively, if the absolute value of the difference between the value of the third horizontal coordinate and the value of the fourth horizontal coordinate is greater than the absolute value of the difference between the value of the third horizontal coordinate and the value of the first horizontal coordinate, the second text mask area is classified as the second text type. That is, according to the first horizontal coordinate of the non-mask area, the third horizontal coordinate of the left boundary of the second text mask area and the fourth horizontal coordinate of the left boundary of the text mask image, the positional relationship between the non-mask area, the left boundary of the second text mask area and the left boundary of the text mask image is determined. When the value of the third horizontal coordinate is greater than the value of the first horizontal coordinate, it means that the left border of the second text mask area is located on the right side of the non-mask area, and the second text mask area can be classified as the second text type. When the absolute value of the difference between the value of the third horizontal coordinate and the value of the fourth horizontal coordinate is greater than the absolute value of the difference between the value of the third horizontal coordinate and the value of the first horizontal coordinate, it means that the distance from the left border of the second text mask area to the left border of the text mask image is greater than the distance from the left border of the second text mask area to the non-mask area, and the second text mask area can be classified as the second text type.

[0046] In an exemplary embodiment of the present invention, in addition to the above two implementations of determining the text type of the first text mask area and the second text mask area, the first horizontal coordinate of the non-mask area, the second horizontal coordinate of the right boundary of the first text mask area, the third horizontal coordinate of the left boundary of the second text mask area, and the fourth horizontal coordinate of the left boundary of the text mask image can also be obtained. If the first horizontal coordinate, the second horizontal coordinate, the third horizontal coordinate, and the fourth horizontal coordinate meet at least one of the preset comparison conditions, the first text mask area is classified as the first text type, and the second text mask area is classified as the second text type. Wherein, the preset comparison conditions include: the value of the second horizontal coordinate is greater than or equal to the value of the first horizontal coordinate, the value of the second horizontal coordinate is less than or equal to the value of the first horizontal coordinate, the absolute value of the difference between the value of the first horizontal coordinate and the value of the second horizontal coordinate is greater than or equal to the preset value, the value of the third horizontal coordinate is less than or equal to the value of the first horizontal coordinate, the absolute value of the difference between the value of the third horizontal coordinate and the value of the fourth horizontal coordinate is less than or equal to the absolute value of the difference between the value of the third horizontal coordinate and the value of the first horizontal coordinate.

[0047] In an exemplary embodiment of the present invention, one implementation of the above-mentioned obtaining of the text mask image to be processed is to accumulate the pixel values ​​of each pixel point in the frame image to which each text area belongs to obtain a pixel value accumulation result, and preprocess the pixel value accumulation result to obtain the text mask image. Among them, the preprocessing may include normalization processing and binarization processing, etc., and the embodiment of the present invention does not specifically limit the processing conditions, processing content, processing steps, processing results, etc. of the preprocessing.

[0048] It should be noted that the first text mask area can be located on the left side of the text area, and the second text mask area can be located on the right side of the text area. The first text type can be a name text type, and the second text type can be a line text type. The text mask image is obtained by filtering the text area in the text detection image. For the filtering steps of the text area in the text detection image, please refer to Figure 2 .like Figure 2 FIG. 1 is a flowchart showing a method for filtering a text area according to an embodiment of the present invention. The method for filtering a text area may specifically include the following steps.

[0049] Step 201: Obtain a frame image to be recognized.

[0050] In an embodiment of the present invention, the frame image may be derived from video data or audio and video data. In practical applications, the frame image may be a frame image of video data or audio and video data. Moreover, the frame image may include a text area. The text area may be understood as an area where text content in a frame image is located. In one embodiment, the frame image may include one text area or multiple text areas.

[0051] Step 202: extract the feature value of each pixel in the frame image.

[0052] In an embodiment of the present invention, the frame image may be input into a pre-trained network model, and then the network model may be used to extract the feature value of each pixel in the frame image.

[0053] Step 203, calculating the gradient of each pixel point according to the characteristic value, and identifying the frame image position corresponding to the gradient peak in each gradient from the frame image.

[0054] In an embodiment of the present invention, the gradient of each pixel point in the positive direction and the gradient in the negative direction can be calculated based on the characteristic value of each pixel point, and then the gradient in the positive direction and the gradient in the negative direction determine the gradient peak in each direction, and then the frame image position corresponding to the gradient peak is identified from the frame image.

[0055] Step 204: Count the number of pixels corresponding to the gradient peak in the frame image position.

[0056] In the embodiment of the present invention, the number of pixel points in a preset surrounding area may be counted with the pixel point corresponding to the gradient peak value as the center in the frame image position.

[0057] Step 205 , filtering out the segmentation position of the text area from the frame image position according to the number of pixels.

[0058] In the embodiment of the present invention, the segmentation position can be screened out from the frame image position according to the number of pixels, a preset threshold, and the positional relationship between the frame image position and the central axis of the text area.

[0059] Step 206, segment the text area along the segmentation position.

[0060] In the embodiment of the present invention, the text region may be divided left and right or up and down along the division position, that is, one text region may be divided into two left and right text regions or two up and down text regions along the division position.

[0061] In an exemplary embodiment of the present invention, an implementation method of selecting the segmentation position of the text area from the frame image position according to the number of pixels is to determine whether there are at least two frame image positions whose number of pixels is greater than a preset threshold and both are located on the same side of the central axis of the text area; if there are at least two frame image positions whose number of pixels is greater than the preset threshold and both are located on the same side of the central axis of the text area, then the frame image position close to the central axis of the at least two frame image positions is used as the segmentation position. For example, there are frame image positions U1, U2, U3, U4 and U5. If the number of pixels of frame image positions U1 and U2 is greater than the preset threshold, and frame image positions U1 and U2 are located on the same side of the central axis of the text area. Among frame image positions U1 and U2, frame image position U1 is closer to the central axis of the text area, then frame image position U1 is used as the segmentation position of the text area. The above-mentioned preset threshold value can be a pre-set empirical value, or the above-mentioned preset threshold value can be the difference between the number of pixels corresponding to the gradient peak values ​​in the current frame image position after sorting the number of pixels corresponding to the gradient peak values ​​in each frame image position and the number of pixels in the first position or the number of pixels in the previous position.

[0062] In an exemplary embodiment of the present invention, the frame image may include at least one text area, and the text display direction in the text area can generally be divided into two directions: horizontal and vertical. Therefore, one implementation method of identifying the frame image position corresponding to the gradient peak in each gradient from the frame image is that when the frame image includes at least one text area, for at least one text area with a horizontal text display direction, at least one row position corresponding to the gradient peak in each gradient is identified. It should be noted that the row position in this implementation method can be a horizontal row position, or a row position at a certain angle to the horizontal row position. Another implementation method of identifying the frame image position corresponding to the gradient peak in each gradient from the frame image is that when the frame image includes at least one text area, for at least one text area with a vertical text display direction, at least one column position corresponding to the gradient peak in each gradient is identified. It should be noted that the column position in this implementation method can be a vertical column position, or a column position at a certain angle to the vertical column position.

[0063] In an exemplary embodiment of the present invention, since the gradient is a vector, it includes a gradient in a positive direction and a gradient in a negative direction. In practical applications, if the text display direction in the text area is horizontal, then one implementation method for identifying at least one row position corresponding to the gradient peak in each gradient is, when the gradient peak is a positive gradient peak, the upper boundary of the text area with a horizontal text display direction is identified; when the gradient peak is a negative gradient peak, the lower boundary of the text area with a horizontal text display direction is identified. If the text display direction in the text area is vertical, then one implementation method for identifying at least one column position corresponding to the gradient peak in each gradient is, when the gradient peak is a positive gradient peak, the right boundary of the text area with a vertical text display direction is identified; when the gradient peak is a negative gradient peak, the left boundary of the text area with a vertical text display direction is identified.

[0064] In an exemplary embodiment of the present invention, one implementation method of counting the number of pixels corresponding to the gradient peak value in the frame image position is to count the number of pixels in a preset surrounding area of ​​the pixel corresponding to the gradient peak value in the frame image position. As can be seen from the above, the frame image position can be the upper boundary, lower boundary, left boundary or right boundary of the text area. For example, the frame image position U is the upper boundary of the text area. For the pixel point d corresponding to the gradient peak value of the frame image position U, in the preset surrounding area [d xy -dis,d xy +dis] to obtain the number of pixels UX. xy represents the horizontal coordinate value and the vertical coordinate value of the pixel point d, and dis represents the preset coordinate threshold.

[0065] In an exemplary embodiment of the present invention, one implementation method of extracting the feature value of each pixel in the frame image is to extract the gray feature value of each pixel in the frame image. In practical applications, the frame image can be a gray image or an RGB image. If the frame image is an RGB image, the RGB image can be further converted into a gray image, and then the gray feature value of each pixel in the frame image is extracted.

[0066] Based on the above description of the text type classification method embodiment, a classification method for name text type and line text type in variety show video data is introduced below. Figure 3 As shown, Figure 3 A flow chart showing the steps of a method for classifying name text types and dialogue text types in variety show video data.

[0067] Get a text mask image. The text area in the text mask image is a filtered text area containing dialogue text content, or a text area containing name text content, or a text area containing dialogue text content and name text content. Determine whether the text area contains a segmentation column. If the text area does not contain a segmentation column, it is considered that the text area does not contain name text content, and the text area contains dialogue text content. If the text area contains a segmentation column, further identify the area with an average pixel value less than 255 from the text area, use the identified area as a non-mask area, and use the middle column of the identified area as a segmentation line. Then, based on the non-mask area or the segmentation line, the text area is divided into a first text mask area and a second text mask area. If the right boundary of the first text mask area is on the left side of the non-mask area or the segmentation line, it is considered that the first text mask area contains name text content and is a name text type. If the right border of the first text mask area is on the right side of the non-masked area or the dividing line, and the distance between the right border of the first text mask area and the non-masked area is less than 10 pixels, the first text mask area is also considered to contain name text content and is of the name text type. If the left border of the second text mask area is on the right side of the non-masked area or the dividing line, the second text mask area is considered to contain dialogue text content and is of the dialogue text type. If the distance from the left border of the second text mask area to the left border of the text mask image is greater than the distance from the left border of the second text mask area to the non-masked area or the dividing line, the second text mask area is also considered to contain dialogue text content and is of the dialogue text type. In other scenarios, it can be considered that the text area contains both dialogue text content and name dialogue content, then the first text mask area is directly classified as the name text type, and the second text mask area is classified as the dialogue text type.

[0068] like Figure 4FIG. 1 is a schematic diagram showing a structure of a text type classification device according to an embodiment of the present invention. The text type classification device may include the following modules.

[0069] An image acquisition module 41 is used to acquire a text mask image to be processed, wherein the text mask image includes at least one text area;

[0070] A region segmentation module 42, configured to determine, for each of the text regions, a non-mask region of each of the text regions, and segment each of the text regions into a first text mask region and a second text mask region based on each of the non-mask regions;

[0071] A type classification module 43, configured to determine, for each of the text regions, a text type of the first text mask region based on a positional relationship between the non-mask region and the first text mask region;

[0072] The type classification module 43 is further configured to determine, for each of the text regions, the text type of the second text mask region based on the positional relationship among the non-mask region, the second text mask region and the text mask image.

[0073] In an exemplary embodiment of the present invention, the region segmentation module 42 includes:

[0074] A pixel value acquisition module, used to acquire the pixel value of each pixel point in each of the text areas;

[0075] The non-mask region determining module is used to use the region whose pixel value meets the preset pixel condition as the non-mask region of the corresponding text region.

[0076] In an exemplary embodiment of the present invention, the non-mask region determining module is used to take a region where the average value of the pixel values ​​is less than a preset pixel threshold as the non-mask region.

[0077] In an exemplary embodiment of the present invention, the type classification module 43 includes:

[0078] A coordinate acquisition module, used for acquiring a first horizontal coordinate of the non-mask area and a second horizontal coordinate of the right boundary of the first text mask area;

[0079] A text type classification module is used to classify the first text mask area as the first text type if the value of the second horizontal coordinate is smaller than the value of the first horizontal coordinate; or to classify the first text mask area as the first text type if the value of the second horizontal coordinate is larger than the value of the first horizontal coordinate and the absolute value of the difference between the values ​​of the first horizontal coordinate and the second horizontal coordinate is smaller than a preset value.

[0080] In an exemplary embodiment of the present invention, the coordinate acquisition module is further used to acquire a third horizontal coordinate of the left boundary of the second text mask area and a fourth horizontal coordinate of the left boundary of the text mask image;

[0081] The text type classification module is also used to classify the second text mask area as the second text type if the value of the third horizontal coordinate is greater than the value of the first horizontal coordinate; or classify the second text mask area as the second text type if the absolute value of the difference between the value of the third horizontal coordinate and the value of the fourth horizontal coordinate is greater than the absolute value of the difference between the value of the third horizontal coordinate and the value of the first horizontal coordinate.

[0082] In an exemplary embodiment of the present invention, the text type classification module is further configured to classify the first text mask area as the second text type, and classify the second text mask area as the first text type if the first horizontal coordinate, the second horizontal coordinate, the third horizontal coordinate, and the fourth horizontal coordinate satisfy at least one of the preset comparison conditions;

[0083] The preset comparison conditions include: the value of the second horizontal coordinate is greater than or equal to the value of the first horizontal coordinate, the value of the second horizontal coordinate is less than or equal to the value of the first horizontal coordinate, the absolute value of the difference between the value of the first horizontal coordinate and the value of the second horizontal coordinate is greater than or equal to a preset value, the value of the third horizontal coordinate is less than or equal to the value of the first horizontal coordinate, and the absolute value of the difference between the value of the third horizontal coordinate and the value of the fourth horizontal coordinate is less than or equal to the absolute value of the difference between the value of the third horizontal coordinate and the value of the first horizontal coordinate.

[0084] In an exemplary embodiment of the present invention, the image acquisition module 41 includes:

[0085] A pixel accumulation module, used for accumulating the pixel values ​​of each pixel point in the frame image to which each text area belongs to obtain a pixel value accumulation result;

[0086] The preprocessing module is used to perform normalization and binarization on the pixel value accumulation result to obtain the text mask image.

[0087] The embodiment of the present invention further provides an electronic device, such as Figure 5 As shown, it includes a processor 51, a communication interface 52, a memory 53 and a communication bus 54, wherein the processor 51, the communication interface 52, and the memory 53 communicate with each other through the communication bus 54.

[0088] A memory 53, for storing computer programs;

[0089] The processor 51 is used to execute the program stored in the memory 53 to implement the following steps:

[0090] Acquire a text mask image to be processed, wherein the text mask image includes at least one text area;

[0091] For each of the text regions, determine a non-mask region of each of the text regions, and based on each of the non-mask regions, segment each of the text regions into a first text mask region and a second text mask region;

[0092] For each of the text regions, determining the text type of the first text mask region based on the positional relationship between the non-mask region and the first text mask region;

[0093] For each of the text regions, the text type of the second text mask region is determined based on the positional relationship among the non-mask region, the second text mask region and the text mask image.

[0094] The determining of the non-masked area of ​​each of the text areas comprises:

[0095] Obtaining the pixel value of each pixel in each of the text areas;

[0096] The area whose pixel value meets the preset pixel condition is used as the non-mask area of ​​the corresponding text area.

[0097] The step of taking the area whose pixel value satisfies the preset pixel condition as the non-mask area of ​​the corresponding text area includes:

[0098] The area where the average value of the pixel value is less than the preset pixel threshold is used as the non-mask area.

[0099] The determining the text type of the first text mask area based on the positional relationship between the non-mask area and the first text mask area includes:

[0100] Obtaining a first horizontal coordinate of the non-mask area and a second horizontal coordinate of the right border of the first text mask area;

[0101] If the value of the second horizontal coordinate is smaller than the value of the first horizontal coordinate, the first text mask area is classified as the first text type; or, if the value of the second horizontal coordinate is larger than the value of the first horizontal coordinate, and the absolute value of the difference between the value of the first horizontal coordinate and the value of the second horizontal coordinate is smaller than a preset value, the first text mask area is classified as the first text type.

[0102] The determining the text type of the second text mask area based on the positional relationship among the non-mask area, the second text mask area and the text mask image includes:

[0103] Acquire a first horizontal coordinate of the non-mask area, a third horizontal coordinate of the left border of the second text mask area, and a fourth horizontal coordinate of the left border of the text mask image;

[0104] If the value of the third horizontal coordinate is greater than the value of the first horizontal coordinate, the second text mask area is classified as the second text type; or, if the absolute value of the difference between the value of the third horizontal coordinate and the value of the fourth horizontal coordinate is greater than the absolute value of the difference between the value of the third horizontal coordinate and the value of the first horizontal coordinate, the second text mask area is classified as the second text type.

[0105] The method further comprises: acquiring a first horizontal coordinate of the non-mask area, a second horizontal coordinate of the right boundary of the first text mask area, a third horizontal coordinate of the left boundary of the second text mask area, and a fourth horizontal coordinate of the left boundary of the text mask image;

[0106] If the first horizontal coordinate, the second horizontal coordinate, the third horizontal coordinate and the fourth horizontal coordinate satisfy at least one of the preset comparison conditions, classifying the first text mask area as the second text type, and classifying the second text mask area as the first text type;

[0107] The preset comparison conditions include: the value of the second horizontal coordinate is greater than or equal to the value of the first horizontal coordinate, the value of the second horizontal coordinate is less than or equal to the value of the first horizontal coordinate, the absolute value of the difference between the value of the first horizontal coordinate and the value of the second horizontal coordinate is greater than or equal to a preset value, the value of the third horizontal coordinate is less than or equal to the value of the first horizontal coordinate, and the absolute value of the difference between the value of the third horizontal coordinate and the value of the fourth horizontal coordinate is less than or equal to the absolute value of the difference between the value of the third horizontal coordinate and the value of the first horizontal coordinate.

[0108] The step of obtaining a text mask image to be processed includes:

[0109] Accumulating the pixel values ​​of each pixel point in the frame image to which each text area belongs to obtain a pixel value accumulation result;

[0110] The pixel value accumulation result is normalized and binarized to obtain the text mask image.

[0111] Based on the description of the above-mentioned text type classification method and device and other embodiments, a system for recognizing subtitles in audio and video files is introduced below. The recognition system can be composed of hardware devices such as a personal computer or a server, a server cluster, etc. A subtitle detection framework is deployed in the recognition system, and the subtitle detection framework mainly includes a subtitle detection model, a subtitle filtering model, a subtitle tracking model, and a text classification model.

[0112] Reference Figure 6 , shows a schematic diagram of the workflow of a system for recognizing lines and subtitles in an audio and video file according to an embodiment of the present invention. In actual application, a complete audio and video file is input into the recognition system, and a graphics processing unit (GPU) of a hardware device is used to extract video frame results from the audio and video file. The specific video frame results may be partial frame images and full frame images. Among them, the partial frame images may be three frames of images extracted from a 1-second time period of the audio and video file. In order to reduce the amount of computation required for extracting the video frame results, extraction processing may be performed on partial video frame images of the audio and video file, for example, only the lower 1 / 3 of the video frame image may be extracted. The extracted partial frame images may be continuously written into a memory queue. The extracted full frame images may be stored in a hard disk.

[0113] Based on multi-threading, multiple dialogue subtitle detection models are started, partial frame images are continuously read from the memory queue, and the read partial frame images are spliced. For example, three consecutive partial frame images are spliced ​​into one frame image. The dialogue subtitle detection model locates the positions of all text boxes from the spliced ​​frame images.

[0114] The dialogue subtitle filtering model counts the frequency of all text boxes appearing in the time draw, determines a high-frequency heat map area based on the frequency, and then uses the high-frequency heat map area as the filtering standard area for dialogue subtitles and non-dialogue subtitles, and uses the filtering standard area to filter all text boxes into dialogue subtitle text boxes and non-dialogue subtitle text boxes. Among them, non-dialogue subtitle text boxes are discarded and do not participate in subsequent processing.

[0115] The dialogue and subtitle tracking model extracts the deep features of optical character recognition (OCR) from the dialogue and subtitle text boxes, and performs tracking processing based on the full frame images stored in the hard disk to obtain the start and end time information of each dialogue and subtitle text box in the audio and video files.

[0116] The text classification model determines the language information corresponding to each subtitle text box, and transmits the deep features to the corresponding OCR prediction network according to the language information. The OCR prediction network uses its own text recognition algorithm to recognize the subtitles in the subtitle text box to obtain the subtitle results. For example, the language information can be Chinese or English. The Chinese language corresponds to the Chinese OCR prediction network, and the English language corresponds to the English OCR prediction network.

[0117] The recognition system provided by the embodiment of the present invention constructs a software development kit (SDK) for line and subtitle frame-level recognition. For the same audio and video file, the ratio of the time for line and subtitle recognition on the GPU to the time for line and subtitle recognition through the SDK is about 1:0.11, which greatly improves the speed and accuracy of line and subtitle recognition. Based on the line results obtained by recognition, external subtitles of audio and video files can be generated, and embedded subtitles of audio and video files can be quickly converted into external subtitles.

[0118] The recognition system provided by the embodiment of the present invention uses the time domain information of the text box to filter the text box into a line subtitle text box and a non-line subtitle text box when filtering the text box, and the accuracy of text box filtering is very high. When the audio and video file is a film and television work, the filtering accuracy exceeds 99%; when the audio and video file is a variety show, the filtering accuracy exceeds 98.5%.

[0119] The recognition system provided by the embodiment of the present invention can also extract video frames for designated areas such as name strips, lyric boards and lyrics in variety shows, and then perform processes such as positioning, filtering, tracking, classification and recognition of names, lyrics, etc., thereby realizing text recognition of names, lyrics, etc.

[0120] The recognition system provided by the embodiment of the present invention can not only recognize the lines and subtitles in Chinese and English, but also realize the intelligent recognition of the lines and subtitles in Chinese and English, Chinese and Japanese, Chinese and Korean, etc. in dual-audio and video files, and supports the recognition of lines and subtitles in multi-language audio and video files.

[0121] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0122] The communication interface is used for communication between the above terminal and other devices.

[0123] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0124] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0125] In another embodiment provided by the present invention, a computer-readable storage medium is provided, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the text type classification method described in any of the above embodiments.

[0126] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes the text type classification method described in any one of the above embodiments.

[0127] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0128] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0129] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0130] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A text type classification method, characterized in that: include: Acquire a text mask image to be processed, wherein the text mask image includes at least one text area; For each of the text regions, determine a non-mask region of each of the text regions, and based on each of the non-mask regions, segment each of the text regions into a first text mask region and a second text mask region; For each of the text regions, determining the text type of the first text mask region based on the positional relationship between the non-mask region and the first text mask region; For each of the text regions, determining the text type of the second text mask region based on the positional relationship among the non-mask region, the second text mask region and the text mask image; The text mask image is an image obtained by filtering the text area in the text detection image; The text type includes: any one of a name text area and a line text area box; The non-mask area is the text area whose pixel values ​​meet the preset pixel conditions.

2. The method according to claim 1, characterized in that The determining of the non-masked area of ​​each of the text areas comprises: Obtaining the pixel value of each pixel in each of the text areas; The area whose pixel value meets the preset pixel condition is used as the non-mask area of ​​the corresponding text area.

3. The method according to claim 2, characterized in that The step of taking the area whose pixel value satisfies the preset pixel condition as the non-mask area of ​​the corresponding text area includes: The area where the average value of the pixel value is less than the preset pixel threshold is used as the non-mask area.

4. The method according to claim 1, characterized in that: The determining the text type of the first text mask area based on the positional relationship between the non-mask area and the first text mask area includes: Obtaining a first horizontal coordinate of the non-mask area and a second horizontal coordinate of the right border of the first text mask area; If the value of the second horizontal coordinate is smaller than the value of the first horizontal coordinate, the first text mask area is classified as the first text type; or, if the value of the second horizontal coordinate is larger than the value of the first horizontal coordinate, and the absolute value of the difference between the value of the first horizontal coordinate and the value of the second horizontal coordinate is smaller than a preset value, the first text mask area is classified as the first text type.

5. The method according to claim 1, characterized in that The determining the text type of the second text mask area based on the positional relationship among the non-mask area, the second text mask area and the text mask image includes: Acquire a first horizontal coordinate of the non-mask area, a third horizontal coordinate of the left border of the second text mask area, and a fourth horizontal coordinate of the left border of the text mask image; If the value of the third horizontal coordinate is greater than the value of the first horizontal coordinate, the second text mask area is classified as the second text type; or, if the absolute value of the difference between the value of the third horizontal coordinate and the value of the fourth horizontal coordinate is greater than the absolute value of the difference between the value of the third horizontal coordinate and the value of the first horizontal coordinate, the second text mask area is classified as the second text type.

6. The method according to claim 1, characterized in that The method further comprises: Acquire a first horizontal coordinate of the non-mask area, a second horizontal coordinate of the right border of the first text mask area, a third horizontal coordinate of the left border of the second text mask area, and a fourth horizontal coordinate of the left border of the text mask image; If the first horizontal coordinate, the second horizontal coordinate, the third horizontal coordinate and the fourth horizontal coordinate satisfy at least one of the preset comparison conditions, classifying the first text mask area as the second text type and classifying the second text mask area as the first text type; The preset comparison conditions include: the value of the second horizontal coordinate is greater than or equal to the value of the first horizontal coordinate, the value of the second horizontal coordinate is less than or equal to the value of the first horizontal coordinate, the absolute value of the difference between the value of the first horizontal coordinate and the value of the second horizontal coordinate is greater than or equal to a preset value, the value of the third horizontal coordinate is less than or equal to the value of the first horizontal coordinate, and the absolute value of the difference between the value of the third horizontal coordinate and the value of the fourth horizontal coordinate is less than or equal to the absolute value of the difference between the value of the third horizontal coordinate and the value of the first horizontal coordinate.

7. The method according to any one of claims 1 to 6, characterized in that The step of obtaining a text mask image to be processed includes: Accumulating the pixel values ​​of each pixel point in the frame image to which each text area belongs to obtain a pixel value accumulation result; The pixel value accumulation result is normalized and binarized to obtain the text mask image.

8. A text type classification device, characterized in that: include: An image acquisition module, used for acquiring a text mask image to be processed, wherein the text mask image includes at least one text area; A region segmentation module, used for determining a non-mask region of each text region, and segmenting each text region into a first text mask region and a second text mask region based on each non-mask region; A type classification module, configured to determine, for each of the text regions, a text type of the first text mask region based on a positional relationship between the non-mask region and the first text mask region; The type classification module is further used to determine, for each of the text regions, the text type of the second text mask region based on the positional relationship between the non-mask region, the second text mask region and the text mask image; The text mask image is an image obtained by filtering the text area in the text detection image; The text type includes: any one of a name text area and a line text area box; The non-mask area is the text area whose pixel values ​​meet the preset pixel conditions.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 7 when executing a program stored in a memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images