Dialog box detection method, device, electronic device and storage medium

By performing binarization of video frame images and region detection, combined with shape and color judgment, the problems of poor dialog box detection accuracy and high labor cost are solved, and high accuracy and low cost dialog box detection are achieved.

CN113869310BActive Publication Date: 2024-11-01BEIJING DAJIA INTERNET INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111137112.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-09-27
Publication Date
2024-11-01
Estimated Expiration
2041-09-27

AI Technical Summary

Technical Problem

In the prior art, the dialog box detection accuracy is poor and a large amount of manual labeling of data is required, resulting in high labor costs.

Method used

By binarizing the video frame image to be detected, the connection area is detected, the preset polygon area is determined, and whether it is a dialog box based on the shape and color is determined. The dialog box that distinguishes sharp corners of the triangle shape is reduced by using the pixel mean proportional value to reduce the number of times the pixels is used to determine the value of the pixels. Dependence of deep learning.

Benefits of technology

Improve the accuracy of dialog box detection, reduce labor costs, and does not require manual labeling of data, and can accurately detect dialog boxes with local details.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113869310B_ABST
    Figure CN113869310B_ABST
Patent Text Reader

Abstract

The present disclosure relates to a dialog box detection method, device, electronic device and storage medium, the method comprising: binarizing a video frame image to be detected to obtain a binarized image; performing region detection on the binarized image to obtain a connected region in the binarized image; determining a connected region of a preset polygon in the connected region as a preset polygon region; and determining the preset polygon region to be a dialog box of a preset dialog box type when a preset shape and / or preset color exists at a preset position of the preset polygon region. The present disclosure can accurately detect a dialog box with local details by detecting a preset shape and a preset color after determining the preset polygon region, thereby improving the accuracy of detection, and does not require the use of a deep learning method for detection, does not require manual data annotation, and reduces labor costs.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a dialog box detection method, device, electronic device and storage medium. Background Art

[0002] Image object detection is a hot topic in image processing. Academically, there are several studies on general object detection and specific object detection. In practical work, there is usually targeted detection of certain types of objects (such as faces, vehicles, etc.). Object detection methods are currently divided into two main categories: traditional methods and deep learning. In recent years, deep learning methods have achieved great results in the field of object detection. Object detection technology based on deep learning and other methods is a popular solution for many detection problems. A target detection network model is trained using a large amount of training data with labeled detection boxes. Then, the image to be detected is input into the target detection network model to obtain the target box coordinate information of the target.

[0003] Currently, many video contents are artificially added with many dialog boxes to help describe the video content or increase the fun and richness of the video content. However, some imported videos use dialog box templates that are not provided by their own applications, which requires the detection of such special dialog boxes. The appearance difference between such special dialog boxes and the dialog boxes provided by the application is very small. If a deep learning-based method is used for detection, due to the downsampling process in the deep neural network, such method cannot distinguish the differences in the details of the detection target well, and the detection performance of such special dialog boxes is poor. In addition, a large amount of labeled training data is required, which consumes a large amount of manpower cost. Summary of the invention

[0004] The present disclosure provides a dialog detection method, device, electronic device and storage medium to at least solve the problem of poor dialog detection accuracy and waste of manpower in the related art. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, a dialog box detection method is provided, including:

[0006] Perform binarization processing on the video frame image to be detected to obtain a binarized image;

[0007] Performing region detection on the binary image to obtain connected regions in the binary image;

[0008] Determine a connected area of ​​the connected area that is a preset polygon as a preset polygon area;

[0009] When a preset shape and / or a preset color exists at a preset position of the preset polygonal area, it is determined that the preset polygonal area is a dialog box of a preset dialog box type.

[0010] Optionally, when a preset shape and / or preset color exists at a preset position of the preset polygonal area, determining that the preset polygonal area is a dialog box of a preset dialog box type includes:

[0011] Determine preset vertices in the preset polygonal area, and extract a rectangular area or a square area of ​​a preset size based on the preset vertices as the area to be detected;

[0012] Based on the diagonal of the area to be detected, the area to be detected is divided into a first area and a second area, wherein the first area is an area located above the diagonal, and the second area is an area located below the diagonal;

[0013] In the binary image, determining a pixel mean of the first region and a pixel mean of the second region;

[0014] Determine a ratio of a pixel mean value of the first area to a pixel mean value of the second area as a first ratio value;

[0015] When the first ratio value is greater than a judgment threshold, the preset polygonal area is determined to be a dialog box including a triangle-shaped sharp corner.

[0016] Optionally, when the first ratio value is greater than a judgment threshold, determining that the preset polygonal area is a dialog box including a triangle-shaped sharp corner includes:

[0017] When the diagonal line is a diagonal line between the lower left corner and the upper right corner of the area to be detected, and the first ratio value is greater than the judgment threshold, determining that the polygonal area is a dialog box including a first triangular corner;

[0018] When the diagonal line is a diagonal line between the upper left corner and the lower right corner of the area to be detected, and the first ratio value is greater than the judgment threshold, it is determined that the preset polygonal area is a dialog box including a second triangular corner.

[0019] Optionally, the judgment threshold is determined by the following steps:

[0020] For a plurality of image samples of dialog boxes including triangle-shaped sharp corners, respectively determine a pixel mean of a first region of each image sample as a first pixel mean, and respectively determine a pixel mean of a second region of each image sample as a second pixel mean;

[0021] Determine the ratio of the second pixel mean value to the first pixel mean value of each image sample as a second ratio value corresponding to each image sample;

[0022] Determining a plurality of candidate thresholds according to the difference between the second ratio values ​​corresponding to each image sample;

[0023] The judgment threshold is determined according to the multiple candidate thresholds.

[0024] Optionally, when a preset shape and / or preset color exists at a preset position of the preset polygonal area, determining that the preset polygonal area is a dialog box of a preset dialog box type includes:

[0025] When the preset polygonal area is a rectangular area, extracting a lower boundary range area and / or a right boundary range area of ​​the rectangular area from the video frame image to be detected, and using the lower boundary range area and / or the right boundary range area as the area to be detected;

[0026] The pixel mean value of the area to be detected is determined, and when the pixel mean value is within a preset color range, the preset polygonal area is determined to be a dialog box including a preset color.

[0027] Optionally, the preset polygon is a rectangle;

[0028] Determining a region in the connected region that is a preset polygon as the preset polygon region includes:

[0029] Performing polygonal approximation processing on the connected area to determine the polygonal boundary of the connected area, and determining the polygonal area in the connected area based on the polygonal boundary of the connected area;

[0030] When the polygonal region has four polygonal boundaries, determining the angle between two intersecting edges of the polygonal region;

[0031] When the angle between the two intersecting edges is within a preset degree range, the polygonal area is determined to be a rectangular area.

[0032] According to a second aspect of an embodiment of the present disclosure, there is provided a dialog box detection device, comprising:

[0033] A binarization processing module is configured to perform binarization processing on the video frame image to be detected to obtain a binarized image;

[0034] A region detection module is configured to perform region detection on the binary image to obtain a connected region in the binary image;

[0035] A region determination module is configured to determine a connected region of the connected region that is a preset polygon as a preset polygon region;

[0036] The dialog detection module is configured to determine that the preset polygonal area is a dialog of a preset dialog type when a preset shape and / or preset color exists at a preset position of the preset polygonal area.

[0037] Optionally, the dialog detection module includes:

[0038] A first to-be-detected area determination unit is configured to determine preset vertices in the preset polygonal area, and extract a rectangular area or a square area of ​​a preset size based on the preset vertices as the to-be-detected area;

[0039] A region division unit is configured to divide the region to be detected into a first region and a second region based on a diagonal line of the region to be detected, wherein the first region is a region located above the diagonal line, and the second region is a region located below the diagonal line;

[0040] A pixel mean value determining unit is configured to determine, in the binarized image, a pixel mean value of the first region and a pixel mean value of the second region;

[0041] A judgment data determination unit is configured to determine a ratio value of a pixel mean value of the first area to a pixel mean value of the second area as a first ratio value;

[0042] The first dialog box determination unit is configured to determine, when the first ratio value is greater than a judgment threshold, that the preset polygonal area is a dialog box including a triangle-shaped sharp corner.

[0043] Optionally, the dialog box determination unit is configured to execute:

[0044] When the diagonal line is a diagonal line between the lower left corner and the upper right corner of the area to be detected, and the first ratio value is greater than the judgment threshold, determining that the polygonal area is a dialog box including a first triangular corner;

[0045] When the diagonal line is a diagonal line between the upper left corner and the lower right corner of the area to be detected, and the first ratio value is greater than the judgment threshold, it is determined that the preset polygonal area is a dialog box including a second triangular corner.

[0046] Optionally, the judgment threshold is determined by the following steps:

[0047] For a plurality of image samples of dialog boxes including triangle-shaped sharp corners, respectively determine a pixel mean of a first region of each image sample as a first pixel mean, and respectively determine a pixel mean of a second region of each image sample as a second pixel mean;

[0048] Determine the ratio of the second pixel mean value to the first pixel mean value of each image sample as a second ratio value corresponding to each image sample;

[0049] Determine a plurality of candidate thresholds according to the second ratio value corresponding to each image sample;

[0050] The judgment threshold is determined according to the multiple candidate thresholds.

[0051] Optionally, the dialog detection module includes:

[0052] The second to-be-detected area determination unit is configured to, when the preset polygonal area is a rectangular area, extract a lower boundary range area and / or a right boundary range area of ​​the rectangular area from the to-be-detected video frame image, and use the lower boundary range area and / or the right boundary range area as the to-be-detected area;

[0053] The second dialog box determination unit is configured to determine a pixel mean value of the area to be detected, and when the pixel mean value is within a preset color range, determine that the preset polygonal area is a dialog box including a preset color.

[0054] Optionally, the preset polygon is a rectangle;

[0055] The area determination module comprises:

[0056] A polygonal approximation unit is configured to perform polygonal approximation processing on the connected area, determine the polygonal boundary of the connected area, and determine the polygonal area in the connected area based on the polygonal boundary of the connected area;

[0057] An angle determination unit is configured to determine the angle between two intersecting sides of the polygonal area when the polygonal area has four polygonal boundaries;

[0058] The rectangular area determination unit is configured to determine that the polygonal area is a rectangular area when the angle between the two intersecting sides is within a preset degree range.

[0059] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0060] processor;

[0061] a memory for storing instructions executable by the processor;

[0062] The processor is configured to execute the instructions to implement the dialog box detection method as described in the first aspect.

[0063] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the dialog box detection method as described in the first aspect.

[0064] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program or computer instructions, which, when executed by a processor, implements the dialog box detection method described in the first aspect.

[0065] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0066] The disclosed embodiment performs binarization processing on a video frame image to be detected to obtain a binarized image, performs region detection on the binarized image to obtain a connected region in the binarized image, determines a connected region in the connected region that is a preset polygon as the preset polygon region, and when a preset shape and / or a preset color exists at a preset position of the preset polygon region, determines the preset polygon region to be a dialog box of a preset dialog box type. After the preset polygon region is determined, the dialog box of the preset dialog box type is detected by detecting the preset shape and the preset color, so that the dialog box with local details can be accurately detected, thereby improving the accuracy of detection, and there is no need to use a deep learning method for detection, and there is no need to manually annotate data, thereby reducing labor costs.

[0067] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0069] Figure 1a-1c is a schematic diagram of the style of the dialog box to be detected in the present disclosure;

[0070] Figure 2a-2b is a schematic diagram of the style of a common dialog box in the present disclosure;

[0071] Figure 3 is a flow chart of a dialog box detection method according to an exemplary embodiment;

[0072] Figure 4a-4bis a schematic diagram of dividing the area to be detected in an embodiment of the present disclosure;

[0073] Figure 5 is a block diagram of a dialog detection device according to an exemplary embodiment;

[0074] Figure 6 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0075] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0076] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0077] Figure 1a-1c is a schematic diagram of the style of the dialog box to be detected in the present disclosure, Figure 2a-2b is a schematic diagram of the style of a common dialog box in the present disclosure. The common dialog box may be a dialog box used by the application itself, such as Figure 1a-1c and Figure 2a-2b As shown, the dialog box to be detected has little difference in appearance from the ordinary dialog box, and some even have only slight differences. If a deep learning detection network is used, these slight differences cannot be distinguished and accurate detection cannot be performed.

[0078] Although the objects to be detected and the non-detected objects are very similar in appearance, they are still different in some details, so a combination of some image processing algorithms can be used to design a reasonable detection method. The embodiment of the present disclosure provides the following processing logic flow to solve this problem.

[0079] Figure 3 is a flow chart of a dialog box detection method according to an exemplary embodiment. Figure 3 As shown, the dialog box detection method is used in electronic devices such as servers and computers, and includes the following steps.

[0080] In step S31, the video frame image to be detected is binarized to obtain a binarized image.

[0081] Wherein, the video frame image to be detected is a color image.

[0082] Firstly, the video frame image to be detected is binarized to convert the video frame image to be detected into a binary image, that is, the color video frame image is converted into a black and white image to obtain a binary image.

[0083] In an exemplary embodiment, binarizing the video frame image to be detected to obtain a binarized image includes: performing grayscale processing on the video frame image to be detected to obtain a grayscale image of the video frame image to be detected; and binarizing the grayscale image to obtain the binarized image.

[0084] First, grayscale processing is performed on the video frame image to be detected, and the three-channel color image is converted into a single-channel grayscale image. By converting to a grayscale image, the interference of some noise can be eliminated. Then, based on the grayscale value threshold, the grayscale image is converted into a binary image, that is, the pixel points in the grayscale image whose grayscale value is greater than the grayscale value threshold are assigned a value of 255, and the pixel points in the grayscale image whose grayscale value is less than the grayscale value threshold are assigned a value of 0, thereby obtaining a binary image. Among them, the grayscale value threshold can be set based on needs. For example, if the color of the dialog box is generally light colors such as white and beige, the grayscale value threshold can be set to 250, etc.

[0085] By converting the video frame image to be detected into a binary image and then performing subsequent dialog box detection, the interference caused by some irrelevant pixel distribution can be eliminated and the accuracy of detection can be improved.

[0086] In step S32, region detection is performed on the binary image to obtain connected regions in the binary image.

[0087] Use the region detection algorithm to perform region detection on the binary image, traverse the binary image, determine the connected regions in the binary image, and obtain the positions of each connected region. The region detection algorithm can use the opencv region detection algorithm.

[0088] In step S33, a connected area in the connected area that is a preset polygon is determined as a preset polygon area.

[0089] The preset polygon is the area shape of a general dialog box. There are some relatively small differences between the dialog box to be detected and the general dialog box, generally there is a preset shape and / or preset color around the preset polygon, so the position of the general dialog box needs to be found first, that is, the preset polygon area needs to be found first.

[0090] Each connected area is processed separately to determine whether it is a preset polygonal area. The connected area is processed by polygon approximation to obtain the boundary of the connected area. If the boundaries in the connected area intersect each other to form a polygon, then based on the characteristics of the preset polygon, it is determined whether the polygon formed by the boundary of the connected area is a preset polygon. If the polygon formed by the boundary of the connected area is a preset polygon, the connected area is determined to be a preset polygonal area. If the polygon formed by the boundary of the connected area is not a preset polygon, the connected area can be excluded. After the above processing, the preset polygonal area in the connected area can be obtained.

[0091] In an exemplary embodiment, the preset polygon is a rectangle;

[0092] Determining a region in the connected region that is a preset polygon as the preset polygon region includes:

[0093] Performing polygonal approximation processing on the connected area to determine the polygonal boundary of the connected area, and determining the polygonal area in the connected area based on the polygonal boundary of the connected area;

[0094] When the polygonal region has four polygonal boundaries, determining the angle between two intersecting edges of the polygonal region;

[0095] If the angle between the two intersecting edges is within a preset degree range, the polygonal area is determined to be a rectangular area.

[0096] When the shape of the dialog box used by the application itself is a rectangle, the preset polygon is a rectangle. When determining whether the connected area is a rectangular area, polygon approximation processing is first performed on the connected area to determine the polygonal boundary of each connected area, determine the intersection of two of the multiple polygonal boundaries of a connected area, and the area surrounded by the multiple polygonal boundaries is the polygonal area in the connected area. The polygonal area with four polygonal boundaries is screened out from the polygonal area, and the polygonal area with less than or more than four polygonal boundaries is filtered out. The angle between the two intersecting edges of the polygonal area with four polygonal boundaries is calculated. If the angle between the two intersecting edges is within the preset degree range, the polygonal area is determined to be a rectangular area. The preset degree range is 90 degrees plus or minus the preset degree, and the preset degree can be, for example, 10 degrees or 15 degrees.

[0097] Based on polygon approximation, boundary quantity judgment and angle judgment, roughly rectangular areas can be screened out to filter out the application's own dialog boxes and dialog boxes similar to the dialog boxes used by the application itself, so that approximately rectangular dialog boxes can be detected more accurately.

[0098] In step S34, when a preset shape and / or a preset color exists at a preset position of the preset polygonal area, it is determined that the preset polygonal area is a dialog box of a preset dialog box type.

[0099] Since a dialog box of a preset dialog box type to be detected generally has a preset shape and / or a preset color at a preset position in a preset polygonal area, a corresponding detection method may be set for a dialog box of the preset dialog box type.

[0100] After determining the preset polygonal area, at least one detection algorithm of the preset dialog box type can be used to detect the preset polygonal area to determine whether the preset polygonal area is a dialog box of the preset dialog box type, and determine the specific preset dialog box type. First, determine the preset position of the preset polygonal area, determine an area near the preset position, detect whether there is a preset shape and / or preset color in the area, and if there is a preset shape and / or preset color in the area, determine that the preset polygonal area is a dialog box of the preset dialog box type, and obtain the specific preset dialog box type based on the preset shape and / or preset color in the area. The preset dialog box type can be a preset shape, a preset color, or a preset shape and a preset color.

[0101] In an exemplary embodiment, when a preset shape and / or a preset color exists at a preset position of the preset polygonal area, determining that the preset polygonal area is a dialog box of a preset dialog box type includes:

[0102] Determine preset vertices in the preset polygonal area, and extract a rectangular area or a square area of ​​a preset size based on the preset vertices as the area to be detected;

[0103] Based on the diagonal of the area to be detected, the area to be detected is divided into a first area and a second area, wherein the first area is an area located above the diagonal, and the second area is an area located below the diagonal;

[0104] In the binary image, determining a pixel mean of the first region and a pixel mean of the second region;

[0105] Determine a ratio of a pixel mean value of the first area to a pixel mean value of the second area as a first ratio value;

[0106] When the first ratio value is greater than a judgment threshold, the preset polygonal area is determined to be a dialog box including a triangle-shaped sharp corner.

[0107] If the preset dialog type has a triangular corner near a preset vertex, such as Figure 1a and Figure 1bAs shown, firstly, based on the position of the preset polygonal area, the position of the preset vertex in the preset polygonal area is determined (for example, when the preset polygonal area is a rectangular area or a hexagonal area, the preset vertex can be one of the two vertices of a side below), and then Figure 1a and Figure 1b , the preset polygonal area is a rectangular area, and the preset vertex is the lower left vertex of the rectangular area. After determining the preset vertex in the preset polygonal area, a rectangular area or a square area of ​​a preset size including the preset vertex is extracted, and the extracted rectangular area or square area is used as the area to be detected to determine whether there is a triangular sharp corner in the area to be detected. Among them, the preset size can be set according to the size of the triangular sharp corner to be detected, for example, it can be 10×10 pixels, 20×20 pixels, 30×30 pixels, etc. When extracting a rectangular area or a square area of ​​a preset size including the preset vertex, the preset vertex can be used as a vertex of the rectangular area or the square area, or the preset vertex can also be inside the rectangular area or the square area and at a preset pixel position away from one of the vertices. For example, for Figure 1a and Figure 1b In the dialog box, when the extracted area is a square area, the preset vertex (i.e., the lower left vertex) of the rectangular area can be used as the upper left vertex of the square area to be extracted, and the square area can be extracted based on the preset size. Alternatively, the preset vertex (i.e., the lower left vertex of the rectangular area) can be used as the upper left vertex of the square area to be extracted, and the square area can be extracted based on the preset size. When extracting the area to be detected, whether the preset vertex is used as a vertex of the area to be detected or located inside the area to be detected, the preset vertex can be compared with a vertex of the area to be detected (such as Figure 1a and Figure 1b In the dialog box shown, a vertex of the area to be detected is the upper left corner vertex) is set as a hyperparameter, and the hyperparameter is dynamically adjusted in advance according to the dialog box to be detected to determine a more appropriate value.

[0108] After the area to be detected is extracted, the area to be detected is divided into a first area and a second area based on the diagonal of the area to be detected. Based on the type of the sharp corner of the triangle shape to be detected, the area to be detected is divided into the first area and the second area using the diagonal corresponding to the type. The area above the diagonal is the first area, and the area below the diagonal is the second area. Figure 1a The dialog box shown in the figure uses the diagonal line between the upper left corner vertex and the lower right corner vertex of the area to be detected to divide the area to be detected. Figure 1b The dialog box shown divides the area to be detected by using the diagonal line between the lower left vertex and the upper right vertex of the area to be detected.

[0109] After dividing the area to be detected into a first area and a second area, in the binary image, the pixel mean of the first area and the pixel mean of the second area are counted to determine the ratio of the pixel mean of the first area to the pixel mean of the second area, and the ratio is a first ratio. If the first ratio is greater than a judgment threshold, the preset polygonal area is determined to be a dialog box including sharp corners of a triangle.

[0110] It should be noted that, when the preset polygonal area is determined to be a dialog box including a triangular corner when the first ratio value is greater than the judgment threshold, it is based on the fact that the dialog box area is processed as a white area in the binary image.

[0111] By extracting the area to be detected near the preset vertex and dividing the area to be detected into a first area and a second area based on the diagonal, the dialog box including a triangular sharp corner can be accurately judged based on the pixel mean of the first area and the pixel mean of the second area, thereby improving the judgment accuracy of the dialog box with a triangular sharp corner.

[0112] In an exemplary embodiment, when the first ratio value is greater than a judgment threshold, determining that the preset polygonal area is a dialog box including a triangle-shaped sharp corner includes:

[0113] When the diagonal line is a diagonal line between the lower left corner and the upper right corner of the area to be detected, and the first ratio value is greater than the judgment threshold, determining that the polygonal area is a dialog box including a first triangular corner;

[0114] When the diagonal line is a diagonal line between the upper left corner and the lower right corner of the area to be detected, and the first ratio value is greater than the judgment threshold, it is determined that the preset polygonal area is a dialog box including a second triangular corner.

[0115] The first type of triangle shape has sharp corners such as Figure 1a As shown, the second triangle shape has sharp corners such as Figure 1b When determining whether the first type of triangle shape is included, the diagonal line between the lower left corner and the upper right corner of the area to be detected is used to divide the area to be detected, such as Figure 4a As shown, if the first ratio value is greater than the judgment threshold, the polygonal area is determined to be a dialog box including the first type of triangle-shaped sharp corners. When judging whether the second type of triangle-shaped sharp corners are included, the diagonal line between the upper left corner and the lower right corner of the area to be detected is used to divide the area to be detected, such as Figure 4b As shown, if the first ratio value is greater than the judgment threshold, the polygonal area is determined to be a dialog box including a second type of triangle-shaped sharp corners. Different types of triangle-shaped sharp corners dialog boxes can be distinguished by different diagonal lines, and the corresponding type of dialog box can be accurately determined.

[0116] In an exemplary embodiment, the judgment threshold is determined by the following steps, including: for multiple image samples including dialog boxes with triangular corners, respectively determining the pixel mean of the first area of ​​each image sample as the first pixel mean, and respectively determining the pixel mean of the second area of ​​each image sample as the second pixel mean; respectively determining the ratio of the second pixel mean to the first pixel mean of each image sample as the second ratio value corresponding to each image sample; determining multiple candidate thresholds according to the second ratio value corresponding to each image sample; and determining the judgment threshold according to the multiple candidate thresholds.

[0117] An image sample of a dialog box including a plurality of triangle-shaped sharp corners is obtained. The position of the dialog box with the triangle-shaped sharp corners may be marked in the image sample. Based on the position, an area to be detected is determined. Based on the diagonal of the area to be detected, the area to be detected is divided into a first area and a second area. The first area is an area above the diagonal, and the second area is an area below the diagonal. In a binarized image corresponding to the image sample, when the dialog box is binarized as a white area, a pixel mean of the first area of ​​each image sample is determined as the first pixel mean, and a pixel mean of the second area of ​​each image sample is determined as the second pixel mean. The ratio of the second pixel mean of each image sample to the first pixel mean is determined as a second ratio value corresponding to each image sample, and the difference between 1 and each second ratio value is determined as a candidate threshold value. That is, the candidate threshold value corresponding to each image sample is determined according to the following formula:

[0118] thr=1-P bottom / P top

[0119] Among them, thr is the candidate threshold corresponding to an image sample, P top is the first pixel mean of the image sample, P bottom is the second pixel mean of the image sample, P bottom / P top is the second ratio value corresponding to the image sample.

[0120] According to the above formula, multiple candidate thresholds are obtained based on multiple image samples, and the distribution of multiple candidate thresholds is determined. The range of most (such as more than 50%) candidate thresholds is statistically determined, and the judgment threshold is determined based on the range. For example, for 100 image samples, 50 image samples have second ratio values ​​between 0.88-0.92, while the second ratio values ​​of other image samples are relatively scattered, then the judgment threshold can be determined to be 0.9. By dynamically determining the judgment threshold based on the first pixel mean and the second pixel mean of multiple image samples, the accuracy of dialog type judgment can be improved.

[0121] In another exemplary embodiment, when a preset shape and / or a preset color exists at a preset position of the preset polygonal area, the preset polygonal area is determined to be a dialog box of a preset dialog box type, including: when the preset polygonal area is a rectangular area, extracting a lower boundary range area and / or a right boundary range area of ​​the rectangular area from the video frame image to be detected, and using the lower boundary range area and / or the right boundary range area as the area to be detected; determining a pixel mean of the area to be detected, and when the pixel mean is within a preset color range, determining that the preset polygonal area is a dialog box including a preset color.

[0122] Figure 1c The dialog box shown is a rectangular dialog box, and there will be a specific color near the lower boundary and the right boundary. When detecting this type of dialog box, the lower boundary range area and / or the right boundary range area of ​​the rectangular area are extracted from the video frame image to be detected, and the extracted lower boundary range area and / or the right boundary range area are used as the area to be detected. Since it is necessary to detect whether it contains a preset color, the area to be detected is not extracted from the binary image, but from the video frame image to be detected (i.e., the original color image). The pixel mean of the area to be detected is counted to determine whether the pixel mean is within the preset color range. If the pixel mean is within the preset color range, the preset polygonal area is determined to be a dialog box including the preset color. If the pixel mean is not within the preset color range, it is determined that the preset polygonal area is not a dialog box including the preset color. By counting the pixel mean of the lower boundary range area and / or the right boundary range area of ​​the preset polygonal area, the dialog box with the preset color at the preset position can be accurately judged, thereby improving the judgment accuracy of the dialog box with the preset color.

[0123] The dialog box detection method provided by the present exemplary embodiment performs binarization processing on the video frame image to be detected to obtain a binarized image, performs region detection on the binarized image to obtain a connected region in the binarized image, determines a connected region in the connected region that is a preset polygon as the preset polygon region, and if a preset shape and / or preset color exists at a preset position of the preset polygon region, determines that the preset polygon region is a dialog box of a preset dialog box type. After the preset polygon region is determined, the dialog box of the preset dialog box type is detected by detecting the preset shape and preset color, so that a dialog box with local details can be accurately detected, thereby improving the accuracy of detection, and does not require the use of deep learning methods for detection, and does not require manual data annotation, thereby reducing labor costs.

[0124] Figure 5 is a block diagram of a dialog detection device according to an exemplary embodiment. Figure 5The device includes a binarization processing module 51, a region detection module 52, a region determination module 53 and a dialog detection module 54.

[0125] The binarization processing module 51 is configured to perform binarization processing on the video frame image to be detected to obtain a binarized image;

[0126] The region detection module 52 is configured to perform region detection on the binary image to obtain connected regions in the binary image;

[0127] The region determination module 53 is configured to determine a connected region of the connected region that is a preset polygon as a preset polygon region;

[0128] The dialog detection module 54 is configured to determine that the preset polygonal area is a dialog of a preset dialog type when a preset shape and / or preset color exists at a preset position of the preset polygonal area.

[0129] Optionally, the dialog detection module includes:

[0130] A first to-be-detected area determination unit is configured to determine preset vertices in the preset polygonal area, and extract a rectangular area or a square area of ​​a preset size based on the preset vertices as the to-be-detected area;

[0131] A region division unit is configured to divide the region to be detected into a first region and a second region based on a diagonal line of the region to be detected, wherein the first region is a region located above the diagonal line, and the second region is a region located below the diagonal line;

[0132] A pixel mean value determining unit is configured to determine, in the binarized image, a pixel mean value of the first region and a pixel mean value of the second region;

[0133] A judgment data determination unit is configured to determine a ratio value of a pixel mean value of the first area to a pixel mean value of the second area as a first ratio value;

[0134] The first dialog box determination unit is configured to determine, when the first ratio value is greater than a judgment threshold, that the preset polygonal area is a dialog box including a triangle-shaped sharp corner.

[0135] Optionally, the dialog box determination unit is configured to execute:

[0136] When the diagonal line is a diagonal line between the lower left corner and the upper right corner of the area to be detected, and the first ratio value is greater than the judgment threshold, determining that the polygonal area is a dialog box including a first triangular corner;

[0137] When the diagonal line is a diagonal line between the upper left corner and the lower right corner of the area to be detected, and the first ratio value is greater than the judgment threshold, it is determined that the preset polygonal area is a dialog box including a second triangular corner.

[0138] Optionally, the judgment threshold is determined by the following steps:

[0139] For a plurality of image samples of dialog boxes including triangle-shaped sharp corners, respectively determine a pixel mean of a first region of each image sample as a first pixel mean, and respectively determine a pixel mean of a second region of each image sample as a second pixel mean;

[0140] Determine the ratio of the second pixel mean value to the first pixel mean value of each image sample as a second ratio value corresponding to each image sample;

[0141] Determine a plurality of candidate thresholds according to the second ratio value corresponding to each image sample;

[0142] The judgment threshold is determined according to the multiple candidate thresholds.

[0143] Optionally, the dialog detection module includes:

[0144] The second to-be-detected area determination unit is configured to, when the preset polygonal area is a rectangular area, extract a lower boundary range area and / or a right boundary range area of ​​the rectangular area from the to-be-detected video frame image, and use the lower boundary range area and / or the right boundary range area as the to-be-detected area;

[0145] The second dialog box determination unit is configured to determine a pixel mean value of the area to be detected, and when the pixel mean value is within a preset color range, determine that the preset polygonal area is a dialog box including a preset color.

[0146] Optionally, the preset polygon is a rectangle;

[0147] The area determination module comprises:

[0148] A polygonal approximation unit is configured to perform polygonal approximation processing on the connected area, determine the polygonal boundary of the connected area, and determine the polygonal area in the connected area based on the polygonal boundary of the connected area;

[0149] An angle determination unit is configured to determine the angle between two intersecting sides of the polygonal area when the polygonal area has four polygonal boundaries;

[0150] The rectangular area determination unit is configured to determine that the polygonal area is a rectangular area when the angle between the two intersecting sides is within a preset degree range.

[0151] Optionally, the binarization processing module includes:

[0152] A grayscale processing unit is configured to perform grayscale processing on the video frame image to be detected to obtain a grayscale image of the video frame image to be detected;

[0153] The binarization processing unit is configured to perform binarization processing on the grayscale image to obtain the binarized image.

[0154] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0155] Figure 6 600 is a block diagram of an electronic device according to an exemplary embodiment. For example, the electronic device 600 may be provided as a server. Figure 6 , the electronic device 600 includes a processing component 622, which further includes one or more processors, and a memory resource represented by a memory 632 for storing instructions, such as applications, that can be executed by the processing component 622. The application stored in the memory 632 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 622 is configured to execute instructions to perform the above-mentioned dialog detection method.

[0156] The electronic device 600 may also include a power supply component 626 configured to perform power management of the electronic device 600, a wired or wireless network interface 650 configured to connect the electronic device 600 to a network, and an input / output (I / O) interface 658. The electronic device 600 may operate based on an operating system stored in the memory 632, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

[0157] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 632 including instructions, and the above instructions can be executed by the processing component 622 of the electronic device 600 to complete the above dialog detection method. Optionally, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0158] In an exemplary embodiment, a computer program product is also provided, including a computer program or a computer instruction, wherein the computer program or the computer instruction implements the above-mentioned dialog box detection method when executed by a processor.

[0159] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0160] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A dialog box detection method, characterized in that: include: Perform binarization processing on the video frame image to be detected to obtain a binarized image; Performing region detection on the binary image to obtain connected regions in the binary image; Determine a connected area of ​​the connected area that is a preset polygon as a preset polygon area; When a preset shape and / or a preset color exists at a preset position of the preset polygonal area, determining that the preset polygonal area is a dialog box of a preset dialog box type includes: When the preset polygonal area is a rectangular area, the pixel mean of the area to be detected is determined, and when the pixel mean is within a preset color range, the preset polygonal area is determined to be a dialog box including a preset color.

2. The method according to claim 1, characterized in that When a preset shape and / or a preset color exists at a preset position of the preset polygonal area, determining that the preset polygonal area is a dialog box of a preset dialog box type includes: Determine preset vertices in the preset polygonal area, and extract a rectangular area or a square area of ​​a preset size based on the preset vertices as the area to be detected; Based on the diagonal of the area to be detected, the area to be detected is divided into a first area and a second area, wherein the first area is an area located above the diagonal, and the second area is an area located below the diagonal; In the binary image, determining a pixel mean of the first region and a pixel mean of the second region; Determine a ratio of a pixel mean value of the first area to a pixel mean value of the second area as a first ratio value; When the first ratio value is greater than a judgment threshold, the preset polygonal area is determined to be a dialog box including a triangle-shaped sharp corner.

3. The method according to claim 2, characterized in that When the first ratio value is greater than a judgment threshold, determining that the preset polygonal area is a dialog box including a triangle-shaped sharp corner includes: When the diagonal line is a diagonal line between the lower left corner and the upper right corner of the area to be detected, and the first ratio value is greater than the judgment threshold, determining that the polygonal area is a dialog box including a first triangular corner; When the diagonal line is a diagonal line between the upper left corner and the lower right corner of the area to be detected, and the first ratio value is greater than the judgment threshold, it is determined that the preset polygonal area is a dialog box including a second triangular corner.

4. The method according to claim 2, characterized in that: The judgment threshold is determined by the following steps: For a plurality of image samples of dialog boxes including triangle-shaped sharp corners, respectively determine a pixel mean of a first region of each image sample as a first pixel mean, and respectively determine a pixel mean of a second region of each image sample as a second pixel mean; Determine the ratio of the second pixel mean value to the first pixel mean value of each image sample as a second ratio value corresponding to each image sample; Determine a plurality of candidate thresholds according to the second ratio value corresponding to each image sample; The judgment threshold is determined according to the multiple candidate thresholds.

5. The method according to claim 1, characterized in that When a preset shape and / or a preset color exists at a preset position of the preset polygonal area, determining that the preset polygonal area is a dialog box of a preset dialog box type includes: When the preset polygonal area is a rectangular area, extracting a lower boundary range area and / or a right boundary range area of ​​the rectangular area from the video frame image to be detected, and using the lower boundary range area and / or the right boundary range area as the area to be detected; The pixel mean value of the area to be detected is determined, and when the pixel mean value is within a preset color range, the preset polygonal area is determined to be a dialog box including a preset color.

6. The method according to any one of claims 1 to 5, characterized in that: The preset polygon is a rectangle; Determining a region in the connected region that is a preset polygon as the preset polygon region includes: Performing polygonal approximation processing on the connected area to determine the polygonal boundary of the connected area, and determining the polygonal area in the connected area based on the polygonal boundary of the connected area; When the polygonal region has four polygonal boundaries, determining the angle between two intersecting edges of the polygonal region; When the angle between the two intersecting edges is within a preset degree range, the polygonal area is determined to be a rectangular area.

7. A dialog box detection device, characterized in that: include: A binarization processing module is configured to perform binarization processing on the video frame image to be detected to obtain a binarized image; A region detection module is configured to perform region detection on the binary image to obtain a connected region in the binary image; A region determination module is configured to determine a connected region of the connected region that is a preset polygon as a preset polygon region; A dialog detection module is configured to determine that the preset polygonal area is a dialog of a preset dialog type when a preset shape and / or a preset color exists at a preset position of the preset polygonal area; The dialog detection module comprises: The second dialog determination unit is configured to determine the pixel mean of the area to be detected when the preset polygonal area is a rectangular area, and determine that the preset polygonal area is a dialog box including a preset color when the pixel mean is within a preset color range.

8. The device according to claim 7, characterized in that The dialog detection module comprises: A first to-be-detected area determination unit is configured to determine preset vertices in the preset polygonal area, and extract a rectangular area or a square area of ​​a preset size based on the preset vertices as the to-be-detected area; A region division unit is configured to divide the region to be detected into a first region and a second region based on a diagonal line of the region to be detected, wherein the first region is a region located above the diagonal line, and the second region is a region located below the diagonal line; A pixel mean value determining unit is configured to determine, in the binarized image, a pixel mean value of the first region and a pixel mean value of the second region; A judgment data determination unit is configured to determine a ratio value of a pixel mean value of the first area to a pixel mean value of the second area as a first ratio value; The first dialog box determination unit is configured to determine, when the first ratio value is greater than a judgment threshold, that the preset polygonal area is a dialog box including a triangle-shaped sharp corner.

9. The device according to claim 8, characterized in that The dialog determination unit is configured to execute: When the diagonal line is a diagonal line between the lower left corner and the upper right corner of the area to be detected, and the first ratio value is greater than the judgment threshold, determining that the polygonal area is a dialog box including a first triangular corner; When the diagonal line is a diagonal line between the upper left corner and the lower right corner of the area to be detected, and the first ratio value is greater than the judgment threshold, it is determined that the preset polygonal area is a dialog box including a second triangular corner.

10. The device according to claim 8, characterized in that The judgment threshold is determined by the following steps: For a plurality of image samples of dialog boxes including triangle-shaped sharp corners, respectively determine a pixel mean of a first region of each image sample as a first pixel mean, and respectively determine a pixel mean of a second region of each image sample as a second pixel mean; Determine a ratio value of the second pixel mean value of each image sample to the first pixel mean value as a second ratio value corresponding to each image sample; Determine a plurality of candidate thresholds according to the second ratio value corresponding to each image sample; The judgment threshold is determined according to the multiple candidate thresholds.

11. The device according to claim 7, characterized in that The dialog detection module comprises: The second to-be-detected area determination unit is configured to, when the preset polygonal area is a rectangular area, extract a lower boundary range area and / or a right boundary range area of ​​the rectangular area from the to-be-detected video frame image, and use the lower boundary range area and / or the right boundary range area as the to-be-detected area; The second dialog box determination unit is configured to determine a pixel mean value of the area to be detected, and when the pixel mean value is within a preset color range, determine that the preset polygonal area is a dialog box including a preset color.

12. The device according to any one of claims 7 to 11, characterized in that: The preset polygon is a rectangle; The area determination module comprises: A polygonal approximation unit is configured to perform polygonal approximation processing on the connected area, determine the polygonal boundary of the connected area, and determine the polygonal area in the connected area based on the polygonal boundary of the connected area; An angle determination unit is configured to determine the angle between two intersecting sides of the polygonal area when the polygonal area has four polygonal boundaries; The rectangular area determination unit is configured to determine that the polygonal area is a rectangular area when the angle between the two intersecting sides is within a preset degree range.

13. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the dialog box detection method as described in any one of claims 1 to 6.

14. A computer-readable storage medium, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the dialog detection method according to any one of claims 1 to 6.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the dialog detection method according to any one of claims 1 to 6 is implemented.