Method for processing structural diagram containing text information, electronic device, storage medium

Abstract

The application discloses a processing method of a structure diagram containing text information, and the method comprises the following steps: performing contour detection based on the line segment composition of the structure diagram containing text information; fitting a geometric polygon according to the detected contour; filtering out shapes not meeting the feature condition of the text box of the structure diagram from the geometric polygon to obtain a frame. The application can efficiently and accurately extract the frame from the structure diagram containing text information.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image data processing, and in particular to a processing method of a structure diagram containing text information, an electronic device and a storage medium. BACKGROUND

[0002] With the popularization and development of Internet applications, big data technology and the like, the processing demand for image data is increasing. Among them, the data processing of a structure diagram with certain structural characteristics is a typical demand. For example, a VIE (Variable Interest Entities) structure diagram, a flowchart, a module diagram and the like. Such image data contains more abstract and complex information compared with text data. How to automatically analyze these images becomes a problem to be solved by relevant practitioners. SUMMARY

[0003] In view of this, the present application discloses a processing method of a structure diagram containing text information, an electronic device and a storage medium, to accurately extract a visual bounding box from the structure diagram containing text information.

[0004] To achieve the above-mentioned target, the technical solution adopted by the present application is as follows:

[0005] The first aspect of the embodiment of the present application discloses a processing method of a structure diagram containing text information, comprising:

[0006] Performing contour detection based on line segment composition of the structure diagram containing text information;

[0007] Fitting a geometric polygon according to the detected contour;

[0008] Filtering out shapes not meeting a feature condition of a text box of the structure diagram from the geometric polygon based on the feature condition, to obtain a bounding box.

[0009] Further optionally, the method further comprises:

[0010] Performing line segment detection based on the structure diagram to obtain initial line segments;

[0011] Performing line segment processing based on the initial line segments to obtain line segment composition of the structure diagram;

[0012] The line segment processing comprises: filtering the initial line segments based on a length constraint condition and an angle constraint condition, and extending the filtered initial line segments based on a length extension condition.

[0013] Further optionally, the fitting of the geometric polygon according to the detected contour comprises: fitting the geometric polygon based on an outer contour and / or an inner contour.

[0014] Further optionally, the feature condition comprises: an area greater than an area threshold; a number of edges satisfying a set condition; a shape being a convex quadrilateral and a distance between end points and an included angle of adjacent edges satisfying a set condition.

[0015] Further optionally, the method further comprises: filtering the line segment composition based on at least the bounding box to obtain to-be-clustered line segments; clustering the to-be-clustered line segments to obtain at least one line segment cluster, the line segments in the line segment cluster being connected to each other; and determining, for any single line segment cluster in the at least one line segment cluster, a first attribute of a bounding box associated with the single line segment cluster according to a positional relationship between the single line segment cluster and the bounding box, the first attribute being used to distinguish whether the bounding box associated with the single line segment cluster is a source bounding box or a target bounding box relative to the single line segment cluster.

[0016] Further optionally, the filtering the line segment composition based on at least the bounding box to obtain to-be-clustered line segments comprises: removing, from the line segment composition, associated line segments of the bounding box and line segments within the bounding box, the associated line segments of the bounding box comprising: edges of the bounding box and edges similar to the edges of the bounding box; or further comprising: removing a horizontal line with the longest length

[0017] Further optionally, the determining, for any single line segment cluster in the at least one line segment cluster, a first attribute of a bounding box associated with the single line segment cluster according to a positional relationship between the single line segment cluster and the bounding box comprises: selecting, from a target box located above the single line segment cluster, a midpoint of an edge closest to an upper end point of the single line segment cluster as a source node of the single line segment cluster, the bounding box corresponding to the source node being a source bounding box of the single line segment cluster; and selecting, from a target box located below the single line segment cluster, a midpoint of an edge closest to a lower end point of the single line segment cluster as a target node, the bounding box corresponding to the target node being a target bounding box of the single line segment cluster; wherein the number of the upper end point and the source node is one, and the lower end point and the target node are one-to-one corresponding.

[0018] Further optionally, the method further comprises: performing a morphological opening operation on a binary image of the structure diagram to obtain a first image; performing a top-hat processing on the first image and the binary image to obtain a second image; performing a straight line detection based on the second image, and selecting a line segment with a length meeting a requirement from a detection result as a target line segment; and determining a second attribute of the bounding box according to a positional relationship of the bounding box relative to the target line segment, the second attribute being used to distinguish a position of the bounding box.

[0019] Further optionally, the method further comprises: performing a morphological opening operation on the binarized graph of the structure graph to obtain a first graph; performing a top-hat processing on the first graph and the binarized graph to obtain a second graph; performing a bitwise NOT operation on the second graph to obtain a third graph; performing a top-hat processing on the third graph and the binarized graph to obtain a fourth graph; performing a straight line detection on the fourth graph, and if no line segment is detected, performing a straight line detection on the second graph; selecting a line segment meeting a length criterion from the detection result as the target line segment; and determining a second attribute of the border according to a positional relationship of the border relative to the target line segment, the second attribute being used to distinguish the position of the border.

[0020] Further optionally, the method further comprises: determining associated content of the border based on an association relationship between a text information box of a text recognition result of the structure graph and the border.

[0021] Further optionally, the determining the associated content of the border based on the association relationship between the text recognition result of the structure graph and the border comprises: aggregating a first type of text in the text recognition result and the border to determine a third attribute corresponding to each of the borders; and / or, judging a distance between a second type of text in the text recognition result and the border to determine a fourth attribute corresponding to each of the borders; wherein the first type of text comprises a name type of text, the second type of text comprises a numerical value type of text, the third attribute is used to distinguish the name of the border, and the fourth attribute is used to distinguish the numerical value of the border.

[0022] Further optionally, the method further comprises: storing the determined attributes of the borders in a data table, the data table being configured with fields corresponding to the attributes of the borders.

[0023] A second aspect of the embodiment of the application discloses an electronic device, comprising:

[0024] a memory configured to store computer instructions or computer programs;

[0025] a processor configured to invoke and execute the computer instructions or computer programs stored in the memory, so as to implement the method provided in the second aspect of the embodiment of the application.

[0026] A third aspect of the embodiment of the application discloses a computer storage medium storing computer instructions or computer programs, the computer instructions or computer programs being executed to implement the method provided in the second aspect of the embodiment of the application.

[0027] Beneficial effects: by using the embodiment of the present application, the edge frame can be extracted from the structural diagram containing text information efficiently and accurately through contour detection, polygon fitting and feature filtering. In combination with the content disclosed in other embodiments of the present application, more extensive beneficial effects can be achieved. BRIEF DESCRIPTION OF DRAWINGS

[0028] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings in which:

[0029] Figure 1 Exemplarily shown is a flowchart of a processing method of a structural diagram containing text information according to an embodiment of the present application;

[0030] Figure 2 Exemplarily shown is a flowchart of a method of obtaining line segment composition of a structural diagram according to an embodiment of the present application;

[0031] Figure 3 Exemplarily shown is a line segment detection result diagram according to an embodiment of the present application;

[0032] Figure 4 Exemplarily shown is a result diagram after the line segment detection result is filtered according to an angle constraint condition according to an embodiment of the present application;

[0033] Figure 5 Exemplarily shown is a contour detection result diagram according to an embodiment of the present application;

[0034] Figure 6a Exemplarily shown is a flowchart of an edge frame attribute confirmation method according to an embodiment of the present application;

[0035] Figure 6b Exemplarily shown is an effect diagram of removing the associated line segments of the edge frame according to an embodiment of the present application;

[0036] Figure 6c Exemplarily shown is an effect diagram of removing the line segments in the edge frame according to an embodiment of the present application;

[0037] Figure 6d Exemplarily shown is an effect diagram of line segment clustering to obtain line segment clusters according to an embodiment of the present application;

[0038] Figure 7a Exemplarily shown is a flowchart of a split line detection method according to an embodiment of the present application;

[0039] Figure 7b An exemplary flow chart of a split line detection method according to an embodiment of the present application is shown;

[0040] Figure 8 An exemplary identified original drawing (left) and an exemplary identified result (right) according to an embodiment of the present application are shown;

[0041] Figure 9a An exemplary field diagram of a name database according to an embodiment of the present application is shown;

[0042] Figure 9b An exemplary field diagram of a spreadsheet according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0043] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0044] The terms used in the embodiments of the present application are merely for the purpose of describing particular embodiments and are not intended to limit the present application. The singular forms "a", "an" and "the" used in the embodiments of the present application and the appended claims are intended to include the plural forms as well, unless the context clearly indicates otherwise. "Plural" generally includes at least two but does not exclude the case of including at least one.

[0045] It should be understood that the term "and / or" used herein is merely an association relationship of the associated objects and represents that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " herein generally represents an "or" relationship between the front and rear associated objects.

[0046] It should also be noted that the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusion, so that a product or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such product or system. Without more limitations, the element defined by the statement "including a" does not exclude the presence of additional identical elements in the product or system including the element.

[0047] Figure 1is a flowchart of a processing method of a structure diagram containing text information according to an embodiment of the present application, wherein the structure diagram containing text information has the following characteristics: multiple text boxes containing visualized frames are contained, at least two (for example, two, three or all) text boxes are connected by lines, and at least one (for example, one, four or all) text box contains text. In a further case, invisible text boxes, dividing lines and the like can also be included. This will be described or shown in detail below. Refer to Figure 1 , the method comprises:

[0048] 100: contour detection is performed based on line segment composition of the structure diagram containing text information. Wherein, the line segment composition of the structure diagram can be determined or obtained in advance.

[0049] 102: a geometric polygon is fitted according to the detected contour.

[0050] 104: shapes not meeting the feature condition of the text box of the structure diagram are filtered out from the geometric polygon based on the feature condition, to obtain a frame.

[0051] By using the method provided by the embodiment of the present application, the frame can be accurately extracted from the structure diagram containing text information by contour detection, polygon fitting and feature filtering, and the line segment interference caused by the text information can be excluded.

[0052] Optionally, in an embodiment of the present application, the line segment composition of the structure diagram is obtained in the manner shown in Figure 2 . As shown in Figure 2 , the line segment detection method comprises:

[0053] 200: line segment detection is performed based on the structure diagram containing text information, to obtain initial line segments.

[0054] For example, first, the structure diagram picture is acquired and is grayed, and then based on the grayed picture, line segment detection is performed by using LSD (Line Segment Detector). LSD can obtain sub-pixel level precision detection results in linear time. The algorithm is designed to be applicable to any digital image without parameter adjustment.

[0055] An exemplary LSD algorithm applied to the present implementation manner has the following steps:

[0056] 1. Gaussian down-sampling is performed on the input image at a sampling rate of s=0.8.

[0057] 2. Gradient value and gradient direction (level-line orientation) of each point are calculated.

[0058] 3. Pseudo-ordered all points according to gradient value, and build a state list, all points are set as UNUSED.

[0059] 4. Set the corresponding position in the state list of the point with gradient value less than the gradient threshold ρ as USED.

[0060] 5. Take out the point with the maximum gradient (the first in the pseudo-ordered list) in the list as the seed point, and set the state list as USED.

[0061] Hereinafter, the VIE structure diagram is taken as an example of the structure diagram containing text information processed by the related embodiments of the present application. Exemplarily, the result after the processing 200 of the present embodiment is processed is as shown in FIG. 6, wherein the thick lines represent the detected initial line segments. Figure 3

[0062] 202: Obtain the line segment composition of the structure diagram based on the line segment processing of the initial line segments. The line segment processing includes: filtering the initial line segments based on the length constraint condition and the angle constraint condition, and extending the filtered initial line segments based on the length extension condition.

[0063] After the processing 200, there are many line segments, and the unnecessary parts need to be filtered out. Therefore, the filtering can be performed based on the angle constraint condition. For example, only the line segments with a set angle or angle range relative to the horizontal line are reserved, and the selection of the angle and the angle range is determined according to the extension angle of the edge of the text box of the structure diagram. In the present implementation, the line segments with 0° and 90° are reserved, and the line segments with the inclined angle are deleted, and the result is as shown in FIG. 7. Figure 4

[0064] Then, the interfering line segments can be further deleted based on the length constraint condition, and the length constraint condition can be that the length of the line segment should be greater than the length threshold. For example, the average value A of all line segments is calculated, and the line segments with the length less than x·A are removed, and exemplarily, x∈[0.2, 0.6], for example, x=0.2, 0.3 or 0.4, etc. The specific value of x can be obtained according to the experiment, or can be optimized according to the feedback of the result after the length constraint condition. In addition, x or A can also adopt the empirical value.

[0065] In other embodiments of the present application, the filtering based on the length constraint condition can be performed first, and then the filtering based on the angle constraint condition can be performed.

[0066] ​​After that, a line_threld representing the length of each line segment is set to extend each line segment. For each line segment, the distance of line_threld is added to both ends of the line segment respectively. According to the experience value, line_threld is usually 3 pixels. That is, 3 pixels are added to the upper and lower ends of the vertical line, and 3 pixels are added to the left and right ends of the horizontal line. The inventor found in the research that the line segment extension can reconnect the line segments that are originally connected but not connected in the detection result, thereby providing more accurate data basis for the subsequent contour detection processing and improving the accuracy and integrity of the subsequent contour detection.

[0067] Optionally, in an embodiment of the present application, the line segments are extended by a line_threld, and the line_threld is set to 3 pixels. Figure 1 The embodiment shown is used as a reference, and the following method is used for contour detection in the processing 100: first, a new blank graph is established; then, the calculated line segments are drawn in the blank graph to obtain a binary graph, and then a contour detection algorithm is used to find all the contours in the binary graph. The related content of an exemplary contour detection algorithm is explained as follows:

[0068] NBD: A boundary can be obtained from the boundary starting point by a boundary tracking algorithm, and a new unique number is assigned to each newly found boundary B. NBD represents the number of the currently tracked boundary.

[0069] LNBD: In the process of raster scanning, the number of the last encountered (previous) boundary B' is recorded as LNBD.

[0070] Define the input picture F=f\{i,j\}, initialize NBD to 1 and LNBD to 1. And at the beginning of each line scanning, LNBD is reset to 1.

[0071] (1) Case one: if f(i,j)=1 and f(i,j-1)=0, then (i,j) is the starting point of the outer contour, NBD+1, (i2,j2)=(i,j-1).

[0072] Case two: if f(i,j)>=1 and f(i,j+1)=0, then (i,j) is the starting point of the hole contour, NBD+1, (i2,j2)=(i,j+1).

[0073] Other cases jump to (4).

[0074] (2) Determine the parent contour based on the contour type.

[0075] (3.1) Start from (i2,j2) and find a non-zero point (i1,j1) clockwise with (i,j) as the center. If there is none, assign -NBD to f(i,j) and jump to step (4).

[0076] (3.2) Set (i2, j2) = (i1, j1), (i3, j3) = (i, j).

[0077] (3.3) Find a non-zero point (i4, j4) counterclockwise from (i2, j2) with (i3, j3) as the center.

[0078] (3.4) Change the value of f(i, j) according to (i3, j3), i.e. the value of the pixel currently scanned. If f(i3, j3+1) = 0, then f(i3, j3) = -NBD. If f(i3, j3+1) ≠ 0 (may be positive or negative) and f(i3, j3) = 1, then f(i3, j3) = NBD. Otherwise, do not change the value.

[0079] (3.5) If (i4, j4) = (i, j) and (i3, j3) = (i1, j1) represent a return to the origin, jump to (4). Otherwise, (i3, j3) = (i4, j4), (i2, j2) = (i3, j3).

[0080] (4) If f(i, j) ≠ 1, then LNBD = |f(i, j)|, continue scanning from (i, j+1) until the bottom right corner of the pixel.

[0081] The entire algorithm is a process of constantly updating the current point (i3, j3), then rotating counterclockwise around the point to find the next point and constantly updating the pixel value.

[0082] In other various embodiments of the present application, with reference to the embodiments shown in Figure 1 the present application does not exhaustively list all the existing contour detection algorithms or tools that can be used in the processing 100.

[0083] Optionally, in an embodiment of the present application, with reference to the embodiments shown in Figure 1 the processing 102 can be based on fitting a geometric polygon according to the inner contour and / or the outer contour obtained by contour detection. According to the research of the inventor, fitting according to both the inner contour and the outer contour is beneficial to subsequently more comprehensively and completely extracting the frame. Of course, even if only fitting according to the inner contour or the outer contour, the frame can usually be comprehensively extracted, or at least part of the frame can be extracted, thereby improving the processing efficiency of the structural diagram.

[0084] Optionally, in an embodiment of the present application, with reference to the embodiments shown in Figure 1In the embodiment shown, the feature conditions of the polygons to be reserved in the fitted geometric polygons should satisfy: the area is greater than an area threshold (for example, a certain empirical value); the number of edges satisfies a set condition (for example, if the structural diagram is a quadrilateral structure, the number of edges should be 4); the shape is convex and a standard quadrilateral. The standard can include: the distance between any two end points of the quadrilateral is greater than a distance threshold, and the angle between two adjacent edges is not less than 30° and not more than 150°. The embodiment can accurately extract the edge frame in the structural diagram.

[0085] Figure 5 FIG. 4 is a schematic diagram of a contour detection result according to an embodiment of the present application, in which the straight line segments belong to the edge frame of the original diagram, and the jagged or zigzag line segments belong to the detected contour.

[0086] Figure 6a FIG. 5 is a schematic diagram of a flow of a method for confirming the attributes of the edge frame according to an embodiment of the present application. After obtaining the edge frame of the structural diagram containing text information, the method shown can obtain part of the attributes of the edge frame. Figure 6a As shown in FIG. 6, the method shown includes: Figure 6a

[0087] 600: Filtering the line segments of the structural diagram based on at least the edge frame of the structural diagram to obtain the to-be-clustered line segments. The edge frame of the structural diagram is obtained by the embodiment described above.

[0088] 602: Clustering the to-be-clustered line segments to obtain at least one line segment cluster, and the line segments in the line segment cluster are connected to each other. More specifically, the adjacent line segments are divided into a class according to the distance between the line segments.

[0089] 604: For any single line segment cluster in the at least one line segment cluster, determining the first attribute of the edge frame associated with the single line segment cluster according to the positional relationship between the single line segment cluster and the edge frame, the first attribute being used to distinguish whether the edge frame associated with the single line segment cluster is a source edge frame or a target edge frame relative to the single line segment cluster. The edge frame associated with the single line segment cluster refers to the edge frame connected by the single line segment cluster.

[0090] The method provided by the embodiment can determine the "directionality" attribute of the edge frame based on the structural relationship of the structural diagram. For example, for a VIE structural diagram, the holding relationship can be determined; for a flowchart, the sequence relationship can be determined.

[0091] Optionally, in an embodiment of the present application, the method shown can further include: Figure 6a ​The shown embodiment is referenced, 600 can be implemented by removing the associated line segments of the border and the line segments within the border from the line segment composition, the associated line segments of the border include: the edges of the border and the edges that meet the similar conditions with the edges of the border. Wherein, the similar condition can be that the horizontal and vertical distance with the edge of the border is less than a threshold value, so as to delete the line segments that are collinear or close to the edge of the border. In this way, the connecting line segments between the borders can be retained. In addition, in some cases, for example, in the case where there is a segmentation line and the segmentation line can be detected by the LSD algorithm, considering that the connection between the segmentation line and the border is irrelevant, at this time, by removing the longest horizontal line or the segmentation line to exclude the interference of such segmentation line.

[0092] As shown in Figure 6b The shown is the effect diagram of removing the associated line segments of the border; as Figure 6c The shown is the effect diagram of removing the line segments within the border. Wherein, by comparing the distance between each line segment and the border with the distance threshold value (for example, the distance threshold value is set as the minimum distance), the line segments within the border can be deleted.

[0093] Optionally, in an embodiment of the present application, as shown in Figure 6a The shown embodiment is referenced, in one implementation of 602, if the different classes obtained by clustering have the same line segments, the different classes with the same line segments are merged into a class.

[0094] Optionally, in an embodiment of the present application, as shown in Figure 6a The shown embodiment is referenced, in one implementation of 602, based on the networkx library of python, the networkx.connected_components library function is called. A new (empty) graph g is created. For each sub-list in lists, its elements are regarded as nodes of the graph, and an edge is added between them (as this scheme only cares about connectivity, it is not necessary to add all edges, only adjacent edges need to be added). add_edge accepts two objects, which are regarded as nodes (if there are no nodes, they are added), and an edge is added between them. The effect diagram after clustering is shown in Figure 6d .

[0095] Optionally, in an embodiment of the present application, as shown in Figure 6aThe embodiment is referenced, 604 can be implemented by the following way: from the target frame located above a single line segment cluster, the midpoint of the edge closest to the upper end point of the single line segment cluster is selected as the source node of the single line segment cluster, and the edge frame corresponding to the source node is the source edge frame of the single line segment cluster; from the target frame located below the single line segment cluster, the midpoint of the edge closest to the lower end point of the single line segment cluster is selected as the target node, and the edge frame corresponding to the target node is the target edge frame of the single line segment cluster; wherein, the number of the upper end point and the source node is one, and the lower end point and the target node are one-to-one corresponding. In addition, if the distance between the end point of a line segment cluster and the nearest edge in the target frame exceeds the threshold value, the line segment cluster can be deleted and does not need to participate in the processing of 604.

[0096] In the embodiment, the line segment cluster is obtained by clustering, and the correspondence among the node (source node, target node)-edge frame (source edge frame, target edge frame)-upper and lower end points of the line segment cluster is utilized, so that the first attribute of the edge frame can be effectively determined.

[0097] The embodiment can be implemented or optimized in combination with the following research experience: there is only one line segment above each line, and there can be several line segments below the line; the line starts from the center point of the upper and lower edges of each frame; the arrow representing the direction is downward; the upper and lower points of all vertical lines are closest to the edge center point of which frame; the upper and lower points of each cluster must have nodes. Generally, if there is a cluster with only upper nodes or only lower nodes, the cluster can be directly deleted.

[0098] Figure 7a The flowchart of the line detection method according to an embodiment of the present application is shown in FIG. 1. Figure 7a The method comprises:

[0099] 700: performing a morphological opening operation on a binary image to obtain a first image. Optionally, the binary image can be obtained by performing grayscale and binary processing on an original image. The morphological opening operation comprises a processing of erosion followed by dilation.

[0100] 702: performing a top-hat processing based on the first image and the binary image to obtain a second image. The top-hat processing refers to subtracting the result of the morphological opening operation from the binary image.

[0101] 704: performing a straight line detection based on the second image, and selecting a line segment with a length meeting a requirement from the detection result as a target line segment. For example, a Hough straight line detection method is used to select the longest line segment from the detected line segments as the target line segment.

[0102] Figure 7b The flowchart of the line detection method according to an embodiment of the present application is shown in FIG. 1.Figure 7b The method comprises:

[0103] First, based on the original drawing, gray scale and binaryzation processing is performed to obtain a binaryzation drawing. Then, based on the binaryzation drawing of the structure drawing, first corrosion and then expansion processing is performed to obtain a first drawing. Then, bitwise AND operation is performed, specifically, top hat processing is performed based on the first drawing and the binaryzation drawing to obtain a second drawing. Then, bitwise NOT operation is performed, specifically, bitwise NOT operation is performed based on the second drawing to obtain a third drawing. Then, bitwise AND operation is performed again, that is, top hat processing is performed based on the third drawing and the binaryzation drawing to obtain a fourth drawing. Then, straight line detection (for example, Hough straight line detection) is performed based on the fourth drawing, and if no line segment is detected, straight line detection is performed based on the second drawing. Finally, a line segment that meets a length (for example, the longest length) is selected from the detection results as the target line segment (for example, the line between the domestic and foreign in the VIE drawing).

[0104] Further, as shown in the judgment block in Figure 7b , in order to more accurately determine the target line segment (for example, the line between the domestic and foreign in the VIE drawing) and exclude interference caused by individual special cases, position and width constraint conditions can be added in the judgment process. For example, if the currently determined line segment with the longest horizontal length is below one fifth of the height of the drawing (max_rownum>h / 5) and its width is greater than one third of the width of the drawing (max_num>w / 3), it is considered that the line segment is the target line segment (for example, the line between the domestic and foreign in the VIE drawing), otherwise, the line segment with the second longest horizontal length is selected to continue the position and width judgment until the target line segment is finally confirmed.

[0105] The inventors have found through research that the segmentation line in the structure drawing includes a straight line, a line composed of points, a line composed of short horizontal lines, a line composed of long horizontal lines, and many other cases. Ordinary line segment detection methods cannot perform relatively complete detection. However, by using the embodiments shown in the present application Figure 7a or Figure 7b , the segmentation line in the structure drawing containing the segmentation line can be effectively identified through a series of image processing. Further, the second attribute of the frame can be determined according to the positional relationship of the frame relative to the target line segment (segmentation line), and the second attribute is used to distinguish the position of the frame, and different positions usually represent different meanings (for example, geographical area distribution, importance, etc.). For example, in the VIE structure drawing, by identifying the segmentation line, it can be determined whether the enterprise corresponding to the frame is a domestic enterprise or a foreign enterprise.

[0106] Through the various embodiments described above, the structure-related attributes of the structure drawing containing text information can be mainly obtained. Next, how to obtain text information and the association between text information and structure information will be described.

[0107] According to an embodiment of the present application, a text recognition method is provided, which comprises: taking a detected bounding box and its content as input, determining whether the proportion of English characters in the bounding box exceeds a threshold (for example, 80%) through a classification algorithm. If the threshold is exceeded, a general English detection and recognition model is called to improve recognition accuracy. Wherein, the character detection can use DB algorithm, and the character recognition can use CRNN algorithm. If the threshold is not exceeded, a Chinese and English character recognition model is called for recognition. Finally, the recognition results of each line are aggregated horizontally to obtain the text recognition result. By using this method, different models are called by determining the proportion of English to improve processing efficiency. Compared with the traditional optical character recognition technology, the sample data does not need to be segmented by characters, and the model can recognize text sequences of any length, has high speed and good performance, and can successfully recognize adjacent text instances, thereby improving the accuracy and precision of recognition. Figure 8 is an example of the original drawing (left side) and the recognition result (right side) according to an embodiment of the present application. As shown in Figure 8 , the position of the text information box of the right text recognition result has a corresponding relationship with the position of the bounding box of the left structural diagram.

[0108] Optionally, in an implementation manner of the embodiment, for any single bounding box, a name in a name library that is most similar to the third attribute of the single bounding box can also be found; if the similarity exceeds a threshold, for example, exceeds 80%, the name in the name library is used to update the third attribute of the single bounding box, otherwise, the initially determined name is still used.

[0109] Optionally, in the implementation manner, a plurality of similarity matching algorithms such as difflib, multi-process difflib and rapidfuzz can be used. The inventors have found that the rapidfuzz algorithm has higher calculation efficiency.

[0110] Optionally, in the implementation manner, a MongoDB lightweight name database can be created, the required fields are established, and the data is stored in the database. An exemplary field format is shown in the accompanying Figure 9a , wherein src_name represents the enterprise name corresponding to the source node / source bounding box, src_type represents whether the source enterprise belongs to overseas or domestic, tar_name represents the enterprise name corresponding to the target node / target bounding box, tar_type represents whether the target enterprise belongs to overseas or domestic, relation_type represents whether the source enterprise and the target enterprise are in a stock control relationship or an agreement holding relationship, and percentage represents the holding proportion. Part or all of these information can be obtained through the related embodiments or similar embodiments described above, and will not be described here.

[0111] According to an embodiment of the present application, the associated content of the border can be determined according to the association between the text information box and the border of the text recognition result of the structure diagram.

[0112] For example, for the text information box of the first type of text, the third attribute (i.e., the first type of text content) corresponding to each border is determined according to the aggregation of the text information box and the border. More specifically, the upper, lower, left and right relative coordinates top box , buttom box , left box , right box of the border can be obtained. Based on the DB detection algorithm, the upper, lower, left and right relative coordinates of all text content and the corresponding text information box are obtained, and the upper, lower, left and right relative coordinates of each text box are top text , buttom text , left text , right text .

[0113] The position of each border is traversed, and each text information box is traversed. If left box -threld horizontal <left text <right text <right box +threld horizontal and top box -threld vertical <top text <buttom text <buttom box +threld vertical (where threld horizontal is a horizontal threshold value, generally 10, and threld vertical is a vertical threshold value, generally 5), it is indicated that the text is in the border. The corresponding text content is added to the text corresponding to the border.

[0114] For the text information box of the second type of text, the fourth attribute (i.e., the second type of text content) corresponding to each border is determined according to the distance (e.g., the Euclidean distance) between the text information box and the border. For example, taking the VIE structure diagram as an example, the first type of text includes the name type text, the second type of text content includes the numerical type text, the third attribute is used to distinguish the name (enterprise name) of the border, and the fourth attribute is used to distinguish the numerical value (holding ratio) of the border. For the text information box containing a percentage, the percentage therein can be extracted by using a regular expression. Of course, in other embodiments, only the first type of text or the second type of text can be included, and the corresponding processing can be performed.

[0115] By adopting the embodiments of the present application, the text information (the third attribute and / or the fourth attribute) corresponding to each frame can be determined. In combination with the structural relationship (the first attribute and the second attribute) of each frame determined in the foregoing, more complete information can be obtained. Taking the VIE structural diagram as an example, the enterprise name corresponding to each frame, the holding relationship between different enterprises, whether the enterprise belongs to overseas or domestic, and the like can be obtained. The related information can be correspondingly stored in a data table, for example, the electronic table shown in FIG. 8, in which the fields of the source node enterprise name, the source node type (domestic / overseas), the target node enterprise name, the target node type (domestic / overseas), the enterprise relationship (agreement holding / stock holding), the holding proportion, and the like are configured. Figure 9b

[0116] The structural attribute, the text attribute of the structural diagram containing the text information, and the relationship therebetween are described in detail above. The auxiliary explanatory diagrams mainly take the VIE structural diagram as an example, but this should not constitute a limitation on the protection scope of the present application. It should be understood by those skilled in the art that the VIE structural diagram is only one of the structural diagrams and cannot represent all the structural diagrams. For example, the structural diagram not containing the domestic / overseas or related attribute can also be processed by using the related embodiments (for example, the embodiment shown in FIG. 8, the embodiment of identifying the text attribute) of the present application to obtain at least part of the structural attribute and the text attribute. For another example, the relationship between the source and the target is from top to bottom in the diagram, but in other possible scenarios, the relationship can be from left to right, from top to bottom, or even diagonal, but as long as the processing logic of the related embodiments of the present application is adopted, the required attribute information can be adaptively adjusted and processed without creative labor. This also falls within the protection scope of the present application. Figure 1

[0117] According to an embodiment of the present application, an electronic device is also provided, which includes a memory for storing computer instructions or computer programs, and a processor for calling and executing the computer instructions or computer programs stored in the memory to realize the structural diagram processing method provided by the embodiments described in the foregoing of the present application.

[0118] According to an embodiment of the present application, a computer storage medium is also provided, which stores computer instructions or computer programs, and the computer instructions or computer programs are executed to realize the structural diagram processing method provided by the embodiments described in the foregoing of the present application.

[0119] In different embodiments provided by the present application, the same parameters, nouns, logics, and the like should be understood as unified meanings, and the present application does not deliberately repeat the description in each embodiment.

[0120] ​​The exemplary embodiments of this disclosure are specifically illustrated and described herein. However, it is understood that this disclosure is not limited to the detailed construction, arrangements, or implementation methods described herein; rather, the disclosure is intended to cover any and all modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A method for processing a structural diagram containing text information, characterized in that, The method includes: Contour detection is performed based on line segments of a structure graph containing text information. A geometric polygon is fitted based on the detected contour; Based on the feature conditions of the text boxes in the structure diagram, shapes that do not meet the feature conditions are filtered out from the geometric polygons to obtain the border. The step of fitting a geometric polygon based on the detected contour includes: Geometric polygons are fitted based on the outer and inner contours; The method further includes: Based on the aforementioned structure diagram, grayscale and binarization processing is performed to obtain a binarized image; The first image is obtained by performing a morphological opening operation on the binarized image based on the structure graph. The second image is obtained by performing top-hat processing based on the first image and the binarized image; The third image is obtained by performing a bitwise NOT operation on the second image. The fourth image is obtained by performing top-hat processing based on the third image and the binarized image; Line detection is performed based on the fourth image. If no line segment is detected, line detection is performed based on the second image. Select the line segment that meets the length requirement from the detection results as the target line segment; Based on the positional relationship of the border relative to the target line segment, a second attribute of the border is determined, and the second attribute is used to distinguish the position of the border.

2. The method as described in claim 1, characterized in that, The method further includes: Based on the aforementioned structure diagram, line segment detection is performed to obtain initial line segments; The line segment composition of the structure diagram is obtained by processing the initial line segment; The line segment processing includes: filtering the initial line segment based on length constraints and angle constraints, and extending the filtered initial line segment based on length extension conditions.

3. The method as described in claim 1, characterized in that, The characteristic conditions include: The area is greater than the area threshold; The number of edges meets the set conditions; The shape is a convex quadrilateral and the distance between the endpoints and the included angle between adjacent sides satisfy the set conditions.

4. The method as described in claim 1, characterized in that, The method further includes: The line segments to be clustered are obtained by filtering the line segments based at least on the bounding box. At least one cluster of line segments is obtained by clustering the line segments to be clustered, and the line segments in the cluster are connected to each other. For any single line segment cluster in the at least one line segment cluster, a first attribute of the border associated with the single line segment cluster is determined based on the positional relationship between the single line segment cluster and the border. The first attribute is used to distinguish whether the border associated with the single line segment cluster is a source border or a target border relative to the single line segment cluster.

5. The method as described in claim 4, characterized in that, The step of filtering the line segment composition based at least on the border to obtain the line segment to be clustered includes: Remove the associated line segments of the border and the line segments within the border from the line segment composition. The associated line segments of the border include: the edges of the border and edges that satisfy similar conditions to the edges of the border. Alternatively, it could further include removing the longest horizontal line.

6. The method as described in claim 4, characterized in that, The step of determining the first attribute of the border associated with the single line segment cluster based on the positional relationship between the single line segment cluster and the border includes: From the target box located above the single line segment cluster, select the midpoint of the edge that is closest to the upper endpoint of the single line segment cluster, and use it as the source node of the single line segment cluster. The border corresponding to the source node is the source border of the single line segment cluster. From the target box located below the single line segment cluster, select the midpoint of the edge that is closest to the lower endpoint of the single line segment cluster as the target node, and the border corresponding to the target node is the target border of the single line segment cluster. There is one upper endpoint and one source node, and one lower endpoint and one target node.

7. The method as described in claim 1, characterized in that, The method further includes: The first image is obtained by performing a morphological opening operation on the binarized image based on the structure graph. The second image is obtained by performing top-hat processing based on the first image and the binarized image; Based on the second image, straight line detection is performed, and line segments that meet the length requirements are selected as target line segments from the detection results. Based on the positional relationship of the border relative to the target line segment, a second attribute of the border is determined, and the second attribute is used to distinguish the position of the border.

8. The method as described in claim 1, characterized in that, The method further includes: Based on the text recognition results of the structure diagram, the association relationship between the text information box and the border is determined, and the associated content of the border is determined.

9. The method as described in claim 8, characterized in that, The association between the text recognition result based on the structure graph and the border, and the determination of the associated content of the border, include: Aggregate the text information boxes of the first type of text in the text recognition results with the borders, and determine the third attribute corresponding to each border; and / or, Based on the distance between the text information box of the second type of text in the text recognition result and the border, the fourth attribute corresponding to each border is determined; The first type of text includes name text, the second type of text includes numerical text, the third attribute is used to distinguish the name of the border, and the fourth attribute is used to distinguish the numerical value of the border.

10. The method according to any one of claims 1, 4, 7 or 9, characterized in that, The method further includes: The determined attributes of the border are stored in a data table, which is configured with fields corresponding to the attributes of the border.

11. An electronic device, characterized in that, The electronic device includes: Memory, used to store computer instructions or computer programs; A processor for calling and executing the computer instructions or computer programs stored in the memory to implement the method as described in any one of claims 1-10.

12. A computer storage medium, characterized in that, It stores computer instructions or computer programs that, when executed, implement the method as described in any one of claims 1-10.

Images