Pairing system applied to graphical representation and text description on specification

Abstract

The invention provides a pairing system, which is used for pairing a graphic representation and a character description on a specification, and comprises an image-text identification device for receiving the specification and finding a graphic representation block and a character block on the specification through image and character identification; wherein the graphic block has a corresponding coverage area; and the preference value calculation device is used for distributing a preference value to each character block according to the position relationship between the character blocks and the graphical representation blocks and the content characteristics of the character blocks, and the preference values are used for matching the character blocks and the graphical representation blocks.

Description

Technical Field

[0001] This invention relates to a pairing system, and more particularly to a pairing system applied to the illustrations and textual descriptions in a specification sheet. Background Technology

[0002] Typically, specifications contain various diagrams (icons) and textual descriptions. The textual descriptions include, for example, the diameter, length, and quantity of screws, as well as the dimensions and inspection locations of other parts or components. In the past, quality control personnel had to manually mark items requiring inspection to facilitate acceptance by factory workers. However, due to the diversity of specifications and the complexity of the textual descriptions, this work was very time-consuming and prone to errors. Therefore, current production line quality control personnel spend a significant amount of time processing the correspondence between diagrams and dimensions in the specifications, yet there is currently no more efficient method for marking and matching specifications.

[0003] Traditionally, quality control personnel must rely on past experience to mark illustrations (such as organizational icons and images) and descriptive text (such as dimensions) on specifications. However, currently, there is no suitable matching method for images and text that can be applied to complex specifications. While matching product images with text descriptions is common, applying this to specifications requires extracting the necessary textual content, such as quantity and dimensions, from complex descriptions. Dimensions can be marked in various ways, and their placement is not fixed. Therefore, quality control personnel must spend a significant amount of time matching and marking specific inspection items with dimensions—a tedious and time-consuming process.

[0004] In addition, different icons in the specification sheet represent different inspection items. The size specifications are usually located near the icons. However, since they are manually marked and provided by the customer or the company's R&D unit, there are no established rules for placing the corresponding text blocks in a specific position above, below, left, or right of the icon, nor are there any regulations on how far away from the icon they should be. This leads to some difficulties when trying to automatically mark the icons and descriptive text in the specification sheet. Summary of the Invention

[0005] In view of this, in order to solve the aforementioned problems, there is a need to provide a matching system for illustrations and text descriptions that can be applied to specifications. By automating the matching process, annotation efficiency can be improved, time costs can be reduced, and human annotation errors can be minimized.

[0006] An embodiment of the present invention provides a pairing system for pairing illustrations and text descriptions on a specification sheet. The pairing system includes: an image and text recognition device and a preference value calculation device. The image and text recognition device receives a specification sheet and identifies illustration blocks and text blocks on the specification sheet through image and text recognition; here, the illustration blocks have corresponding coverage areas. The preference value calculation device assigns a preference value to each of the text blocks and illustration blocks based on the positional relationship between the text blocks and illustration blocks, and the characteristics of the text blocks, for further pairing of the text blocks and illustration blocks.

[0007] In a pairing system according to an embodiment of the present invention, the preference value calculation device further performs the following operations: (1) calculating the overlapping coverage area of ​​the text block and the first illustrated block in the illustrated block, and obtaining an area fraction for each of the overlapping text blocks; (2) calculating the distance to the first illustrated block for each of the overlapping text blocks, and obtaining a distance fraction for each of the overlapping text blocks; (3) determining a negative score for each of the overlapping text blocks based on the location, description format, and frame range of the first illustrated block; and (4) calculating a preference value for each of the overlapping text blocks based on the area fraction, distance fraction, and negative score.

[0008] In a pairing system according to an embodiment of the present invention, the preference value calculation device further adds text blocks with overlapping elements whose preference scores meet the candidate conditions to the candidate list of the first illustration; and the pairing system further includes a filtering device. Wherein, when a first text block in the candidate list of the first illustration is associated with a second illustration block of the illustration block, the filtering device determines whether to retain or delete the first text block from the candidate list of the first illustration based on the distance score, area score, and coverage integral of the first text block corresponding to the first illustration block and the second illustration block, respectively.

[0009] In a pairing system according to an embodiment of the present invention, the filtering device compares the distance scores of the first text block corresponding to the first graphic block and the second graphic block, respectively; if the distance score of the first text block is lower, the first text block is deleted from the candidate list of the first graphic. If the distance scores of the first text block corresponding to the first graphic block and the second graphic block are the same, the area scores corresponding to the first graphic block and the second graphic block are further compared; and if the area score of the first text block is lower, the first text block is deleted from the candidate list of the first graphic. If the area scores of the first text block corresponding to the first graphic block and the second graphic block are the same, the coverage integrals corresponding to the first graphic block and the second graphic block are further compared; and if the coverage integral of the first text block is lower, it is retained in the candidate list of the first graphic, and if the coverage integral of the first text block is higher, it is deleted from the candidate list of the first graphic.

[0010] In a matching system according to an embodiment of the present invention, a recommendation device is further included. When the candidate list of the first illustrated block contains only one text block, the recommendation device retains the text block; when the candidate list of the first illustrated block contains multiple text blocks, the recommendation device selects text blocks from the candidate list of the first illustrated block according to a highest score strategy or a relative position strategy. Specifically, when using the highest score strategy, the recommendation device selects the text block with the highest preference value from the candidate list and matches it to the first illustrated block. When using the relative position strategy, the recommendation device prioritizes selecting the text blocks with the highest preference value from the candidate list that are horizontally or vertically positioned relative to the first illustrated block and matches them to the first illustrated block.

[0011] In a pairing system according to an embodiment of the present invention, the configuration of the coverage area corresponding to the illustrated block is such that, with the size of the illustrated block as the basic unit, a first predetermined number of units are horizontally extended to the left and right respectively, centered on the illustrated block, and a second predetermined number of units are horizontally extended to the upper and lower sides of the illustrated block respectively. Furthermore, the first predetermined number is, for example, 3.5, and the second predetermined number is, for example, 5. Attached Figure Description

[0012] Figure 1 Examples of "illustrations" commonly found in specifications and their corresponding "textual descriptions".

[0013] Figure 2 This diagram illustrates a configuration block diagram of a pairing system applied to the illustrated and textual descriptions in a specification sheet, according to an embodiment of the present invention.

[0014] Figure 3 This diagram shows the block P and its corresponding coverage area COV.

[0015] Figure 4 This diagram shows block P, its corresponding coverage area COV, and text block T.

[0016] Figure 5 A schematic diagram of the processing flow of the preference value calculation device is displayed.

[0017] Figure 6A , 6B This shows examples of a sub-region of the COV coverage area overlapping with a text block, as well as examples of non-overlapping areas.

[0018] Figure 7 This illustrates the operation flow of the filtration device in this embodiment.

[0019] Figure 8 The diagram shows the corresponding text blocks of the filter device 203 before and after processing.

[0020] Figure 9 This diagram shows the operation flow of the recommended device 204 in the pairing system of this embodiment.

[0021] [Symbol Explanation]

[0022] 11a~11e: Illustration

[0023] 12a~12e: Textual description

[0024] 200: Pairing System

[0025] 201: Image and text recognition device

[0026] 202: Preference value calculation device

[0027] 203: Filtration device

[0028] 204: Recommended device

[0029] P: Illustration (Illustrated Block)

[0030] T: Text Block

[0031] COV: Coverage

[0032] CU: Subregion Range

[0033] S51~S57: Steps

[0034] S71~S79: Steps

[0035] P1~P5: Illustrations

[0036] S91~S95: Steps Detailed Implementation

[0037] The following description is a preferred embodiment of the invention and is intended to describe the basic spirit of the invention, but is not intended to limit the invention. The actual invention must be understood by referring to the claims that follow.

[0038] Figure 1 The symbols commonly found in the specifications include “illustrations (icons)” such as “box Cpk” 11a, “(elliptical) circle Cpk” 11b, “square bracket Cpk” 11c, “(elliptical) circle Cpk” 11d, “hexagon” 11e, and “text descriptions” 12a to 12e corresponding to these symbols. Figure 1 The examples shown here are for illustrative purposes only and are not intended to limit the content of the specification.

[0039] Figure 2 This is a block diagram illustrating the configuration of a pairing system 200 applied to the illustrations and text descriptions in a specification sheet, according to an embodiment of the present invention.

[0040] like Figure 2 As shown, the pairing system 200 of the present invention includes a graphic recognition device 201, a preference value calculation device 202, a filtering device 203, and a recommendation device 204.

[0041] The pairing system 200 of the present invention, for example, is a computer system or electronic device with a processor or controller, which loads and executes a program to achieve the desired function. The aforementioned identification device 201, preference value calculation device 202, filtering device 203, and recommendation device 204 are implemented, for example, by the processor or controller executing the corresponding program. Furthermore, the aforementioned identification device 201, preference value calculation device 202, filtering device 203, and recommendation device 204 can also be implemented by hardware such as application-specific integrated circuits (ASICs) and field-programmable gate arrays (FPGAs).

[0042] In addition, the pairing system 200 may also include an input device and a setting device. Figure 2 (Not shown). Input devices include, for example, keyboards, mice, and other interfaces of computer systems or electronic devices that allow user interaction. Using the pairing system 200 of this invention, users (quality control personnel) can scan or upload specifications to the pairing system 200 or a database.

[0043] After receiving the uploaded specification sheet or retrieving it from the database, the matching system 200 hands it over to the image and text recognition device 201 for processing. The image and text recognition device 201 identifies the graphic and text blocks on the specification sheet through image and text recognition; wherein the graphic blocks have corresponding coverage areas. The so-called "graphic blocks" refer to areas covered in the specification sheet such as... Figure 1 The scope of one of the "illustrations" 11a...11d; the so-called "text block" refers to the area covered in the specification as... Figure 1 The scope of one of the “textual descriptions” 12a…12d. More specifically, the illustrated block is a framed area used to indicate the area where the “illustration” is located after image recognition. In the following description, the illustration and the illustrated block are interchangeable when used to indicate the framed area.

[0044] Image and text recognition device 201 performs image and text recognition on the specification document. Image and text recognition device 201 can, for example, use an image segmentation model to identify and outline illustrated blocks. Furthermore, image and text recognition device 201 can, for example, use optical character recognition (OCR) to convert graphics other than illustrations in the specification document into text format, including numbers, symbols, and general descriptive text, and label them as text blocks. The image segmentation model can be, for example, a machine learning model or a deep learning model trained on commonly used illustrations in the specification document, but is not limited to these.

[0045] Figure 3 This diagram shows the block P and its corresponding coverage area COV. Figure 4 This diagram shows block P, its corresponding coverage area COV, and text block T.

[0046] Figure 3 In this context, P represents the graphic area identified by the graphic recognition device 201, hereinafter referred to as graphic area P. The coverage area corresponding to graphic area P is configured such that, with the size of graphic area P as the basic unit, it extends horizontally to the left and right sides of graphic area P by a first predetermined number of units (sub-area range) CU, centered on graphic area P, and simultaneously extends horizontally to the top and bottom of graphic area P by a second predetermined number of units, as shown below. Figure 3 The area indicated by the symbol COV. Here, the height of the illustrated block P can be 1 to 1.5 times the actual identified graphic size. The coverage area of ​​the COV and the number of sub-areas CU can both be set by the optional setting device.

[0047] Reference Figure 4Since the area enclosed by text block T is usually rectangular, if a circular coverage area (not shown) is set centered on the illustrated block P, the vertical range of the coverage area will be too large, easily leading to misidentification of areas other than text block T. This is because different text blocks may be too close together, so the coverage area COV cannot be too large. Furthermore, in the specification, much of the text description is arranged vertically (e.g., text block T); therefore, the coverage area COV is set only to the top and bottom of the illustrated block P with the height of only one illustrated block P, equivalent to the range of multiple units (sub-area ranges) in one column. In addition, since text block T is usually rectangular or composed of multiple rectangles, when text block T is to the left or right of the illustrated block P, the length of the coverage area COV needs to be increased to ensure sufficient coverage. In this embodiment, it is preferable that the first predetermined number of extensions to the left and right sides of the illustrated block P is 3.5 grids (sub-area range); and it is preferable that the second predetermined number of extensions to the top and bottom sides of the illustrated block P is 5 grids (sub-area range) wide, so as to cover the diagonal text block.

[0048] Figure 5 The processing flow diagram of the preference value calculation device 202 is shown. The preference value calculation device 202 calculates the preference value of each text block that is related to (overlaps with) the illustrated block of each illustration in the specification. The following describes the process of calculating the preference value of each text block that overlaps with the illustrated block P.

[0049] First, the preference value calculation device 202 excludes text blocks from all text blocks that do not overlap with the coverage area COV of the illustrated block P (step S51). For each text block that overlaps with the coverage area COV of the illustrated block P (step S52), in step S53, the overlapping area between the text block and the coverage area COV of the illustrated block P is calculated to obtain an area score. Next, in step S54, the distance between the overlapping text block and the illustrated block P is calculated to obtain a distance score for the text block. In step S55, based on the location, description format, and frame range of the overlapping text block, a negative score is determined for each overlapping text block. In step S56, based on the area score, distance score, and negative score, the preference value of the overlapping text block is calculated, and the preference value of the text block is added to the candidate list of the illustrated block P (step S57). Subsequently, if the preference value calculation device 202 determines that there are still overlapping text blocks in the illustrated block P that have not been assigned preference values, then it repeats steps S53 to S57.

[0050] The following describes how the preference value calculation device 202 calculates the area fraction (or the fraction of the coverage area).

[0051] intersection pt =|Max(0,Min(x) t4 ux p4 )-Max(x t1 ux p1 ))×Max(0,Min(y t4 ,uy p4 )-Max(y t1 ,uy p1 ))|

[0052] Width = (Minimum x-coordinate of bottom right corner) - (Maximum x-coordinate of top left corner)

[0053] Height = (Minimum y-coordinate of the bottom right corner) - (Maximum y-coordinate of the top left corner)

[0054] Overlapping area = Width × Height.

[0055] Area_scope pt Representing area fractions:

[0056]

[0057] Where (x) t1 y t1 ), (x t2 y t2 ), (x t3 y t3 ), (x t4 y t4 (ux represents the four coordinates of the text block;) p1 ,uy p1 ), (ux p2 ,uy p2 ), (ux p3 ,uy p3 ), (ux p4 ,uy p4 ) represents the coordinates of four points within the coverage area of ​​the sub-region.

[0058] The diagram shows that the coverage area COV of block P consists of multiple sub-areas, with W2 representing the weight. Here, the intersection... pt This represents the overlap area between a sub-region within the COV coverage area and the text block. Area_score pt Represents the area fraction, which is the sum of the overlapping areas of all small boxes in the COV and the text block, divided by the area of ​​the text block.

[0059] In addition, note that here, the +X direction of the coordinate axis is from left to right, and the +Y direction of the coordinate axis is from top to bottom.

[0060] Figure 6A , 6B This shows examples of a sub-region of the COV coverage area overlapping with a text block, as well as examples of non-overlapping areas.

[0061] Reference Figure 6A The coordinates of the text block are (x1, y1) = (1, 2), (x2, y2) = (1, 7), (x3, y3) = (6, 2), and (x4, y4) = (6, 7). The coordinates of the small box area are (x1, y1) = (4, 4), (x2, y2) = (4, 6), (x3, y3) = (8, 4), and (x4, y4) = (8, 6). The calculation is as follows:

[0062] Width=Min(6,8)-Max(1,4)=6-4=2

[0063] Height=Min(7,6)-Max(2,4)=6-4=2

[0064] Area_score=(2×2) / ((7-2)×(6-1))=4 / 25.

[0065] Reference Figure 6B The coordinates of the text block are (x1, y1) = (1, 1), (x2, y2) = (1, 3), (x3, y3) = (3, 1), and (x4, y4) = (3, 3). The coordinates of the small box area are (x1, y1) = (4, 4), (x2, y2) = (4, 6), (x3, y3) = (6, 4), and (x4, y4) = (6, 6). The calculation is as follows:

[0066] Width = Min(3, 6) - Max(1, 4) = 3 - 4 = -1, the maximum of which is 0.

[0067] Height = Min(3, 6) - Max(1, 4) = 3 - 4 = -1, the maximum of which is 0 is 0.

[0068] Area_score=(0×0) / ((3-1)×(3-1))=0.

[0069] After calculating the area fractions, for example, we obtain the area fractions for the six text blocks corresponding to the blocks shown in the diagram.

[0070]

[0071] The following describes how the preference value calculation device 202 calculates the distance score.

[0072]

[0073] N represents the order of the diagram, such as 1, 2, 3...

[0074] t indicates the order of text blocks, such as 1, 2, 3...

[0075] Dpt represents the minimum distance from the left and right boundaries of the illustration p to the boundary of the text block t.

[0076] Where (x) t1 y t1 ), (x t2 y t2 ), (x t3 y t3 ), (x t4 y t4 () represents the four coordinates of the text block t; while (cx) p1 cy p1 ), (cx p2 cy p2 ), (cx p3 cy p3 ), (cx p4 cy p4 ) represents the coordinates of the four points of block P in the diagram.

[0077] After obtaining the distance from the illustrated block P to the text block t, the following transformation is performed.

[0078]

[0079] The sum of the distances between the DT icon and the text block.

[0080] DL pt The greater the distance, the smaller the value will be. After conversion, take the positive value and calculate the ratio.

[0081] Where t represents the text block, such as the 1st, 2nd, 3rd... text blocks, and n represents the illustration number.

[0082] Next, calculate the sum and average of the distances of all k text blocks in the illustrated area, as follows.

[0083]

[0084] The average distance between the AD icon and the text block.

[0085] Where t represents a text block, such as the 1st, 2nd, 3rd... text blocks, and p represents the illustration number.

[0086] Next, the distance error score is calculated to obtain the distance score, Distance_score. pt The method is as follows.

[0087]

[0088] Distance_score pt =W1×DL pt +(1-W1)*DB pt

[0089] DB pt The error value between the distance of text block t and the average distance.

[0090] DL pt The distance between the text block t and the illustration p is converted to a positive value, and the ratio is calculated.

[0091] W1 represents the distance weight, and this weight will have different values ​​depending on the different diagrams.

[0092] Where t represents a text block, such as the 1st, 2nd, 3rd... text blocks, and p represents the illustration number.

[0093] Assuming that block P in the diagram covers 5 text blocks, and the total distance between the 5 text blocks is 25, the average distance is 6, and the weight W1 is set to 0.7, the distance score is calculated as follows. Furthermore, in this embodiment, generally, the greater the distance, the lower the distance score.

[0094]

[0095] The following explains how the preference value calculation device 202 determines the negative score. pt The method is as follows.

[0096] Negative_score pt =W3×∑Bad_Score pt

[0097] ∑Bad_Score pt This represents the total number of points deducted.

[0098] W3 represents the weight of the negative score, and this part has different values ​​depending on the chart.

[0099] Since the text block may have problems, the preference value calculation device 202 will further deduct points accordingly. Possible problems include, but are not limited to: (1) the text block is surrounded by other text, indicating that it may be an illustration of text description or table content; (2) the text content within the text block does not conform to common size specifications, such as two decimal points in the description of numbers, or two dashed lines ('-'), which may be caused by OCR recognition errors; (3) the frame of the illustration is too large, which may result in misframing other graphics, which is a problem of graphic recognition results and can be judged according to the length and width of the illustration block. When there are the aforementioned (1) to (3) or other deduction items, for example, a deduction of Bad_score = 100 is given in all cases, and Bad_score = 0 if there are no such cases, but it is not limited to this. Negative scores can be optimization options and can also be omitted when calculating preference values.

[0100] For example, text blocks 1-4 have the aforementioned situations (1)-(3), while text block 5 does not, so the negative scores are as follows. That is, if the aforementioned situations (1)-(3) occur, the matching system tends to avoid using them and try to exclude them as much as possible.

[0101] Text blocks negative scores 1 <![CDATA[Negative_score 11 =0.5×0=0.00]]> 2 <![CDATA[Negative_score 12 =0.5×0=0.00]]> 3 <![CDATA[Negative_score 13 =0.5×0=0.00 <!-- 7 -->]]> 4 <![CDATA[Negative_score 14 =0.5×0=0.00]]> 5 <![CDATA[Negative_score 15 =0.5×100=50.00]]>

[0102] Preference value calculation device 202, based on area fraction Area_score pt Distance score pt Negative score pt The method for calculating preference values ​​is as follows.

[0103]

[0104] Where t represents the text block order, such as number 1, 2, 3..., and p represents the illustration number.

[0105] Text_scorept represents the score of text block t for the graphic p.

[0106] W4 represents the occupancy of text descriptions within text blocks, and different weights will be assigned to them.

[0107] Among them, the text blocks with higher preference scores are the most likely to be paired with the illustrated blocks.

[0108] For example, if there are 5 text blocks within the coverage area COV of block P shown in the figure, the preference value is calculated as follows.

[0109]

[0110] After calculating the preference value for each text block, it can be further determined whether each preference value meets the minimum criteria, and those that meet the criteria are added to the candidate list of the illustrated block P. The minimum criteria, for example, are values ​​determined after considering distance, coverage, and the aforementioned logarithmic value (Log value).

[0111] When a candidate list for an illustration contains multiple text blocks, the matching system 200 needs to use a suitable strategy to determine the correct matching relationship, while filtering out text blocks that are irrelevant to the illustration. This processing is mainly performed by the filtering device 203 in this embodiment.

[0112] Figure 7 The operation flow of the filter device 203 in this embodiment is shown. The operation of the filter device 203 will be described below.

[0113] The matching system 200 performs horizontal and vertical calculations on each text block, further considering the position of the text block relative to the illustration, thereby adopting different strategies for more accurate matching. The matching system 200 can use the filtering device 203 to determine the horizontal and vertical orientation of each text block, but is not limited to this; other devices can also perform this function.

[0114] The calculation method for determining whether a text block is within a horizontal or vertical range is as follows. The determination of horizontal or vertical orientation primarily utilizes the center points of the left and right edges of the text block and the graphic block to calculate the relative angle. The formula is as follows:

[0115]

[0116] Where (x) t1 y t1 ), (x t2 y t2 ), (x t3 y t3 ), (x t4 y t4 () represents the four coordinates of the text block t; while (cx) p1 cy p1 ), (cx p2 cy p2 ), (cx p3 cy p3 ), (cx p4 cy p4 The numbers () represent the coordinates of the four points in block P shown in the diagram. Additionally, if the angle Angle... pt If the angle is less than 45 degrees, it is considered horizontal; otherwise, it is considered vertical.

[0117] For example, the coordinates of the text block are (x1, y1) = (1, 1), (x2, y2) = (1, 3), (x3, y3) = (3, 1), and (x4, y4) = (3, 3). The center point on the right side of the text block is (3, 2), and the center point on the left side is (1, 2). The coordinates of the illustrated block P are (xp1, yp1) = (6, 2), (xp2, yp2) = (6, 4), (xp3, yp3) = (9, 2), and (xp4, yp4) = (9, 4). The center point on the left side of the illustrated block P is (6, 3), and the center point on the right side is (9, 3).

[0118] The calculation result is:

[0119] Angle1=atan((3-2) / (6-3))×100=atan(1 / 3)×100=32,

[0120] This is the right side of the text block relative to the left side of the illustration P. Angle2 = atan((3-2) / (9-1)) × 100 = atan(1 / 3) × 100 = 12.4, this is the left side of the text block relative to the right side of the illustration P. Taking the larger value results in Angle2. pt The value is 32, therefore the text block is considered to be in a vertical position relative to the diagram P.

[0121] Here, it is assumed that the pairing system 200 adopts a level-first pairing strategy. First, the filtering device 203 extracts text blocks that match the horizontal range from the candidate list of illustration P (step S71). Then, the filtering device 203 determines whether the text blocks within the horizontal range are associated with other illustrations (step S72). If there are other illustrations (e.g., illustration Q) associated with the text block (step S72: yes), the filtering device 203 compares the distance scores of illustrations P and Q with the text blocks (step S73). If there are no other illustrations associated with the text block (step S72: no), the text block is retained in the candidate list of illustration P (step S77).

[0122] If the distance score between the text block and the illustration Q is low, it is removed from the candidate list of illustration Q (step S74). If the distance score between the text block and the illustrations P and Q is the same, the area scores of the text block with the illustrations P and Q are compared (step S75).

[0123] If the area fraction of the text block is lower than that of illustration Q, it is removed from the candidate list of illustration Q (step S74). If the area fraction of the text block is the same as that of illustrations P and Q respectively, the coverage integrals of the text block with illustrations P and Q are compared (step S76). Based on the comparison results, the text block with the lower coverage integral is retained in the candidate list of illustration P (step S77), and the text block with the higher coverage integral is removed from the candidate list of illustration P (step S74).

[0124] Furthermore, the filtering device 203 also identifies text blocks that match the vertical range from the candidate list of illustration P (step S78). Then, the filtering device 203 determines whether the text block within the vertical range is associated with other illustrations (step S79). If other illustrations (e.g., illustration R) are associated with the text block (step S79: Yes), the filtering device 203 compares the distance scores of illustrations P and R with the text block (step S73). If no other illustrations are associated with the text block (step S79: No), the text block is retained in the candidate list of illustration P (step S77). If the distance score of the text block with illustration R is lower, it is removed from the candidate list of illustration R (step S74). If the distance scores of the text block with illustrations P and R are the same, the area scores of the text block with illustrations P and R are compared (step S75). If the area score of the text block with illustration R is lower, it is removed from the candidate list of illustration R (step S74). If the text block has the same area fraction as both Figure P and Figure R, then compare the coverage integrals of the text block with those of Figure P and Figure R (step S76). Based on the comparison results, retain the text block with the lower coverage integral in the candidate list of Figure P (step S77), and remove the text block with the higher coverage integral from the candidate list of Figure P (step S74).

[0125] The pairing system 200 of this embodiment executes the above process through the filtering device 203, and can intelligently select and filter text blocks according to the needs of different illustrations.

[0126] The aforementioned coverage integral is calculated as follows:

[0127]

[0128] Among them, Area_scope pt This represents the area fraction of the text block t centered at P in the diagram; Area_TotalScope pt The total area covered by the illustration P is represented by t; and t represents the sequence number of the text block, such as 1, 2, 3..., and p is the illustration number.

[0129] The following is for reference Figure 8This section describes the correspondence between the illustrations and text blocks of the filter device 203 before and after its operation.

[0130] Reference Figure 8 The Before section above the filter device 203 displays a schematic diagram showing the preference scores, distances, and relative positions of the text blocks t1 to t5 and the illustrations (illustration blocks) P1 to P4 before the filter device 203 processes the filter. The After section below the filter device 203 displays a schematic diagram showing the preference scores, distances, and relative positions of the text blocks t1 to t5 and the illustrations P1 to P4 after the filter device 203 processes the filter.

[0131] In the Before section, the candidate list of P1 includes text blocks t1 to t4, with preference values ​​of 75.57, 88.47, 74.00, and 100.432 respectively; and distances from P1 of 4 (horizontal), 4 (vertical), 8 (horizontal), and 2 (horizontal) respectively.

[0132] For illustration P1, firstly, the filtering device 203 identifies text blocks t1 and t3 within the horizontal range (step S71). In step S72, it is determined that text block t1 is not associated with other illustrations (P2-P4), so it is retained in the candidate list (step S77). In step S72, it is determined that text block t3 is associated with other illustration P4 (step S72: Yes), therefore the filtering device 203 further compares the distances of text blocks t1 and t3 with illustration P1 (step S73). Since the distance of text block t1 is less than that of text block t3 (i.e., the distance score of text block t1 is greater than that of text block t3), the text block t3 with the lower distance score is removed from the candidate list of illustration P1 (step S74).

[0133] For illustration P1, the filtering device 203 searches for text blocks t2, t4, and t5 within the vertical range (step S78). In step S79, it is determined that text block t2 is not associated with other illustrations (P2-P4), so the text block is retained in the candidate list (step S77). In step S79, although it is determined that text block t4 is associated with other illustration P2, because the pairing system 200 adopts a horizontal priority strategy, and illustration P2 is only associated with illustration P in the horizontal range, the filtering device 203 will retain text block t4 for illustration P2, thus deleting text block t4 from the candidate list of illustration P1. Similarly, in step S79, although it is determined that the text block t5 is associated with other illustrations P3, because the pairing system 200 adopts a horizontal priority strategy and the text block t5 is only associated with illustration P3 in the horizontal range, the filtering device 203 will reserve the text block t5 for illustration P3, and thus delete the text block t5 from the candidate list of illustration P1.

[0134] Figure 8 The "After" section below the filter device 203 shows a schematic diagram of the relationship between the illustrated blocks P1 to P4 and the text blocks after processing by the filter device 203. The candidate list of illustrated P1 contains text blocks t1 and t2, and the candidate lists of illustrated P2 to P4 contain text blocks t4, t5, and t2, respectively.

[0135] Figure 9 This diagram shows the operation flow of the recommended device 204 in the pairing system 200 of this embodiment. The following will refer to... Figure 9 Explain the operation of the recommended device 204.

[0136] After the filtering device 203 processes the data, the recommendation device 204 determines in step S91 whether the illustrated block P can correspond to multiple text blocks. When the candidate list of the illustrated block P contains only one text block (step S91: No), the recommendation device 204 retains this text block (step S92). When the illustrated block P can correspond to multiple text blocks (step S91: Yes), the recommendation device 204 can select a text block from the candidate list of the illustrated block P based on the highest score strategy (step S93) or the relative position strategy (horizontal priority or vertical priority) (step S94).

[0137] When using the highest score strategy (step S93), the recommendation device 204 selects the text block with the highest preference value from the candidate list and pairs it with the illustrated block P (step S95). When using the relative position strategy (step S94), if a horizontal priority strategy is adopted, the recommendation device 204 first deletes the vertical text blocks, then selects the text block with the highest preference value and pairs it with the illustrated block P (step S95); if a vertical priority strategy is adopted, the recommendation device 204 may first delete the horizontal text blocks, then select the text block with the highest preference value and pair it with the illustrated block P (step S95). After the recommendation device performs pairing based on the highest score strategy or the relative position strategy, it further deletes the unpaired text blocks from the candidate list of the illustrated block P.

[0138] Reference Figure 8 As shown in the After section, the highest score is not necessarily retained. For example, t4 is filtered out because illustration P2 only corresponds to text block t4, so it will be assigned to illustration P2 first and removed from the candidate list of illustration P1. When illustration P corresponds to text blocks t1 and t2, if illustration P1 can only correspond to one illustration, if the selection is based on the highest score strategy, then t2 will be selected (because t2: 88.47 > t1: 75.57). If the selection is based on the horizontal priority strategy, then t1 will be selected by illustration P1, and t3 will be assigned to illustration P4 instead of illustration P2. If the selection is based on the vertical priority strategy, then t2 will be selected by illustration P1; if illustration P1 can correspond to multiple text blocks, then there is no need to select from text blocks t1 and t2.

[0139] The matching system according to the present invention can be applied to the matching system of illustrations and text descriptions in specifications. By automating the matching process, it can improve annotation efficiency, reduce time costs, and reduce human annotation errors.

Claims

1. A matching system for matching illustrations and textual descriptions in a specification, comprising: an image-text recognition device configured to receive a specification and to identify, by image and text recognition, illustration blocks and text blocks in the specification, wherein the illustration blocks have corresponding coverage areas; and a preference value calculation device configured to assign a preference value to each of the text blocks based on a positional relationship between the text blocks and the illustration blocks, and a content characteristic of the text blocks, for matching the text blocks with the illustration blocks.

2. The matching system of claim 1, wherein the preference value calculation device is further configured to: calculate an overlapping area of the text blocks and a coverage area of a first illustration block of the illustration blocks to obtain an area fraction of each of the text blocks having an overlap with the first illustration block; calculate a distance of each of the text blocks having an overlap with the first illustration block to obtain a distance fraction of each of the text blocks having an overlap with the first illustration block; determine a negative fraction of each of the text blocks having an overlap based on a location of the text blocks having an overlap, a description format, and a frame range of the first illustration block; and calculate the preference value of each of the text blocks having an overlap based on the area fraction, the distance fraction, and the negative fraction.

3. The matching system of claim 2, wherein the preference value calculation device is further configured to add a text block of the text blocks having an overlap to a candidate list of the first illustration block if the text block satisfies a candidate condition.

4. The pairing system of claim 3, further comprising: a filtering device; wherein, when a first text block in the candidate list of the first illustration has an association with a second illustration block of the illustration blocks, the filtering device determines whether to keep or delete the first text block from the candidate list of the first illustration based on a distance fraction, an area fraction, and a coverage fraction of the first text block corresponding to the first illustration block and the second illustration block, respectively.

5. The matching system of claim 4, wherein the coverage fraction is represented as: wherein wherein p represents a number of the illustration blocks, and k represents a total number of all the text blocks having an overlap with the illustration block p. Area_score pt corresponds to the sum of the overlapping areas of the k text blocks that overlap with the above-described illustrated block p.

6. The matching system of claim 4 or 5, wherein the filtering device is configured to compare the distance fraction of the first text block corresponding to the first illustration block and the second illustration block, respectively; if the distance fraction of the first text block is lower, the first text block is deleted from the candidate list of the first illustration; if the distance fraction of the first text block corresponding to the first illustration block and the second illustration block is the same, the filtering device is further configured to compare the area fraction of the first text block corresponding to the first illustration block and the second illustration block, and if the area fraction of the first text block is lower, the first text block is deleted from the candidate list of the first illustration. If the first text block corresponds to the same area fraction of the first and second graphic blocks, the recommendation device further compares the coverage integral of the first and second graphic blocks, and if the coverage integral of the first text block is lower, the first text block is kept in the candidate list of the first graphic block, and if the coverage integral of the first text block is higher, the first text block is deleted from the candidate list of the first graphic block.

7. The pairing system of claim 6, further comprising: The recommendation device; If the candidate list of the first graphic block has only one text block, the recommendation device keeps the text block; If the candidate list of the first graphic block has more than one text block, the recommendation device selects the text block in the candidate list of the first graphic block according to a highest score strategy or a relative position strategy; If the highest score strategy is used, the recommendation device selects the text block with the highest preference value in the candidate list and pairs the text block with the first graphic block; If the relative position strategy is used, the recommendation device selects the text block with the highest preference value from the text blocks horizontally or vertically arranged opposite to the first graphic block in the candidate list and pairs the text block with the first graphic block.

8. The pairing system of claim 7, wherein After pairing the text blocks in the candidate list according to the highest score strategy or the relative position strategy, the recommendation device deletes the text blocks in the candidate list which are not paired.

9. The pairing system of claim 1, wherein The configuration of the coverage range corresponding to the graphic block is that the size of the graphic block is taken as a basic unit, and the graphic block is taken as a center, a first predetermined number of units are horizontally extended to the left and right sides of the graphic block, and a second predetermined number of units are horizontally extended to the upper and lower sides of the graphic block.

10. The pairing system of claim 9, wherein The first predetermined number is 3.5, and the second predetermined number is 5.

Images