Table question answer pair automatic extraction method, device, equipment and medium

Abstract

The application discloses an automatic extraction method of a table question-answer pair, which comprises the following steps: obtaining a table to be analyzed; extracting appearance features of the table to be analyzed; judging a table type of the table to be analyzed according to the appearance features; if the judgment fails, extracting content features of the table to be analyzed; determining a first table type of the table to be analyzed according to the content features; extracting a table question and a table answer of the table to be analyzed according to the first table type; and composing the table question and the table answer into a question-answer pair. The table type can be easily judged according to the appearance features of the table to be analyzed, and the table type can be judged according to the content features when the table type cannot be judged according to the appearance features. The judgment of the content features involves the statistics of texts and numbers. The table type can be quickly judged by analyzing the appearance features and then analyzing the content features. The question-answer pair can be automatically extracted according to the table type, and the method has higher efficiency compared with manual extraction of the question-answer pair.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of text detection, for example to a table question and answer pair automatic extraction method, device, equipment and medium. BACKGROUND

[0002] At present, a large part of customer service robots are implemented based on question and answer pairs. The operation personnel configure question and answer pairs for the robot, and then the robot matches the preset question with the highest similarity according to the question input by the user, and gives the corresponding answer according to the matched preset question.

[0003] The performance of the customer service robot is greatly related to the quality and quantity of the question and answer pairs, so the operation personnel need to configure more question and answer pairs. At present, the operation personnel need to manually extract question and answer pairs from existing materials such as documents and web pages for configuration, but this configuration method needs to consume a large amount of human cost. SUMMARY

[0004] The present application aims to provide a table question and answer pair automatic extraction method, device, equipment and medium, which can extract question and answer pairs according to the appearance features and content features of the table to be analyzed, and improve the efficiency of extracting question and answer pairs.

[0005] To achieve the above-mentioned purpose, the present application provides a table question and answer pair automatic extraction method, comprising:

[0006] obtaining a table to be analyzed;

[0007] extracting the appearance features of the table to be analyzed;

[0008] judging the table type of the table to be analyzed according to the appearance features;

[0009] if the judgment fails, extracting the content features of the table to be analyzed;

[0010] determining the first table type of the table to be analyzed according to the content features;

[0011] extracting the table question and the table answer of the table to be analyzed according to the first table type; and combining the table question and the table answer to form a question and answer pair.

[0012] Further, after judging the table type of the table to be analyzed according to the appearance features, the method further comprises:

[0013] if the judgment is successful, obtaining the second table type of the table to be analyzed, extracting the table question and the table answer of the table to be analyzed according to the second table type; and combining the table question and the table answer to form a question and answer pair.

[0014] the appearance features of each row cell of each row of the table to be analyzed are read;

[0015] the appearance features of each row cell of each row of the table to be analyzed are read;

[0016] the appearance features of each column cell of each column of the table to be analyzed are read;

[0017] the appearance features include font thickness, font color, background color and font size.

[0018] the content features of the table to be analyzed are extracted, including:

[0019] whether each row and each column of the table to be analyzed has repeated content is detected, if yes, the internal content repetition degree of the table to be analyzed is 1; if not, the internal content repetition degree of the table to be analyzed is 0;

[0020] text characters of each row and each column of the table to be analyzed are removed to obtain numerical characters; whether the numerical characters of each row and each column have numerical order is detected, if yes, the internal order degree of the table to be analyzed is 1; if not, the internal order degree of the table to be analyzed is 0;

[0021] the row text length of each row of the table to be analyzed is extracted, the row text length variance of each row of the table to be analyzed is calculated according to the row text length; the row structure consistency degrees of all rows of the table to be analyzed are added to obtain the row structure consistency degree; the column text length of each column of the table to be analyzed is extracted, the column text length variance of each column of the table to be analyzed is calculated according to the column text length; the column structure consistency degrees of all columns of the table to be analyzed are added to obtain the column structure consistency degree; the row structure consistency degree and the column structure consistency degree are combined to obtain the structure consistency degree.

[0022] the table question and the table answer of the table to be analyzed are extracted according to the first table type, including:

[0023] if the first table type is a row table, the total length of the sentence of each row of the table to be analyzed is obtained;

[0024] the sentence pattern of each row of the table to be analyzed is obtained;

[0025] the row of the table to be analyzed, in which the total length of the sentence is greater than the total length threshold value and the sentence pattern is a question sentence, is taken as a question row;

[0026] the keyword of each row of the table to be analyzed is extracted, and the row having the most keywords with the question row is taken as a target row;

[0027] determining whether the semantic similarity between the target row and the question row is greater than a semantic similarity threshold, and if so, taking the target row as the answer row.

[0028] The extracting the table question and the table answer of the table to be analyzed according to the first table type comprises:

[0029] If the first table type is a list table, obtaining the total length of the sentence of each column of the table to be analyzed;

[0030] Obtaining the sentence pattern of each column of the table to be analyzed;

[0031] Taking the column in the table to be analyzed whose total length of the sentence is greater than a total length threshold and whose sentence pattern is a question sentence as a question column;

[0032] Extracting the keyword of each column of the table to be analyzed, and taking the column having the most keywords with the question column as a target column;

[0033] Determining whether the semantic similarity between the target column and the question column is greater than a semantic similarity threshold, and if so, taking the target column as the answer column.

[0034] The determining the table type of the table to be analyzed according to the appearance feature comprises:

[0035] If the table to be analyzed has one or more rows different from other rows in the appearance feature, the second table type is a row table;

[0036] If the table to be analyzed has one or more columns different from other columns in the appearance feature, the second table type is a list table;

[0037] If the appearance features of all rows of the table to be analyzed are the same and the appearance features of all columns of the table to be analyzed are the same, the determination fails.

[0038] The determining the first table type of the table to be analyzed according to the content feature comprises:

[0039] Weighted sum of the internal content repetition degree, the internal order degree and the structure consistency degree to obtain a cohesion degree; wherein the cohesion degree comprises a row cohesion degree and a column cohesion degree;

[0040] If the row cohesion degree is greater than the column cohesion degree, the first table type is a row table;

[0041] If the row cohesion degree is less than the column cohesion degree, the first table type is a list table.

[0042] The obtaining the table to be analyzed comprises:

[0043] Obtain a table containing question-answer pairs from a data source, wherein the data source includes web pages and documents.

[0044] The application also provides a table question-answer pair automatic extraction device, comprising:

[0045] A table obtaining module is configured to obtain a table to be analyzed.

[0046] An appearance feature extraction module is configured to extract appearance features of the table to be analyzed.

[0047] A table type judging module is configured to judge a table type of the table to be analyzed according to the appearance features.

[0048] A content feature extraction module is configured to extract content features of the table to be analyzed if the judgment fails.

[0049] A first table type determining module is configured to determine a first table type of the table to be analyzed according to the content features.

[0050] A question and answer extraction module is configured to extract a table question and a table answer of the table to be analyzed according to the first table type.

[0051] A question-answer pair forming module is configured to form a question-answer pair by combining the table question and the table answer.

[0052] The application also provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the table question-answer pair automatic extraction method and the steps of the table question-answer pair automatic extraction method.

[0053] The application also provides a computer-readable storage medium storing a computer program, wherein the computer program is executed by a processor to implement the table question-answer pair automatic extraction method and the steps of the table question-answer pair automatic extraction method.

[0054] The table question-answer pair automatic extraction method of the present application obtains a table to be analyzed; extracts the appearance features of the table to be analyzed; determines the table type of the table to be analyzed according to the appearance features; if the determination fails, extracts the content features of the table to be analyzed; determines the first table type of the table to be analyzed according to the content features; extracts the table question and the table answer of the table to be analyzed according to the first table type; and forms the table question and the table answer into a question-answer pair. The table type can be easily determined according to the appearance features of the table to be analyzed. When the table type cannot be determined according to the appearance features, the first table type is determined according to the content features, and the determination of the content features involves the statistics of text and numbers. The table type can be quickly determined by analyzing the appearance features first and then analyzing the content features. The question-answer pair is automatically extracted according to the table type, and the question-answer pair is extracted by the present application with higher efficiency than by manual extraction. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 The flowchart of the table question-answer pair automatic extraction method of an embodiment;

[0056] Figure 2 The structural schematic block diagram of the table question-answer pair automatic extraction device of an embodiment;

[0057] Figure 3 The structural schematic block diagram of the computer device of an embodiment.

[0058] The object, functional characteristics and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0059] In order to make the object, technical scheme and advantages of the present application more clear, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0060] It is to be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. It is further understood that the terms "comprise" and "comprising" and the like, when used in this specification, specify the presence of stated features, integers, steps, operations, elements, components, and / or groups, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It is further understood that when an element or layer is referred to as being "on" or "connected to" another element or layer, it can be directly on or connected to the other element or layer or intervening elements or layers can be present. In addition, the term "connected" and "coupled" as used herein can include wirelessly connected and wirelessly coupled. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0061] It is to be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. It is further understood that the terms "comprise" and "comprising" and the like, when used in this specification, specify the presence of stated features, integers, steps, operations, elements, components, and / or groups, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It is further understood that when an element or layer is referred to as being "on" or "connected to" another element or layer, it can be directly on or connected to the other element or layer or intervening elements or layers can be present. In addition, the term "connected" and "coupled" as used herein can include wirelessly connected and wirelessly coupled. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0062] Reference Figure 1 is a flowchart of a table question and answer pair automatic extraction method disclosed in the present application, comprising:

[0063] S1: obtaining a table to be analyzed.

[0064] Obtain a table containing question and answer pairs from a data source, wherein the data source includes web pages and documents.

[0065] Obtain table data from a web page, use the tool beautifulsoup4 combined with regular expressions to locate the position of the table in the web page, and extract the content of the table in the web page.

[0066] Use pandas and openpyxl tools to extract the content of the table in the document, which includes Excel and Word.

[0067] S2: extract the appearance features of the table to be analyzed.

[0068] Read the appearance features of all row cells of each row of the table to be analyzed;

[0069] Read the appearance features of all column cells of each column of the table to be analyzed;

[0070] The appearance features include font thickness, font color, background color, and font size.

[0071] By extracting appearance features of each row and each column of the table to be analyzed, appearance features of different rows and different columns can be compared.

[0072] S3: judging the table type of the table to be analyzed according to the appearance features.

[0073] If there is a row, the appearance features of all cells in the row are consistent, and inconsistent with the appearance features of other rows, indicating that the table is a row table. If there is a column, the appearance features of all cells in the column are consistent, and inconsistent with the appearance features of other columns, indicating that the table is a list table.

[0074] For a table to be detected in a webpage, the font thickness or background color of a certain row in the table to be detected in the webpage is detected by a script. If it is different from other rows, it is a row table. The font thickness or background color of a certain column in the table to be detected in the webpage is detected. If it is different from other columns, it is a list table.

[0075] For a table to be detected in a document, the style of each row of the table to be detected is obtained using the python tool openpyxl. If the style of a certain row is different from the style of other rows, the table to be detected is a row table. If the style of a certain column is different from the style of other columns, the table to be detected is a list table.

[0076] If the appearance features of all rows of the table to be analyzed are the same and the appearance features of all columns of the table to be analyzed are the same, the judgment fails.

[0077] S4: if the judgment fails, extract the content features of the table to be analyzed.

[0078] Detect whether each row and each column of the table to be analyzed has repeated content. If yes, the internal content repetition degree of the table to be analyzed is 1; if no, the internal content repetition degree of the table to be analyzed is 0.

[0079] Remove the text characters of each row and each column of the table to be analyzed to obtain the numerical characters. Detect whether the numerical characters of each row and each column have numerical order. If yes, the internal order degree of the table to be analyzed is 1; if no, the internal order degree of the table to be analyzed is 0.

[0080] extracting a row text length of each row of the table to be analyzed, calculating a row text length variance of each row of the table to be analyzed according to the row text length, adding the row text length variances of all rows of the table to be analyzed to obtain a row structure consistency degree, extracting a column text length of each column of the table to be analyzed, calculating a column text length variance of each column of the table to be analyzed according to the column text length, adding the column text length variances of all columns of the table to be analyzed to obtain a column structure consistency degree, and combining the row structure consistency degree and the column structure consistency degree to obtain a structure consistency degree.

[0081] When the font thickness, font color, background color and font size appearance features of all rows or all columns of the table to be analyzed are consistent, it is impossible to determine whether the table to be analyzed is a row table or a column table according to the appearance features, and it is necessary to determine the first table type of the table to be analyzed according to the content features.

[0082] Compared with determining the second table type according to the appearance features, determining the first table type according to the content features is more accurate but takes more time.

[0083] S5: determining the first table type of the table to be analyzed according to the content features.

[0084] performing weighted summation on the internal content repetition degree, the internal order degree and the structure consistency degree to obtain a cohesion degree, wherein the cohesion degree includes a row cohesion degree and a column cohesion degree;

[0085] if the row cohesion degree is greater than the column cohesion degree, the first table type is a row table;

[0086] if the row cohesion degree is less than the column cohesion degree, the first table type is a column table.

[0087] The greater the row cohesion degree is, the stronger the regularity of the rows of the table to be analyzed is, and the greater the column cohesion degree is, the stronger the regularity of the columns of the table to be analyzed is.

[0088] S6: extracting a table question and a table answer of the table to be analyzed according to the first table type.

[0089] if the first table type is a row table, obtaining a total length of a sentence of each row of the table to be analyzed;

[0090] obtaining a sentence pattern of the sentence of each row of the table to be analyzed;

[0091] regarding a row in which the total length of the sentence is greater than a total length threshold value and the sentence pattern is a question as a question row;

[0092] extracting a keyword of each row of the table to be analyzed, and regarding a row having the most keywords with the question row as a target row.

[0093] determining whether the semantic similarity between the target row and the question row is greater than a semantic similarity threshold, and if so, taking the target row as the answer row.

[0094] if the first table type is a list table, obtaining the total length of the sentence of each column of the table to be analyzed;

[0095] obtaining the sentence pattern of each column of the table to be analyzed;

[0096] taking the column of the table to be analyzed, in which the total length of the sentence is greater than a total length threshold and the sentence pattern is a question, as a question column;

[0097] extracting the keywords of each column of the table to be analyzed, and taking the column having the most keywords in common with the question column as a target column;

[0098] determining whether the semantic similarity between the target column and the question column is greater than a semantic similarity threshold, and if so, taking the target column as the answer column.

[0099] the answer row and the question row have the most same keywords, and the answer column and the question column have the most same keywords.

[0100] Preferably, when the first table type is a row table, after determining the question row, the semantic representation of each row of the table to be analyzed is performed. The row having the greatest semantic similarity with the question row is taken as a target row, and it is determined whether the keywords shared by the target row and the question row are greater than a keyword threshold, and if so, the target row is taken as the answer row.

[0101] Preferably, when the first table type is a list table, after determining the question column, the semantic representation of each column of the table to be analyzed is performed. The column having the greatest semantic similarity with the question column is taken as a target column, and it is determined whether the keywords shared by the target column and the question column are greater than a keyword threshold, and if so, the target column is taken as the answer column.

[0102] S7: forming a question-answer pair by the table question and the table answer.

[0103] if the first table type or the second table type is a row table, forming a question-answer pair by the question row and the answer row; if the first table type or the second table type is a list table, forming a question-answer pair by the question column and the answer column.

[0104] As described above, the table question and answer pair automatic extraction method of the embodiment of the present application obtains a table to be analyzed; extracts appearance features of the table to be analyzed; determines a table type of the table to be analyzed according to the appearance features; if the determination fails, extracts content features of the table to be analyzed; determines a first table type of the table to be analyzed according to the content features; extracts a table question and a table answer of the table to be analyzed according to the first table type; and forms a question and answer pair by the table question and the table answer. The appearance features of the table to be analyzed are easier to determine the table type. When the appearance features cannot determine the table type, the first table type is determined according to the content features, and the determination of the content features involves the statistics of text and numbers. The analysis of the appearance features and then the analysis of the content features can quickly determine the table type, and the question and answer pair are automatically extracted according to the table type, which has higher efficiency than the manual extraction of the question and answer pair.

[0105] In one embodiment, after the determination of the table type of the table to be analyzed according to the appearance features, the method further includes:

[0106] If the determination succeeds, a second table type of the table to be analyzed is obtained, a table question and a table answer of the table to be analyzed are extracted according to the second table type, and the table question and the table answer are formed into a question and answer pair.

[0107] When there is one row with different appearance features from other rows in the table to be analyzed, the second table type is a row table, the row with different appearance features is taken as a question row, and an answer row matching the question row is searched in the table to be analyzed. When there are multiple rows with different appearance features from other rows in the table to be analyzed, the second table type is a row table. One row with different appearance features is selected as a question row, and an answer row matching the question row is searched in the table to be analyzed.

[0108] When there is one column with different appearance features from other columns in the table to be analyzed, the second table type is a list table, the column with different appearance features is taken as a question column, and an answer column matching the question column is searched in the table to be analyzed. When there are multiple columns with different appearance features from other columns in the table to be analyzed, the second table type is a list table. One column with different appearance features is selected as a question column, and an answer column matching the question column is searched in the table to be analyzed.

[0109] In one embodiment, the extraction of the appearance features of the table to be analyzed includes:

[0110] Reading appearance features of all row cells of each row of the table to be analyzed;

[0111] Reading appearance features of all column cells of each column of the table to be analyzed;

[0112] The appearance features include font thickness, font color, background color, and font size.

[0113] It can be determined whether the table to be analyzed is a row table or a list table according to the appearance features of the table to be analyzed.

[0114] The appearance features of each cell in each row or each column of the table are read using the beautifulsoup4 or openpyxl tool, such as bold, font color, background color, and font size.

[0115] As described above, by extracting the appearance features of each row and each column of the table to be analyzed, the appearance features of different rows and different columns can be compared, and rows with different appearance features from other rows are screened to obtain target rows. Columns with different appearance features from other columns are screened to obtain target columns.

[0116] In one embodiment, the extracting the content features of the table to be analyzed comprises:

[0117] It is detected whether each row and each column of the table to be analyzed has repeated content, if yes, the internal content repetition degree of the table to be analyzed is 1; if not, the internal content repetition degree of the table to be analyzed is 0.

[0118] The internal content repetition degree of the table to be analyzed includes row internal content repetition degree r_a and column internal content repetition degree l_a.

[0119] When different cells in a row or several rows in the table to be analyzed have partial repeated content or complete repeated content, it indicates that the content of the row or the rows may be derived from the same field, and the row internal content repetition degree r_a = 1.

[0120] When different cells in a column or several columns in the table to be analyzed have partial repeated content or complete repeated content, it indicates that the content of the column or the columns may be derived from the same field, and the column internal content repetition degree l_a = 1.

[0121] When all rows in the table to be analyzed do not have partial repeated content, it indicates that the content of all rows is not derived from the same field, and the row internal content repetition degree r_a = 0.

[0122] When all columns in the table to be analyzed do not have partial repeated content, it indicates that the content of all columns is not derived from the same field, and the column internal content repetition degree l_a = 0.

[0123] Removing the text characters of each row and each column of the table to be analyzed to obtain the numerical characters; detecting whether the numerical characters of each row and each column have numerical order, if yes, the internal order degree of the table to be analyzed is 1; if not, the internal order degree of the table to be analyzed is 0;

[0124] Removing the text of each row and each column of the table to be analyzed, detecting whether only numbers are left, if yes, adjacent numbers of each row and each column are subtracted one by one, if a certain row or a certain column presents an arithmetic sequence relationship, the table to be analyzed has internal order degree. Internal order degree includes row internal order degree r_b and column internal order degree l_b. When a certain row or several rows present an arithmetic sequence relationship, r_b = 1, when a certain column or several columns present an arithmetic sequence relationship, l_b = 1.

[0125] For example, after removing the text of each row and each column of the table to be analyzed, it is detected that the second row only has the number 9753, and the adjacent subtraction of the number 9753 obtains an arithmetic sequence 222, and the table to be analyzed has internal order degree.

[0126] Extracting the row text length of each row of the table to be analyzed, calculating the row text length variance of each row of the table to be analyzed according to the row text length. Adding the row text length variances of all rows of the table to be analyzed to obtain the row structure consistency degree.

[0127] Extracting the column text length of each column of the table to be analyzed, calculating the column text length variance of each column of the table to be analyzed according to the column text length. Adding the column text length variances of all columns of the table to be analyzed to obtain the column structure consistency degree. The row structure consistency degree and the column structure consistency degree constitute the structure consistency degree.

[0128] The smaller the row structure consistency degree r_c is, the higher the content consistency of the row is, and the larger the row structure consistency degree r_c is, the lower the content consistency of the row is.

[0129] The smaller the column structure consistency degree l_c is, the higher the content consistency of the column is, and the larger the column structure consistency degree l_c is, the lower the content consistency of the column is.

[0130] The structure consistency degree of the table to be analyzed includes the row structure consistency degree r_c and the column structure consistency degree l_c.

[0131] As described above, the internal content repetition degree, the internal order degree and the structure consistency degree of the table to be analyzed are extracted, the internal content repetition degree reflects the content repetition of the row or the column, the internal order degree reflects the content order of the row or the column, and the internal content repetition degree reflects the content repetition degree of the row or the column. The first table type can be judged through the internal content repetition degree, the internal order degree and the structure consistency degree.

[0132] In one embodiment, the extracting the table question and the table answer of the table to be analyzed according to the first table type comprises:

[0133] If the first table type is a row table, obtaining the total length of the sentence of each row of the table to be analyzed;

[0134] Obtaining the sentence pattern of each row of the table to be analyzed;

[0135] Taking the row in which the total length of the sentence is greater than the total length threshold and the sentence pattern is a question sentence as a question row in the table to be analyzed;

[0136] Extracting the keywords of each row of the table to be analyzed, and taking the row having the most keywords with the question row as a target row;

[0137] Determining whether the semantic similarity of the target row and the question row is greater than a semantic similarity threshold, and if yes, taking the target row as the answer row.

[0138] If the first table type is a row table, generally, one row in the table to be analyzed is a question row, and another row is an answer row. The answer row has the most same keywords with the question row.

[0139] Firstly, it is necessary to identify which row is a question row. The question row is generally a sentence, the length of the sentence is within a certain range, and the pattern of the sentence is a question sentence or a negative sentence. The question row can also be a row containing keywords such as “phenomenon”, “problem” and “fault”.

[0140] Setting a total length threshold, and taking the row in which the total length of the sentence is greater than the total length threshold and the pattern of the sentence is a question sentence as a question row in the table to be analyzed.

[0141] Extracting all the keywords of the question row, extracting the keywords of each row of the table to be analyzed, and screening out the row having the most same keywords with the question row as a target row. Setting a semantic similarity threshold, performing semantic representation on the question row and the target row, determining whether the semantic similarity of the target row and the question row is greater than the semantic similarity threshold, and if yes, taking the target row as the answer row. If no, the target row is removed, the row having the most same keywords with the question row is screened out from the table to be analyzed from which the target row is removed as a target row, and semantic representation is performed on the question row and the target row, and it is determined whether the semantic similarity of the target row and the question row is greater than the semantic similarity threshold.

[0142] As described above, the question row is generally a question sentence and has a length within a certain range. The total length of a sentence in each row of the table to be analyzed is obtained, and the sentence pattern of the sentence in each row of the table to be analyzed is obtained. The row in the table to be analyzed, in which the total length of the sentence is greater than the total length threshold and the sentence pattern is a question sentence, is taken as the question row. The keywords in each row of the table to be analyzed are extracted, and the row having the most keywords with the question row is taken as the target row. It is judged whether the semantic similarity between the target row and the question row is greater than the semantic similarity threshold. If yes, the target row is taken as the answer row. There are more keywords between the question row and the answer row, and the question row and the answer row have a higher semantic similarity. The keywords in each row of the table to be analyzed are extracted, and the row having the most keywords with the question row is taken as the target row. It is judged whether the semantic similarity between the target row and the question row is greater than the semantic similarity threshold. If yes, the target row is taken as the answer row.

[0143] In one embodiment, the extracting the table question and the table answer of the table to be analyzed according to the first table type comprises:

[0144] If the first table type is a list table, the total length of a sentence in each column of the table to be analyzed is obtained.

[0145] The sentence pattern of the sentence in each column of the table to be analyzed is obtained.

[0146] The column in the table to be analyzed, in which the total length of the sentence is greater than the total length threshold and the sentence pattern is a question sentence, is taken as the question column.

[0147] The keywords in each column of the table to be analyzed are extracted, and the column having the most keywords with the question column is taken as the target column.

[0148] It is judged whether the semantic similarity between the target column and the question column is greater than the semantic similarity threshold. If yes, the target column is taken as the answer column.

[0149] The specific process of extracting the question column and the answer column is the same as that of extracting the question row and the answer row, which will not be described here.

[0150] As described above, the question column is generally a question sentence and has a length within a certain range. The total length of the sentence in each column of the table to be analyzed is obtained. The sentence pattern of the sentence in each column of the table to be analyzed is obtained. The column in the table to be analyzed, whose total length of the sentence is greater than the total length threshold and whose sentence pattern is a question sentence, is taken as a question column. The keywords of each column of the table to be analyzed are extracted, and the column having the most keywords with the question column is taken as a target column. It is judged whether the semantic similarity of the target column and the question column is greater than the semantic similarity threshold. If yes, the target column is taken as the answer column. The answer column matched with the question column is obtained according to the number of keywords and the semantic similarity, and has a higher accuracy.

[0151] In one embodiment, the table type of the table to be analyzed is determined according to the appearance feature, comprising:

[0152] If one or more rows of the table to be analyzed have different appearance features from other rows, the second table type is a row table.

[0153] If one or more columns of the table to be analyzed have different appearance features from other columns, the second table type is a list table.

[0154] If the appearance features of all rows of the table to be analyzed are the same and the appearance features of all columns of the table to be analyzed are the same, the determination fails.

[0155] The appearance feature includes font thickness, font color, background color and font size.

[0156] If there is a row, the appearance features of all cells in the row are consistent, and the appearance features of other rows are inconsistent, which indicates that the second table feature is a row table. If there is a column, the appearance features of all cells in the column are consistent, and the appearance features of other columns are inconsistent, which indicates that the second table feature is a list table.

[0157] For the table to be detected in the webpage, the table to be detected in the webpage is detected by a script. If the font thickness or the background color of a certain row is different from those of other rows, it indicates that the second table feature is a row table. The table to be detected in the webpage is detected. If the font thickness or the background color of a certain column is different from those of other columns, it indicates that the second table feature is a list table.

[0158] For the table to be detected in the document, the style of each row of the table to be detected is obtained by using the python tool openpyxl. If the style of a certain row is different from that of other rows, it indicates that the second table feature is a row table. If the style of a certain column is different from that of other columns, it indicates that the second table feature is a list table.

[0159] If the appearance features of all rows of the table to be analyzed are the same and the appearance features of all columns of the table to be analyzed are the same, the judgment fails.

[0160] As described above, in the embodiment, the table type of the table to be analyzed is judged according to the appearance features, if the appearance features of one or more rows of the table to be analyzed are different from those of other rows, the second table type is a row table, if the appearance features of one or more columns of the table to be analyzed are different from those of other columns, the second table type is a column table, and if the appearance features of all rows of the table to be analyzed are the same and the appearance features of all columns of the table to be analyzed are the same, the judgment fails. Since the problem row is generally different from other rows and the problem column is generally different from other columns, by screening the problem row different from other rows or the problem column different from other columns, it can be judged whether the second table feature is a row table or a column table.

[0161] In one embodiment, the first table type of the table to be analyzed is determined according to the content features, comprising:

[0162] The internal content repetition degree, the internal sequence degree and the structure consistency degree are weighted and summed to obtain a cohesion degree; wherein the cohesion degree comprises a row cohesion degree and a column cohesion degree;

[0163] If the row cohesion degree is greater than the column cohesion degree, the first table type is a row table;

[0164] If the row cohesion degree is less than the column cohesion degree, the first table type is a column table.

[0165] An internal content repetition degree coefficient x is set, an internal sequence degree coefficient y is set, a structure consistency degree coefficient z is set, the row cohesion degree and the column cohesion degree of the table to be analyzed are calculated, and the formula of the row cohesion degree is as follows:

[0166] r = x * r_a + y * r_b + z * r_c;

[0167] Wherein, r_a is the row internal content repetition degree, r_b is the row internal sequence degree, and r_c is the row structure consistency degree.

[0168] The formula of the column cohesion degree is as follows:

[0169] l = x * l_a + y * l_b + z * l_c;

[0170] Wherein, l_a is the column internal content repetition degree, l_b is the column internal sequence degree, and l_c is the column structure consistency degree.

[0171] The higher the row cohesion is, the stronger the regularity of information contained in the row is, and the greater the possibility that the first table type is a row table is. The higher the column cohesion is, the stronger the regularity of information contained in the column is, and the greater the possibility that the first table type is a list table is.

[0172] The row cohesion and the column cohesion are compared in size. If the row cohesion is greater than the column cohesion, it is indicated that the first table type is a row table. If the row cohesion is less than the column cohesion, it is indicated that the first table type is a list table. If the row cohesion is equal to the column cohesion, it is indicated that the table to be analyzed can be a row table or a list table, and needs to be further judged.

[0173] As described above, the first table type of the table to be analyzed is determined according to the content characteristics in the embodiment of the application. The internal content repetition degree, the internal order degree and the structure consistency degree are weighted and summed to obtain the cohesion. The cohesion includes the row cohesion and the column cohesion. If the row cohesion is greater than the column cohesion, the first table type is a row table. If the row cohesion is less than the column cohesion, the first table type is a list table. The higher the row cohesion is, the stronger the regularity of information contained in the row is. The higher the column cohesion is, the stronger the regularity of information contained in the column is. By comparing the size of the row cohesion and the column cohesion, it can be determined whether the table to be analyzed is a row table or a list table.

[0174] In one embodiment, the table to be analyzed is obtained, including:

[0175] The table containing the question-answer pair is obtained from the data source, wherein the data source includes a webpage and a document.

[0176] The table data is obtained from the webpage. The position of the table in the webpage is located by using the tool beautifulsoup4 combined with a regular expression, and the content of the table in the webpage is extracted.

[0177] The content of the table in the document is extracted by using the tools pandas and openpyxl. The document includes Excel and Word.

[0178] As described above, the table to be analyzed is obtained from the data source in the embodiment of the application. The table containing the question-answer pair is obtained from the data source, wherein the data source includes a webpage and a document. The table in the webpage and the table to be analyzed in the document are extracted by different tools. Subsequently, the question-answer pair is extracted according to the table to be analyzed.

[0179] Reference Figure 2 is a structural schematic block diagram of a table question-answer pair automatic extraction device disclosed by the application. The device includes:

[0180] The table obtaining module 10 is configured to obtain a table to be analyzed.

[0181] Appearance feature extraction module 20 is used to extract the appearance features of the table to be analyzed;

[0182] The table type determination module 30 is used to determine the table type of the table to be analyzed based on the appearance characteristics.

[0183] The content feature extraction module 40 is used to extract the content features of the table to be analyzed if the judgment fails.

[0184] First table type determination module 50 is used to determine the first table type of the table to be analyzed based on the content characteristics;

[0185] Question and answer extraction module 60 is used to extract the table questions and table answers of the table to be analyzed according to the first table type;

[0186] The question-answer pair composition module 70 is used to compose the table questions and the table answers into question-answer pairs.

[0187] As described above, the automatic question-answer pair extraction device of this application embodiment can realize the automatic question-answer pair extraction method.

[0188] Reference Figure 3 This application also provides a computer device whose internal structure can be as follows: Figure 3 As shown. The computer device includes a processor, memory, network interface, and database connected via a system bus. The processor is designed to provide computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores operating devices, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The database stores appearance features and content features, etc. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for automatically extracting table question-answer pairs, including the following steps: obtaining a table to be analyzed; extracting the appearance features of the table to be analyzed; determining the table type of the table to be analyzed based on the appearance features; if the determination fails, extracting the content features of the table to be analyzed; determining a first table type of the table to be analyzed based on the content features; extracting the table question and table answer of the table to be analyzed based on the first table type; and forming a question-answer pair from the table question and the table answer.

[0189] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer equipment on which the present application is applied.

[0190] An embodiment of the present application further provides a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement a method for automatically extracting a table question-answer pair, comprising the following steps: obtaining a table to be analyzed; extracting appearance features of the table to be analyzed; judging a table type of the table to be analyzed according to the appearance features; if the judgment fails, extracting content features of the table to be analyzed; determining a first table type of the table to be analyzed according to the content features; extracting a table question and a table answer of the table to be analyzed according to the first table type; and composing the table question and the table answer into a question-answer pair. It can be understood that the computer readable storage medium in the embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0191] It can be understood by those skilled in the art that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiments. Any reference to memory, storage, databases, and other media in this application and throughout the embodiments refers to both non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), and flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0192] It should be noted that in this document, the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusions, so that a process, device, article, or method including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or inherent to such a process, device, article, or method. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, device, article, or method including the element.

[0193] The above merely provides the preferred embodiments of the present application, and is not intended to limit the patent scope of the present application. Any equivalent structure and equivalent flow transformation based on the content of the present application specification and drawings, and direct and indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for automatically extracting answer pairs of table questions, characterized in that, The method comprises the following steps: acquiring a table to be analyzed; extracting appearance features of the table to be analyzed; judging a table type of the table to be analyzed according to the appearance features; if the judgment fails, extracting content features of the table to be analyzed; determining a first table type of the table to be analyzed according to the content features; extracting table questions and table answers of the table to be analyzed according to the first table type; composing the table questions and the table answers into question-answer pairs; the step of extracting the content features of the table to be analyzed comprises the following steps: detecting whether each row and each column of the table to be analyzed has repeated content, if yes, the internal content repetition degree of the table to be analyzed is 1; if not, the internal content repetition degree of the table to be analyzed is 0; removing literal characters of each row and each column of the table to be analyzed to obtain digital characters; detecting whether the digital characters of each row and each column have numerical order, if yes, the internal order degree of the table to be analyzed is 1; if not, the internal order degree of the table to be analyzed is 0; extracting row text lengths of each row of the table to be analyzed, calculating row text length variances of each row of the table to be analyzed according to the row text lengths; adding the row text length variances of all rows of the table to be analyzed to obtain a row structure consistency degree; extracting column text lengths of each column of the table to be analyzed, calculating column text length variances of each column of the table to be analyzed according to the column text lengths; adding the column text length variances of all columns of the table to be analyzed to obtain a column structure consistency degree; and composing the row structure consistency degree and the column structure consistency degree into a structure consistency degree.

2. The form question answer pair automatic extraction method according to claim 1, characterized in that, after the step of judging the table type of the table to be analyzed according to the appearance features, the method further comprises the following steps: if the judgment succeeds, obtaining a second table type of the table to be analyzed, extracting table questions and table answers of the table to be analyzed according to the second table type; and composing the table questions and the table answers into question-answer pairs. 3.The form question answer pair automatic extraction method according to claim 1, characterized in that, the step of extracting the appearance features of the table to be analyzed comprises the following steps: reading appearance features of all row cells of each row of the table to be analyzed; reading appearance features of all column cells of each column of the table to be analyzed; the appearance features comprise font thickness, font color, background color and font size.

4. The form question answer pair automatic extraction method according to claim 1, characterized in that, the step of extracting the table questions and the table answers of the table to be analyzed according to the first table type comprises the following steps: if the first table type is a row table, acquiring total lengths of sentences of each row of the table to be analyzed; acquiring sentence patterns of each row of the table to be analyzed; taking a row in which the total length of the sentence is greater than a total length threshold value and the sentence pattern is a question sentence as a question row in the table to be analyzed; extracting keywords of each row of the table to be analyzed, and taking a row having the most keywords with the question row as a target row; judging whether a semantic similarity degree between the target row and the question row is greater than a semantic similarity threshold value, if yes, taking the target row as an answer row.

5. The form question answer pair automatic extraction method according to claim 4, characterized in that, the step of extracting the table questions and the table answers of the table to be analyzed according to the first table type comprises the following steps: If the first table type is a list table, obtaining total length of sentences of each column of the table to be analyzed; Obtaining sentence patterns of each column of the table to be analyzed; Taking a column in which the total length of the sentences is greater than a total length threshold and the sentence pattern is a question as a question column in the table to be analyzed; Extracting keywords of each column of the table to be analyzed, and taking a column having the most keywords with the question column as a target column; If the semantic similarity between the target column and the question column is greater than a semantic similarity threshold, taking the target column as the answer column.

6. The form question answer pair automatic extraction method according to claim 1, wherein, The determining the table type of the table to be analyzed according to the appearance feature comprises: If one or more rows of the table to be analyzed have the appearance feature different from other rows, the table type of the table to be analyzed is a row table; If one or more columns of the table to be analyzed have the appearance feature different from other columns, the table type of the table to be analyzed is a list table; If the appearance features of all rows of the table to be analyzed are the same and the appearance features of all columns of the table to be analyzed are the same, the determining fails.

7. The form question answer pair automatic extraction method according to claim 1, wherein, The determining the first table type of the table to be analyzed according to the content feature comprises: Weighted summing the internal content repetition degree, the internal order degree and the structure consistency degree to obtain a cohesion degree; wherein the cohesion degree comprises a row cohesion degree and a column cohesion degree; If the row cohesion degree is greater than the column cohesion degree, the first table type is a row table; If the row cohesion degree is less than the column cohesion degree, the first table type is a list table.

8. The form question answer pair automatic extraction method according to claim 1, characterized in that, The obtaining the table to be analyzed comprises: Obtaining a table containing question-answer pairs from a data source, wherein the data source comprises a webpage and a document.

9. A table question answer pair automatic extraction apparatus characterized by comprising: It comprises: A table obtaining module is configured to obtain a table to be analyzed; An appearance feature extraction module is configured to extract an appearance feature of the table to be analyzed; A table type determining module is configured to determine a table type of the table to be analyzed according to the appearance feature; A content feature extraction module is configured to extract a content feature of the table to be analyzed if the determining fails; A first table type determining module is configured to determine a first table type of the table to be analyzed according to the content feature; A question and answer extraction module is configured to extract a table question and a table answer of the table to be analyzed according to the first table type; A question-answer pair forming module is configured to form a question-answer pair by using the table question and the table answer; The extracting the content feature of the table to be analyzed comprises: Detecting whether each row and each column of the table to be analyzed has repeated content, if yes, an internal content repetition degree of the table to be analyzed is 1; If no, the internal content repetition degree of the table to be analyzed is 0; Removing text characters of each row and each column of the table to be analyzed to obtain numerical characters, and detecting whether the numerical characters of each row and each column have numerical order, if yes, an internal order degree of the table to be analyzed is 1; if no, the internal order degree of the table to be analyzed is 0; The row text length of each row of the table to be analyzed is extracted, and the row text length variance of each row of the table to be analyzed is calculated according to the row text length; the row text length variances of all rows of the table to be analyzed are added to obtain a row structure consistency degree; the column text length of each column of the table to be analyzed is extracted, and the column text length variance of each column of the table to be analyzed is calculated according to the column text length; the column text length variances of all columns of the table to be analyzed are added to obtain a column structure consistency degree; and the row structure consistency degree and the column structure consistency degree are combined to obtain a structure consistency degree. 10.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-9. The processor implements the steps of the table question and answer automatic extraction method in any one of claims 1 to 8 when executing the computer program.

11. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the table question and answer automatic extraction method in any one of claims 1 to 8.

Images