Table information extraction method, system and device based on compressed pre-training language model

Abstract

The application discloses a table information extraction method, system and device based on a compressed pre-training language model. The application models the table information extraction as a reading comprehension problem and inputs the pre-training language model, thereby avoiding error propagation and reducing the participation of rules. Meanwhile, the strong generalization ability and robustness of the pre-training language model can obtain relatively accurate cells in the table that can answer natural language questions, thereby completing the extraction of the associated information in the table.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing, specifically relating to a method, system, and device for extracting table information based on a compressed pre-trained language model. Background Technology

[0002] Table information extraction is a natural language processing task with extremely high computational complexity. The performance of models has always been limited by the small amount of training data, poor generalization ability of models in different scenarios, and low extraction efficiency.

[0003] To address the issue of improving the efficiency of information extraction from tables in a general way, traditional approaches include first converting the table into a relational database and then using Structured Query Language (SCL) statements for information extraction; or writing complex rules for information extraction specific to a particular type of table. The former approach leads to error propagation, meaning that problems in the table conversion will affect the accuracy of subsequent SCL queries. The latter approach is only applicable to a specific type of table and cannot be generalized. Furthermore, as the tables become increasingly complex, using pre-trained language models introduces significant time complexity, making real-time inference impossible.

[0004] In recent years, the rapid development of pre-trained neural network models has greatly improved the performance of downstream natural language processing tasks, making it essential to apply pre-trained language models to table information extraction tasks. However, the computational complexity of table information extraction is extremely high, which significantly reduces the inference speed of the model, making it unsuitable for use in real-world scenarios. Summary of the Invention

[0005] The purpose of this invention is to universally solve the problems of poor generalization ability and low extraction efficiency in table information extraction, and to provide a table information extraction method, system and device based on compressed pre-trained language model.

[0006] The specific technical solution adopted in this invention is as follows:

[0007] In a first aspect, the present invention provides a method for extracting tabular information based on a compressed pre-trained language model, comprising:

[0008] S1. Convert the target information fields to be extracted into natural language questions;

[0009] S2. Extract information from each non-header row of the target table, and concatenate the cell content and position information of each row into row feature information.

[0010] S3. Based on the compressed pre-trained language model after neuron pruning and weight compression, the line feature information and the natural language question are input into the compressed pre-trained language model to classify the text in the line feature information, and output classification labels to represent whether the text is the answer to the natural language question; locate the text corresponding to the classification label marked as the answer, and extract it from the table as the extraction result.

[0011] As a preferred embodiment of the first aspect above, the target table is pre-identified by recognizing the table's header row, total number of rows, total number of columns, and the number of columns in each row excluding empty cells.

[0012] As a preferred embodiment of the first aspect, the compressed pre-trained language model is obtained by pruning neurons and compressing weights from the pre-trained language model. In the neuron pruning operation, based on the mutual information between neurons in each layer, the pre-trained language model is pruned layer by layer from the highest layer to the lowest layer, and each layer retains a fixed number of neurons with the largest mutual information with the neurons retained in the previous layer. In the weight compression operation, the model weight matrix of the pre-trained language model is compressed according to the neurons retained after pruning at each layer.

[0013] As a preferred embodiment of the first aspect, in the neuron pruning operation, the Softmax layer of the pre-trained language model is taken as the highest layer. The neurons in the Softmax layer are kept unchanged. Then, a preset number of neurons are randomly selected from the next layer of the network to be retained, while the rest are pruned. The performance difference of the pre-trained language model before and after pruning is tested on the dataset. If the performance difference meets the preset requirements, the pruning of this layer of the network is completed. Otherwise, the neuron selection and pruning are repeated. For each other layer of the pre-trained language model, a preset number of neurons with the largest mutual information with the neurons retained in the previous layer are selected, and the rest are pruned.

[0014] As a preferred embodiment of the first aspect above, the position information of each row in the target table includes the row number of the row among all non-header rows of the target table, and the column number of the last non-empty cell in the row.

[0015] As a preferred embodiment of the first aspect above, the target information field is converted into a natural language question by calling a template library for template filling.

[0016] As a preferred embodiment of the first aspect above, the row feature information extracted from the target table is input into a compressed pre-trained language model row by row along with the natural language question for classification. When the model output contains a label corresponding to the answer, it is considered that the information to be extracted has been obtained, and the input of subsequent row feature information is stopped.

[0017] Secondly, the present invention provides a method for extracting tabular information based on a compressed pre-trained language model, comprising:

[0018] The question generation module is used to convert the target information fields to be extracted into natural language questions.

[0019] The row feature generation module is used to extract information from each non-header row of the target table, and to concatenate the cell content and position information of each row into row feature information.

[0020] The table information extraction module is used to input the row feature information and the natural language question into the compressed pre-trained language model based on the compressed pre-trained language model after neuron pruning and weight compression. The module classifies the text in the row feature information and outputs classification labels to represent whether the text is the answer to the natural language question. The module locates the text corresponding to the classification label marked as the answer and extracts it from the table as the extraction result.

[0021] Thirdly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the table information extraction method based on a compressed pre-trained language model as described in any of the first aspects.

[0022] Fourthly, the present invention provides a computer electronic device, which includes a memory and a processor;

[0023] The memory is used to store computer programs;

[0024] The processor is configured to, when executing the computer program, implement the table information extraction method based on a compressed pre-trained language model as described in any of the first aspects.

[0025] Compared with the prior art, the present invention has the following advantages:

[0026] This invention provides a table information extraction scheme based on a compressed pre-trained language model. By modeling table information extraction as a reading comprehension question and feeding it into the compressed pre-trained language model, error propagation is avoided and the involvement of rules is reduced. Simultaneously, leveraging the strong generalization ability and robustness of the pre-trained language model, relatively accurate cells within the table that can answer natural language questions can be obtained, thus completing the extraction of relevant information within the table. Attached Figure Description

[0027] Figure 1 This is a flowchart of a method for extracting tabular information based on a compressed pre-trained language model.

[0028] Figure 2This is a schematic diagram illustrating the basic process of pruning and compression for an exemplary pre-trained language model.

[0029] Figure 3 This is a schematic diagram of a table information extraction system module based on a compressed pre-trained language model. Detailed Implementation

[0030] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in the various embodiments of the present invention can be combined accordingly without mutual conflict.

[0031] The inventive concept of this invention is to transform the task of extracting table information into a task of answering natural language questions. The table information extraction is modeled as a reading comprehension question and fed into a compressed pre-trained language model. The strong generalization ability and robustness of the pre-trained language model are used to accurately locate cells in the table that can answer the natural language question, thereby completing the extraction of related information within the table. Simultaneously, to address the problem of the significant time complexity caused by applying the pre-trained language model to complex tables, which prevents the system from performing real-time reasoning, the pre-trained language model is pruned and compressed, improving its efficiency in the table information extraction task and enabling real-time reasoning. The specific implementation of this invention is described in detail below.

[0032] In a preferred embodiment of the present invention, a method for extracting table information based on a compressed pre-trained language model is provided, comprising the following steps:

[0033] S1. Convert the target information fields to be extracted into natural language questions;

[0034] S2. Extract information from each non-header row of the target table, and concatenate the cell content and position information of each row into row feature information.

[0035] S3. Based on the compressed pre-trained language model after neuron pruning and weight compression, the line feature information and the natural language question are input into the compressed pre-trained language model to classify the text in the line feature information, and output classification labels to represent whether the text is the answer to the natural language question; locate the text corresponding to the classification label marked as the answer, and extract it from the table as the extraction result.

[0036] In the above embodiments of the present invention, the target information field to be extracted in step S1 can be input by the user or generated by the system through specified means, without limitation. Furthermore, the target information field can generally be converted into a natural language question by calling a template library for template filling; alternatively, other generative models can be used to achieve the conversion between the target information field and the natural language question.

[0037] Table 1 shows an example table that records information such as the continent, capital city, currency, and main language of different countries. When performing information extraction tasks from this table, for example, one can input "the capital of the United Kingdom" as the target information field to be extracted. A natural language question, "What is the capital city of the United Kingdom?", can be generated using a template-filling method.

[0038] Table 1 Example Target Table 1

[0039] nation Continent Capital City Circulating currency Main languages Japan Asia Tokyo JPY Japanese France Europe Paris EUR French USA America Washington Dollar English U.K. Europe London GBP English

[0040] In the above embodiments of the present invention, step S2 requires extracting information from each row of the non-header rows of the target table. Therefore, in order to identify the header and non-header rows within the table and enable the computer device to automatically extract table information, basic information identification of the target table can be performed in advance. That is, the header row (i.e., table header), total number of rows, total number of columns, and the number of columns in each row excluding empty cells can be obtained in advance through identification. Thus, based on this basic information, feature extraction can be performed on each row of the non-header rows of the table. For each row of the non-header rows in the target table, the cell content contained in each row and the position information of the row need to be concatenated to form the row feature information corresponding to that row. The position information of each row in the target table represents the row and column information of that row in the entire table. In the embodiments of the present invention, the position information of each row in the target table can include two types of information: the row number of the row in all non-header rows of the target table, and the column number of the last non-empty cell in the row. For example, in Table 1 above, the location information of the row containing the UK can be represented as "5 rows, 5 columns," where "5 rows" represents the row number of the UK row among all non-header rows in the target table, and "5 columns" represents the column number of the last non-empty cell in the UK row. Therefore, the row characteristic information of the UK row can be represented as "UK, 5 rows, 5 columns, Europe, London, Pound Sterling, English." The information "UK," "Europe," "London," "Pound Sterling," and "English" are all text content within the cells, while "5 rows, 5 columns" represents the corresponding location information. Different content within the entire row characteristic information can be concatenated using specific delimiters, such as the comma mentioned above, or the symbol "|," without limitation.

[0041] In the above embodiments of the present invention, the compressed pre-trained language model in step S3 is a pre-trained language model after neuron pruning and weight compression. The specific form of the original pre-trained language model is not limited; any large pre-trained model capable of answering natural language questions can be used. However, the original large pre-trained model has a large number of neurons, making it unsuitable for computationally complex tasks such as table information extraction. Therefore, the purpose of neuron pruning and weight compression is to reduce the sparsity of the model and improve the real-time performance of model inference. The compressed pre-trained language model is obtained by performing two operations: neuron pruning and weight compression. In the first neuron pruning operation, based on the mutual information between neurons, the pre-trained language model is pruned layer by layer from the highest layer to the lowest layer, retaining a fixed number of neurons with the largest mutual information with the neurons retained in the previous layer at each layer. In the second weight compression operation, the model weight matrix of the pre-trained language model is compressed based on the neurons retained after pruning at each layer.

[0042] Pruning and compression of pre-trained language models are accomplished using the mutual information between adjacent layers of a neural network model. The basic logic is to select the neurons with the highest mutual information between the current layer and the previous layer, retain them, and delete the rest; this process is repeated until the bottom layer of the model. The number of neurons to be pruned in each layer is manually determined. If a higher sparsity is desired, more neurons are removed; otherwise, a smaller number of neurons is chosen. Mutual information is a measure of the correlation between two random variables X and Y. The higher the mutual information, the greater the correlation between X and Y. It can be represented by I(X,Y). For example, if X = watermelon, Y = pear, and Z = dog, then obviously, the mutual information between X and Y is greater than the mutual information between X and Z, because the correlation between apples and pears is greater than the correlation between apples and cats.

[0043] This invention utilizes mutual information for hierarchical neural network model pruning, which is more global than traditional model pruning methods and thus better preserves the expressive power of the neural network model. Pruning is performed hierarchically from top to bottom, ensuring that the same number of model parameters are pruned in each layer. This avoids the problem of irregular network structure caused by random pruning, making the pruned model more structured.

[0044] See Figure 2 As shown, a basic process for pruning and compressing a pre-trained language model is given. The implementation steps are described in detail below for easier understanding.

[0045] Step 1: Prune the model from top to bottom. This step is divided into two sub-steps, a) and b), which are performed alternately, from the highest layer (Softmax layer) down to the lowest layer (layer 0). Before pruning, a number of neurons K to be retained in each network layer needs to be pre-defined. K is a hyperparameter that can be optimized according to actual needs.

[0046] a) Calculate mutual information. Assume the current layer is... Then it is necessary to start from the first Selecting from layers The set of neurons is denoted as . This makes it connect with the previous layer (i.e., the first layer). The selected layer one neuron The mutual information is maximized, which can be expressed mathematically as follows:

[0047]

[0048] After selection, This includes The first The neurons in the layer will then be used to prune the model parameters.

[0049] b) Pruning based on the selected neurons. Neurons are the result of each layer of the model; what we really need to prune are the model's parameters. The model's parameters can be represented by a matrix. After selecting the neurons with the highest mutual information between two adjacent layers, we can select the corresponding elements in the matrix based on the location of these neurons. For example... Figure 1 As shown, the first Layer and first If a layer selects black neurons, it means that the corresponding black parts in the matrix need to be retained, and the remaining white parts in the matrix need to be removed.

[0050] c) Starting from the highest level of the model, execute the above two sub-steps layer by layer until you reach the lowest level of the model.

[0051] However, it is important to note that this invention converts the field to be extracted into a natural language question. The natural language question, along with the text and location information of each cell, is input into a compressed pre-trained language model. This compressed pre-trained language model then performs matching and prediction to obtain the extracted answer. Therefore, the Softmax layer of the pre-trained language model needs to output the prediction result, and its number of neurons cannot be arbitrarily changed. Furthermore, the layer below the Softmax layer serves as the first layer for initial pruning, and the pruned neurons in this layer cannot be computed using mutual information. Therefore, the Softmax layer and the layer below it require special handling. In an embodiment of this invention, the Softmax layer of the pre-trained language model can be the highest layer. The neurons in the Softmax layer remain unchanged. Then, a predetermined number of neurons are randomly selected from the next layer of the network to be retained, while the rest are pruned. The performance difference of the pre-trained language model before and after pruning is tested on a dataset. If the performance difference meets the preset requirements, the pruning of this network layer is completed; otherwise, neuron selection and pruning are repeated. For each other network layer in the pre-trained language model, a predetermined number of neurons with the largest mutual information with the neurons retained in the previous layer are selected, and the rest are pruned.

[0052] It should be further clarified that the phrase "performance difference meets preset requirements" generally requires that the reduction in prediction accuracy of the pruned model be less than a certain threshold. In practical applications, the K neurons with the lowest reduction in prediction accuracy after pruning can be selected for retention, while the rest are pruned.

[0053] Step 2: Model weight compression. For example... Figure 2 As shown, after removing the white portion of the matrix, the black portion is not tightly packed. Therefore, the model's weights need to be compressed to form a new, tightly packed matrix. Figure 2 As shown, after deleting the white part, keeping the black part, and recompressing the model weights, a new 4*4 weight matrix was obtained, thus successfully compressing the original 10*10 matrix into a 4*4 matrix. The number of parameters was reduced to 16% of the original, which greatly improved the real-time inference performance of complex table information extraction tasks.

[0054] Once the pruned and compressed pre-trained language model is obtained, the row feature information and the natural language question can be input into the compressed pre-trained language model to classify the text in the row feature information. The output classification labels represent whether the text is the answer to the aforementioned natural language question. Using Table 1 as an example, the model input is the natural language question "What is the capital city of the United Kingdom?" and the row feature information "United Kingdom, 5 rows 5 columns, Europe, London, Pound Sterling, English". The model output is a label marked with [I, O], where the I label indicates that the text at the current position can be the answer to the question, and the O label indicates that the text at the current position cannot be the answer. The length of the model output sequence is the same as the length of the row feature information text; that is, each text will output a label to represent that the text at the current position can be the answer to the question. For the above model input, the compressed pre-trained language model will mark "London" as the answer to the task question. Therefore, the text corresponding to the classification label marked as the answer can be located through the output labels and extracted from the table as the extraction result.

[0055] In practical applications, to improve efficiency, the row feature information extracted from the target table can be input into a compressed pre-trained language model for classification, row by row, along with the natural language question. When the model output contains a label corresponding to the answer, it is considered that the information to be extracted has been obtained, and the input of row feature information of other non-header rows is stopped.

[0056] Of course, the table shown in Table 1 above is a relatively simple table. For such tables, the table information extraction method based on compressed pre-trained language models of this invention may not demonstrate its advantages in efficiency and versatility. However, in another embodiment of this invention, as shown in Table 2, this table information extraction method based on compressed pre-trained language models can be applied to other more complex tables. Such tables contain a large number of blank cells, and the content within individual cells is also more complex and unstructured, making it difficult for conventional methods to efficiently extract the required information.

[0057] Table 2 Example Target Table 2

[0058]

[0059] The table information extraction method provided by this invention can utilize a compressed pre-trained language model to achieve table information extraction based on natural language problem transformation. For example, for the field to be extracted received from user input, such as (Writer.AI, product features), the extraction process can be implemented according to the following steps a) to g):

[0060] a) First, preprocess the information in the table: identify the table header, total number of rows, total number of columns, and the last column number of each non-empty row.

[0061] b) Using a predefined template library, the fields to be extracted are converted into a natural language question through template filling. For example, if the fields to be extracted are (Writer.AI, product features), the natural language question is "What are the product features of Writer.AI?".

[0062] c) Represent the position and text information of each cell in the table. For each cell, provide the row number and the column number of the last non-empty cell in the row as the position information for that row. The text information for that row can be obtained by concatenating the text of each non-empty cell into a string using a specific delimiter such as "|". For example, in Table 2, the row number and the column number of the last non-empty cell of "PIC.AI" are encoded, and the position information is encoded as text, such as "PIC.AI|2 rows 5 columns|Image|Location: Generate images through natural language description. Features: Good Discord community discussion environment, generates 4 images at once.|Google, Microsoft|Pre-A".

[0063] d) Concatenate the text representations of the task question "What is the capital city of the United Kingdom?" and the text representations of each row in the table obtained in step c) into a single string of text and input it into the compressed pre-trained language model. For example, for the first row (non-header row), the model input would be: "What are the features of Writer.AI? PIC.AI | 2 rows 5 columns | Image | Location: Generates images through natural language descriptions. Features: Has a good Discord community discussion environment, generates 4 images at once. | Google, Microsoft | Pre-A".

[0064] e) The pre-trained language model outputs a sequence of labels marked with [I, O], where an I label indicates that the text at the current position can be used as an answer to the question, and an O label indicates that the text at the current position cannot be used as an answer to the question. The number of labels obtained is the same as the length of the input text obtained in step c).

[0065] f) Extract the text corresponding to consecutive "I" predictions from the model, and use it as the answer to the question "What are the product features of Writer.AI?". If there is no "I" label in the model's predicted label sequence, it means that there is no answer to the question in the current row, that is, the information to be extracted is not in the current row of the table, and the prediction continues to the next row.

[0066] g) Starting from the first non-header row of the table, repeat steps c), d), e), and f) above one row at a time until the answer is found or all rows in the table have been traversed.

[0067] Finally, when iterating through the table with 3 rows, the model input was "Writer.AI | 5 rows 3 columns | Text | Positioning: AI creation tool | Features: 10x faster at creating SEO-optimized and plagiarism-free content for blogs, ads, emails, and websites." The answer to the question "What are the features of Writer.AI?" was "10x faster at creating SEO-optimized and plagiarism-free content for blogs, ads, emails, and websites."

[0068] In summary, this invention models cell information extraction as a reading comprehension problem and leverages the robustness and generalization ability of a pre-trained model to accurately locate cells in a table that can answer natural language questions, thereby extracting relevant information within the table. Furthermore, by utilizing a pruned / compressed pre-trained language model, real-time inference on computationally complex tasks can be achieved, improving the model's efficiency and making it easier to deploy on hardware devices.

[0069] Based on the same inventive concept, another preferred embodiment of the present invention also provides a table information extraction system based on a compressed pre-trained language model, corresponding to the table information extraction method based on a compressed pre-trained language model provided in the above embodiments. For example... Figure 3 As shown, this table information extraction system based on a compressed pre-trained language model includes three basic modules:

[0070] The question generation module is used to convert the target information fields to be extracted into natural language questions.

[0071] The row feature generation module is used to extract information from each non-header row of the target table, and to concatenate the cell content and position information of each row into row feature information.

[0072] The table information extraction module is used to input the row feature information and the natural language question into the compressed pre-trained language model based on the compressed pre-trained language model after neuron pruning and weight compression. The module classifies the text in the row feature information and outputs classification labels to represent whether the text is the answer to the natural language question. The module locates the text corresponding to the classification label marked as the answer and extracts it from the table as the extraction result.

[0073] Since the principle of the table information extraction system based on compressed pre-trained language model in this embodiment of the invention is similar to the table information extraction method based on compressed pre-trained language model in the above embodiment of the invention, the specific implementation forms of each module of the system in this embodiment can also refer to the specific implementation forms of the above method, and the repeated parts will not be described again.

[0074] Similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides an electronic device corresponding to the table information extraction method based on compressed pre-trained language model provided in the above embodiments, which includes a memory and a processor;

[0075] The memory is used to store computer programs;

[0076] The processor is configured to implement the table information extraction method based on the compressed pre-trained language model as described above when executing the computer program.

[0077] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0078] Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer-readable storage medium corresponding to the table information extraction method based on compressed pre-trained language model provided in the above embodiments. The storage medium stores a computer program, which, when executed by a processor, can realize the table information extraction method based on compressed pre-trained language model as described above.

[0079] Specifically, in the computer-readable storage medium of the two embodiments described above, the stored computer program is executed by a processor, which can perform the aforementioned steps S1 to S3.

[0080] It is understood that the aforementioned storage media may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Furthermore, the storage media may also be various media capable of storing program code, such as USB flash drives, external hard drives, magnetic disks, or optical discs.

[0081] It is understood that the processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0082] It should also be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. In the embodiments provided in this application, the division of steps or modules in the system and method is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple modules or steps may be combined or integrated together, and a module or step may also be split.

[0083] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A method for extracting tabular information based on a compressed pre-trained language model, characterized in that, include: S1. Convert the target information fields to be extracted into natural language questions; S2. Extract information from each non-header row of the target table, and concatenate the cell content and position information of each row into row feature information. S3. Based on the compressed pre-trained language model after neuron pruning and weight compression, the line feature information and the natural language question are input into the compressed pre-trained language model to classify the text in the line feature information, and output classification labels to represent whether the text is the answer to the natural language question; locate the text corresponding to the classification label marked as the answer, and extract it from the table as the extraction result; The compressed pre-trained language model is obtained by pruning neurons and compressing weights from the pre-trained language model. In the neuron pruning operation, based on the mutual information between neurons in each layer, the pre-trained language model is pruned layer by layer from the highest layer to the lowest layer, and a fixed number of neurons with the largest mutual information with the neurons retained in the previous layer are retained in each layer; in the weight compression operation, the model weight matrix of the pre-trained language model is compressed according to the neurons retained after each layer of pruning. In the neuron pruning operation, the Softmax layer of the pre-trained language model is taken as the highest layer. The neurons in the Softmax layer are kept unchanged. Then, a preset number of neurons are randomly selected from the next layer of the network to be retained, while the rest are pruned. The performance difference of the pre-trained language model before and after pruning is tested on the dataset. If the performance difference meets the preset requirements, the pruning of this layer is completed. Otherwise, the neuron selection and pruning are repeated. For each other layer in the pre-trained language model, a preset number of neurons with the largest mutual information with the neurons retained in the previous layer are selected, and the rest are pruned.

2. The method for extracting table information based on a compressed pre-trained language model as described in claim 1, characterized in that, The target table is pre-identified by recognizing the table's header row, total number of rows, total number of columns, and the number of columns in each row excluding empty cells.

3. The method for extracting table information based on a compressed pre-trained language model as described in claim 1, characterized in that, The position information of each row in the target table includes the row number of that row among all non-header rows in the target table, and the column number of the last non-empty cell in that row.

4. The method for extracting table information based on a compressed pre-trained language model as described in claim 1, characterized in that, The target information field is converted into a natural language question by calling a template library to fill in the template.

5. The method for extracting table information based on a compressed pre-trained language model as described in claim 1, characterized in that, The row feature information extracted from the target table is input into a compressed pre-trained language model row by row along with the natural language question for classification. When the model output contains a label corresponding to the answer, it is considered that the information to be extracted has been obtained, and the input of subsequent row feature information is stopped.

6. A table information extraction system based on a compressed pre-trained language model, characterized in that, include: The question generation module is used to convert the target information fields to be extracted into natural language questions. The row feature generation module is used to extract information from each non-header row of the target table, and to concatenate the cell content and position information of each row into row feature information. The table information extraction module is used to input the row feature information and the natural language question into the compressed pre-trained language model based on the compressed pre-trained language model after neuron pruning and weight compression. The module classifies the text in the row feature information and outputs classification labels to represent whether the text is the answer to the natural language question. The module locates the text corresponding to the classification label marked as the answer and extracts it from the table as the extraction result. The compressed pre-trained language model is obtained by pruning neurons and compressing weights from the pre-trained language model. In the neuron pruning operation, based on the mutual information between neurons in each layer, the pre-trained language model is pruned layer by layer from the highest layer to the lowest layer, and a fixed number of neurons with the largest mutual information with the neurons retained in the previous layer are retained in each layer; in the weight compression operation, the model weight matrix of the pre-trained language model is compressed according to the neurons retained after each layer of pruning. In the neuron pruning operation, the Softmax layer of the pre-trained language model is taken as the highest layer. The neurons in the Softmax layer are kept unchanged. Then, a preset number of neurons are randomly selected from the next layer of the network to be retained, while the rest are pruned. The performance difference of the pre-trained language model before and after pruning is tested on the dataset. If the performance difference meets the preset requirements, the pruning of this layer is completed. Otherwise, the neuron selection and pruning are repeated. For each other layer in the pre-trained language model, a preset number of neurons with the largest mutual information with the neurons retained in the previous layer are selected, and the rest are pruned.

7. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the table information extraction method based on a compressed pre-trained language model as described in any one of claims 1 to 5.

8. A computer electronic device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to, when executing the computer program, implement the table information extraction method based on a compressed pre-trained language model as described in any one of claims 1 to 5.

Images