Tabular data processing method and system for large model pre-training in the bidding field

Automatically judge the table data type through keyword retrieval and semantic distribution methods, and use text template library and OCR recognition technology to convert table data into text format, solving the problem of inefficiency of traditional methods when dealing with large-scale and diversified table data. Improves the accuracy and adaptability of data processing.

CN118410775BActive Publication Date: 2024-11-08国泰新点软件股份有限公司
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410868912.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-01
Publication Date
2024-11-08
Estimated Expiration
2044-07-01

AI Technical Summary

Technical Problem

Traditional tabular data processing methods are inefficient when facing a huge bidding data set, and it is difficult to deal with a large number of and diverse tabular data, making it difficult to guarantee the accuracy of processing results.

Method used

Keyword retrieval and semantic distribution methods are used to automatically judge table data types, and table data is converted into text formats suitable for large-scale model input through pre-constructed text template libraries and OCR recognition technology, including business-specific tables, numeric tables and text-type tables. processing flow.

Benefits of technology

It improves data processing efficiency and accuracy, can flexibly respond to table data in different forms and contents, reduces manual intervention, and enhances the application effect of large models in the bidding field.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118410775B_ABST
    Figure CN118410775B_ABST
Patent Text Reader

Abstract

The present application relates to the field of artificial intelligence technology, and in particular to a table data processing method and system for pre-training of large models in the field of bidding and tendering, the method comprising obtaining table data, judging whether it belongs to a business-specific table, and if not, judging whether the table data belongs to a digital table or a text table based on heuristic rules; if it is a business-specific table, selecting a corresponding text template in a pre-built text template library, extracting data from the table data and filling it into the text template to obtain a text description of the business-specific table; if it is a digital table, converting it into structured data and adding annotations to obtain a text description of the digital table; if it is a text table, using an OCR recognition method to recognize the text in the table data, organizing the recognized text into a text format to obtain a text description of the text table. The present application can cope with diverse table data, improve the processing efficiency of table data in the field of bidding and tendering, and improve the accuracy of table data processing results.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a table data processing method and system for pre-training large models in the bidding field. Background Art

[0002] In the field of bidding, the application of artificial intelligence technology is becoming more and more widespread, especially text generation models such as ChatGPT, which can provide powerful data analysis and decision support capabilities by providing various types of data for large model pre-training. However, the input of large models usually requires text data, while a large amount of information in the bidding field exists in the form of tables. Tabular data contains rich information such as supplier information, product quotations, technical indicators, etc. Therefore, it is necessary to pre-process the tabular data and convert it into text data suitable for large model input in order to give full play to the capabilities of the large model. By optimizing the processing method of tabular data, the rich information in the table can be retained and utilized to the maximum extent, and the application effect of the large model in the bidding field can be improved.

[0003] At present, traditional data processing methods mainly rely on manual screening and sorting of table data, which usually includes manually checking the table content, extracting key data and converting it into text format. The processing process involves detailed review and analysis of various types of data such as supplier information, product quotations, technical indicators, etc.

[0004] However, the above traditional data processing methods work well when processing smaller-scale data, but they are inefficient when facing large bidding data sets. In addition, traditional methods usually use fixed frameworks and rules, which are difficult to flexibly respond to changes in data form and content. Therefore, traditional data processing methods are relatively inefficient when facing tabular data in the bidding field, and it is difficult to cope with large amounts of diverse tabular data, which makes it difficult to guarantee the accuracy of tabular data processing results. Summary of the invention

[0005] This application provides a table data processing method and system for pre-training large models in the bidding field, which can handle large amounts of diverse table data, improve the processing efficiency of table data in the bidding field and the accuracy of table data processing results. This application provides the following technical solutions:

[0006] In a first aspect, the present application provides a table data processing method for pre-training a large model in the bidding field, the method comprising:

[0007] Acquire table data, and determine whether the table data belongs to a business-specific table by keyword retrieval; if it does not belong to the business-specific table, determine whether the table data belongs to a digital table or a text table based on a preset heuristic rule;

[0008] If the table data belongs to a business-specific table, a corresponding text template is selected from a pre-built text template library based on the specific type of the business-specific table to which the table data belongs, data is extracted from the table data and filled into the text template, and a text description of the business-specific table is obtained;

[0009] If the table data is a digital table, convert the table data into structured data, and add annotations to the structured data to obtain a text description of the digital table;

[0010] If the table data is a text-type table, detect and split the merged cells in the table data, use the OCR recognition method to recognize the text in the table data, organize the recognized text into a text format according to rows, columns and headers, and obtain a text description of the text-type table.

[0011] In a specific implementation scheme, the method of determining whether the table data belongs to a business-specific table by searching through keywords includes:

[0012] Build and dynamically update the business keyword library;

[0013] Expanding the business keyword library using a semantic distribution method;

[0014] Using the expanded business keyword library to search the table data, determine whether there is a keyword in the table data, and if so, tentatively determine it as a business-specific table;

[0015] The form data tentatively determined as a business-specific form is manually confirmed, and if confirmed to be correct, it is finally determined to be a business-specific form.

[0016] In a specific implementation scheme, the method of using semantic distribution to expand the business keyword library includes:

[0017] Collect a large number of texts related to the bidding field to form a text corpus, and pre-process the collected texts;

[0018] Construct a word co-occurrence matrix, where each element of the matrix represents a word and The number of co-occurrences in the same context window, where the context window is pre-defined, calculates the number of co-occurrences of each pair of words in the context window, and calculates the probability of each pair of words appearing as follows:

[0019] ;

[0020] ;

[0021] ;

[0022] The vocabulary is calculated by the following formula and of value:

[0023] ;

[0024] Calculate and of After the value is obtained, it is compared with the preset threshold. If it exceeds the threshold, the pair of words is identified as synonyms.

[0025] For each keyword in the business keyword library, the semantic distribution method is used to calculate and filter out its synonyms in the text corpus, and the synonyms are added to the business keyword library to obtain an expanded business keyword library.

[0026] In a specific implementation scheme, judging whether the table data belongs to a digital table or a text table based on a preset heuristic rule includes:

[0027] The heuristic rules are as follows:

[0028] 1) Tables with an average cell length of less than 10;

[0029] 2) Contains special characters More than 10 tables;

[0030] 3) Tables with a alphanumeric ratio of more than half;

[0031] 4) A table with 5 or more columns and no repeated elements in each row;

[0032] If the table data meets any heuristic rule, the table data is determined to be a digital table; otherwise, the table data is determined to be a text table.

[0033] In a specific possible implementation scheme, the selecting a corresponding text template from a pre-built text template library based on the specific type of the business-specific table to which the table data belongs, extracting data from the table data and filling the data into the text template includes:

[0034] In the pre-built text template library, each specific type of business-specific form corresponds to a text template;

[0035] Select the text template corresponding to the specific type of business-specific form to which the table data belongs in the text template library;

[0036] A Python-based text template engine is used to extract data from the table data, and the data is filled into corresponding positions in the text template one by one.

[0037] In a specific implementation scheme, converting the tabular data into structured data and adding annotations to the structured data includes:

[0038] Convert tabular data into structured data using Python’s Pandas library or R’s data frame;

[0039] Add annotations to the structured data to describe the meaning of each field.

[0040] In a specific implementation manner, before using the OCR recognition method to recognize the text in the table data, the method further includes:

[0041] The OCR recognition method is optimized.

[0042] In the second aspect, the present application provides a table data processing system for pre-training large models in the bidding field, which adopts the following technical solutions:

[0043] A table data processing system for pre-training large models in the bidding field, comprising:

[0044] A table type determination module is used to obtain table data and determine whether the table data belongs to a business-specific table by keyword retrieval. If it does not belong to the business-specific table, it determines whether the table data belongs to a digital table or a text table based on a preset heuristic rule;

[0045] A business-specific table processing module is used for, if the table data belongs to a business-specific table, selecting a corresponding text template from a pre-built text template library based on the specific type of the business-specific table to which the table data belongs, extracting data from the table data and filling it into the text template, so as to obtain a text description of the business-specific table;

[0046] A digital table processing module, used for converting the table data into structured data if the table data belongs to a digital table, and adding annotations to the structured data to obtain a text description of the digital table;

[0047] The text-type table processing module is used to detect and split the merged cells in the table data if the table data belongs to a text-type table, use the OCR recognition method to recognize the text in the table data, organize the recognized text into a text format according to rows, columns and headers, and obtain a text description of the text-type table.

[0048] In a third aspect, the present application provides an electronic device comprising a processor and a memory; the memory stores a program, and the program is loaded and executed by the processor to implement a tabular data processing method for pre-training of large models in the bidding field as described in the first aspect.

[0049] In a fourth aspect, the present application provides a computer-readable storage medium, in which a program is stored. When the program is executed by a processor, it is used to implement a tabular data processing method for pre-training of large models in the bidding field as described in the first aspect.

[0050] In summary, the beneficial effects of this application include at least:

[0051] 1) By using the business keyword library and semantic distribution method, the table data type is automatically determined, avoiding the traditional manual screening and classification process, greatly improving data processing efficiency. Especially when facing a large bidding data set, it can quickly and accurately process various types of table data.

[0052] 2) The method proposed in this application can process business-specific forms, digital forms and text forms, and each type has corresponding processing procedures and technical support. Whether it is filling in text templates, converting structured data, or splitting and OCR recognition of text forms, the integrity and accuracy of the data can be maintained.

[0053] 3) Compared with the traditional fixed processing framework and rules, the method of this application adopts heuristic rules and dynamically updated business keyword library, which can flexibly deal with tabular data of different forms and contents. This flexibility enables the method to be adjusted and optimized as business needs and data characteristics change, improving the adaptability and practicality of data processing.

[0054] Through automated processes, the diverse tabular data in bidding documents can be effectively processed, including business-specific tables, digital tables, and text tables. The key steps include using the business keyword library and semantic distribution method to determine the table type, selecting the corresponding text template or conversion method, and converting the tabular data into a text format suitable for large model pre-training through structured data conversion and OCR recognition and other technologies. This method optimizes traditional manual processing methods, improves data processing efficiency and accuracy, and effectively utilizes the advantages of large models in text data processing and decision support, thereby solving the problems of inefficiency and poor application effects in the background technology when processing large-scale and diversified tabular data.

[0055] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present application in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a flow chart of the table data processing method for pre-training of large models in the bidding and tendering field in the embodiment of the present application.

[0057] Figure 2 It is a flowchart of a method for determining whether table data belongs to a business-specific table by keyword retrieval in an embodiment of the present application.

[0058] Figure 3 It is an example diagram of converting a digital table into a corresponding text description in an embodiment of the present application.

[0059] Figure 4 It is an example diagram of converting a text table into a corresponding text description in an embodiment of the present application.

[0060] Figure 5 It is an overall schematic diagram of the tabular data processing method for pre-training of large models in the bidding and tendering field in an embodiment of the present application.

[0061] Figure 6 It is a structural block diagram of a table data processing system for pre-training large models in the bidding and tendering field in an embodiment of the present application.

[0062] Figure 7 It is a block diagram of an electronic device for processing tabular data for pre-training of large models in the bidding field in an embodiment of the present application. DETAILED DESCRIPTION

[0063] The specific implementation methods of the present application are further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present application but are not intended to limit the scope of the present application.

[0064] Optionally, the present application uses the table data processing method for pre-training of large models in the bidding field provided in various embodiments as an example of being used in an electronic device, where the electronic device is a terminal or a server. The terminal may be a computer, a tablet computer, etc. This embodiment does not limit the type of electronic device.

[0065] Reference Figure 1 , is a flow chart of a table data processing method for pre-training a large model in the bidding field provided by an embodiment of the present application, and the method includes at least the following steps:

[0066] Step S101, obtaining table data, and determining whether the table data belongs to a business-specific table by keyword search. If it does not belong to a business-specific table, determining whether the table data belongs to a digital table or a text table based on a preset heuristic rule.

[0067] Step S102: If the table data belongs to a business-specific table, select a corresponding text template from a pre-built text template library based on the specific type of the business-specific table to which the table data belongs, extract data from the table data and fill it into the text template to obtain a text description of the business-specific table.

[0068] Step S103: If the table data is a digital table, convert the table data into structured data, and add annotations to the structured data to obtain a text description of the digital table.

[0069] Step S104: If the table data is a text-type table, detect and split the merged cells in the table data, use the OCR recognition method to recognize the text in the table data, organize the recognized text into a text format according to rows, columns and headers, and obtain a text description of the text-type table.

[0070] In step S101, firstly, table data is obtained from the bidding document, and then the table data is determined by keyword search whether it belongs to a business-specific table. Figure 2 , is a flow chart of a method for determining whether table data belongs to a business-specific table by keyword retrieval in an embodiment of the present application, which at least includes the following steps:

[0071] Step S1011: Build and dynamically update a business keyword library.

[0072] During implementation, it is first necessary to construct a business keyword library for business-specific forms, include all keywords related to the business-specific forms, and at the same time, regularly update the business keyword library.

[0073] Step S1012: Expand the business keyword library using a semantic distribution method.

[0074] Specifically, we first collect a large number of texts related to the bidding field to form a text corpus, such as bidding documents, industry reports or standard documents, and preprocess the collected texts, including word segmentation, removal of stop words and punctuation marks, etc. Then we construct a word co-occurrence matrix, where each element of the matrix represents the word and The number of co-occurrences in the same context window, where the context window is pre-defined, such as the five words before and after, calculates the number of co-occurrences of each pair of words in the context window, and calculates the probability of each pair of words appearing as follows:

[0075] ;

[0076] ;

[0077] ;

[0078] The vocabulary is then calculated using the following formula and The point mutual information value (i.e. value):

[0079] ;

[0080] Calculate and of After the value is calculated, it is compared with the preset threshold. If it exceeds the threshold, the pair of words is identified as synonyms. For each keyword in the business keyword library, the above method is used to calculate and filter out its synonyms in the text corpus, and the synonyms are added to the business keyword library to obtain an expanded business keyword library.

[0081] Optionally, in another feasible embodiment, the word frequency weight can be introduced to calculate the weighted mutual information value based on the traditional point mutual information value: , the specific formula is as follows:

[0082] ;

[0083] in and Vocabulary and Calculate the word frequency of and After the weighted mutual information value of is calculated, it is compared with the preset threshold. If it exceeds the threshold, the pair of words is identified as synonyms. For each keyword in the business keyword library, the above method is used to calculate and filter out its synonyms in the text corpus, and the synonyms are added to the business keyword library to obtain an extended business keyword library. In the above formula, by introducing the word frequency weight, the weighted mutual information value formula not only considers their co-occurrence probability but also their frequency of occurrence in the text when calculating the association strength between two words. This enables the formula to more accurately reflect the importance and relevance of the two words in the corpus. For example, high-frequency words will have a greater impact on the results, thereby enhancing the weight of common business keywords and ensuring their prominent position in the keyword library. At the same time, the word frequency weight in the formula ensures that those important words that often appear in the business field are given priority. This helps to build a business keyword library with wider coverage and more accurate. By introducing the word frequency weight, the extended keyword library can better represent common terms in actual business scenarios and improve the effect and accuracy of keyword retrieval. In addition, the traditional point mutual information value may produce a higher value when processing low-frequency words, causing noise words to be mistakenly identified as important business keywords. By introducing word frequency weights, the impact of low-frequency words can be reduced, making the final expanded keyword library purer, reducing the interference of noise words, and improving the reliability of the keyword library.

[0084] In addition, as a preferred embodiment, after the keyword library is expanded using the calculation methods of the point mutual information value and the weighted mutual information value, the union of the two expanded keyword libraries is taken as the final expanded keyword library. The point mutual information value and the weighted mutual information value each have their own advantages. The point mutual information value is good at capturing rare but important vocabulary pairs, while the weighted mutual information value pays more attention to the weight of high-frequency vocabulary. Taking the union of the results of the two can combine the advantages of the two methods to ensure that both rare but important vocabulary and frequently occurring important vocabulary are included, thereby improving the comprehensiveness and coverage of the keyword library. In addition, relying on a single algorithm or method may bring certain risks, such as algorithm limitations or poor performance in specific scenarios. By combining the results of the two, the dependence on a single algorithm can be reduced, providing a more stable and reliable keyword library, and improving the stability and risk resistance of the overall system.

[0085] Step S1013: Use the extended business keyword library to search the table data to determine whether there are keywords in the table data. If so, it is temporarily determined to be a business-specific table.

[0086] Step S1014: manually confirm the table data tentatively determined as a business-specific table, and if confirmed to be correct, finally determine it as a business-specific table.

[0087] In step S1014, while manually confirming that the table data is a business-specific table, the specific type of the business-specific table is also confirmed for subsequent operations.

[0088] In implementation, if it is finally determined that the table data does not belong to a business-specific table, the obtained table data is classified according to a preset heuristic rule. Specifically, the heuristic rule includes but is not limited to:

[0089] 1) Tables where the average length of each cell is less than 10.

[0090] 2) Contains special characters More than 10 tables.

[0091] 3) Tables where the ratio of letters to numbers exceeds half.

[0092] 4) A table with 5 or more columns and no repeated elements in each row.

[0093] If the table data meets any of the above heuristic rules, the table data is judged to be a digital table, otherwise the table data is a text table. Optionally, the above heuristic rules in the embodiments of the present application are all based on the experience summary of experts in the bidding field. In actual application, the heuristic rules are regularly adjusted and optimized, and mathematical models or algorithms can also be used to derive the heuristic rules. This application does not limit the setting method of the heuristic rules.

[0094] For ease of understanding, this application provides examples of various types of forms. Business-specific forms include but are not limited to the appendix to the bid evaluation method, model comparison table, project schedule, budget table, etc. Digital forms include but are not limited to quotation tables, technical indicator tables, inventory lists, financial statements, statistical analysis tables, etc. Text forms include but are not limited to supplier information tables, product description tables, contract terms tables, meeting minutes tables, etc.

[0095] In step S102, if the table data belongs to a business-specific table, the specific type of the business-specific table to which the table data belongs has been known through previous manual confirmation. In the pre-built text template library, each specific type of business-specific table corresponds to a text template, and each text template includes a placeholder, which is used to represent the data to be filled. The placeholder is generally represented by a special mark or a string, usually {} or ____. For example, the text template of the model comparison table is as follows:

[0096] Purchasing a ____ product, its specification requirement is ____, product model is ____, appearance requirement is ____, and price is ___.

[0097] In the text template library, select the text template corresponding to the specific type of business-specific table to which the table data belongs, use the Python-based text template engine to extract the data in the table data, and fill the data into the corresponding positions in the text template one by one, and finally form a text description of the table data.

[0098] Optionally, the data mapping operation is completed using a Python-based text template engine in this application, and can also be implemented using natural language processing technology or other programming languages. This application does not limit the specific data extraction and filling methods.

[0099] In step S103, first use Python's Pandas library or the data frame in the R language to convert the tabular data into structured data, such as HTML tags or Markdown format. Optionally, other conversion methods can also be used to convert the tabular data into structured data, and this application does not limit the specific conversion method. Then add comments to the structured data to describe the meaning of each field. By converting the digital table into a structured data format and annotating it on this basis, the data can be effectively prepared for pre-training of large models. Annotations can help the model understand the semantics and structure of the data, and improve the accuracy and efficiency of the model when processing and using tabular data. Reference Figure 3 , which is an example diagram of converting a digital table into a corresponding text description in an embodiment of the present application.

[0100] In step S104, first, the merged cells in the text-type table are detected and split to ensure that each line of text is logically causally related, and the table is restored to its original state. This application uses the OpenPyXL programming library to read the table file and detect whether there are merged cells in the table. If there are merged cells, the merged cells are split back into each original cell. For example, if a cell spans three rows, the same content will be filled in the corresponding positions of the three rows after splitting. For example, the content of a cell is "Supplier A", and the cell merges three rows. Then, after splitting, the corresponding columns of the three rows should be filled with "Supplier A". This step is intended to eliminate the confusion and inconsistency of data structure that may be caused by merging cells and ensure the integrity and accuracy of the data. The processed text-type table is then recognized using an optimized OCR (optical character recognition) method. The optimization method includes adjusting image preprocessing parameters, using a high-precision OCR engine, etc., to improve the accuracy and stability of identifying table content. Finally, the text-type table data after OCR recognition is reorganized into a specified text format. The specific format requires that each table row is identified according to the column header, and each column of data is separated by a tab character (\t). For example, the format of each row is "column 1 header: table element\tcolumn 2 header: table element". Figure 4, is an example diagram of converting a text table into a corresponding text description in the embodiment of the present application. Through the above steps, text table data can be effectively processed to make it suitable for large model pre-training, thereby improving data utilization efficiency and model application effect.

[0101] In summary, refer to Figure 5 , is an overall schematic diagram of a table data processing method for pre-training a large model in the bidding field in an embodiment of the present application. The present application proposes a method for processing table data for pre-training a large model in the bidding field. The method effectively processes a variety of table data in bidding documents through an automated process, including business-specific tables, digital tables, and text tables. The key steps include using a business keyword library and a semantic distribution method to determine the type of table, selecting a corresponding text template or conversion method, and converting the table data into a text format suitable for pre-training of a large model through techniques such as structured data conversion and OCR recognition. This method optimizes the traditional manual processing method, improves data processing efficiency and accuracy, and effectively utilizes the advantages of large models in text data processing and decision support, thereby solving the problems of low efficiency and poor application effect when processing large-scale and diversified table data in the background technology. First, by constructing and dynamically updating a business keyword library and combining the semantic distribution method to expand the keyword library, business-specific tables can be automatically identified. This step enables the system to quickly and accurately determine the type of table data, thereby determining the subsequent processing flow.

[0102] Specifically, for business-specific tables, the system selects the corresponding text template based on the pre-built text template library, fills the table data into the template, and generates a text description of the business-specific table. This method ensures the consistency and standardization of the text, while saving a lot of time and cost for manual filling. For digital tables, Python's Pandas library or other data frameworks are used to convert the table data into a structured data format. On this basis, annotations are added to describe the meaning of each field, which improves the understanding and utilization efficiency of the data, thereby providing preparation data for large model pre-training. For text-based tables, the system first detects and splits merged cells to ensure the logical relevance and integrity of the data, and then uses the optimized OCR method to recognize the text and reorganize it into the specified text format. This method not only improves the accuracy and stability of recognition, but also effectively handles the complex structure in the table. When judging non-business-specific tables, heuristic rules are used for classification, such as based on table length, number of special characters, alphanumeric ratio, etc. These rules are based on the experience summary of domain experts and can be adjusted and optimized regularly according to actual needs to cope with changes in data form and content. Through the above method, this application can efficiently process large amounts of diverse tabular data, improving the efficiency and accuracy of processing tabular data in the bidding field. The automated process reduces manual intervention, reduces the processing burden, and enhances the application effect of large models in data analysis and decision support through standardized output results.

[0103] Figure 6 This is a structural block diagram of a table data processing system for pre-training a large model in the bidding field provided by an embodiment of the present application. The device includes at least the following modules:

[0104] The table type judgment module is used to obtain table data and determine whether the table data belongs to a business-specific table through keyword retrieval. If it does not belong to a business-specific table, it determines whether the table data belongs to a digital table or a text table based on preset heuristic rules.

[0105] The business-specific table processing module is used to select the corresponding text template from the pre-built text template library based on the specific type of business-specific table to which the table data belongs if the table data belongs to a business-specific table, extract data from the table data and fill it into the text template to obtain a text description of the business-specific table.

[0106] The digital table processing module is used to convert the table data into structured data if the table data belongs to a digital table, and add annotations to the structured data to obtain a text description of the digital table.

[0107] The text table processing module is used to detect and split the merged cells in the table data if the table data belongs to a text table, use the OCR recognition method to recognize the text in the table data, organize the recognized text into a text format according to rows, columns and headers, and obtain a text description of the text table.

[0108] For relevant details, refer to the above method embodiment.

[0109] Figure 7 4 is a block diagram of an electronic device provided by an embodiment of the present application. The device at least includes a processor 401 and a memory 402.

[0110] The processor 401 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 401 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 401 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 401 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 401 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0111] The memory 402 may include one or more computer-readable storage media, which may be non-transitory. The memory 402 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 402 is used to store at least one instruction, which is used to be executed by the processor 401 to implement the table data processing method for pre-training of large models in the bidding field provided in the method embodiment of the present application.

[0112] In some embodiments, the electronic device may further optionally include: a peripheral device interface and at least one peripheral device. The processor 401, the memory 402 and the peripheral device interface may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface via a bus, a signal line or a circuit board. Schematically, the peripheral devices include but are not limited to: a radio frequency circuit, a touch display screen, an audio circuit, and a power supply.

[0113] Of course, the electronic device may also include fewer or more components, which is not limited in this embodiment.

[0114] Optionally, the present application also provides a computer-readable storage medium, in which a program is stored, and the program is loaded and executed by a processor to implement the tabular data processing method for pre-training of large models in the bidding field of the above-mentioned method embodiment.

[0115] Optionally, the present application also provides a computer product, which includes a computer-readable storage medium, in which a program is stored, and the program is loaded and executed by a processor to implement the tabular data processing method for pre-training of large models in the bidding field of the above-mentioned method embodiment.

[0116] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0117] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A table data processing method for pre-training large models in the bidding field, characterized in that: The method comprises: Acquire table data, determine whether the table data belongs to a business-specific table by keyword retrieval, and if it does not belong to the business-specific table, determine whether the table data belongs to a digital table or a text table based on a preset heuristic rule; if the table data belongs to a business-specific table, select a corresponding text template from a pre-built text template library based on the specific type of the business-specific table to which the table data belongs, extract data from the table data and fill it into the text template, and obtain a text description of the business-specific table; The method for determining whether the table data belongs to a business-specific table by keyword retrieval includes: constructing and dynamically updating a business keyword library; expanding the business keyword library using a semantic distribution method; searching the table data using the expanded business keyword library to determine whether there are keywords in the table data, and if so, tentatively determining it as a business-specific table; manually confirming the table data tentatively determined as a business-specific table, and finally determining it as a business-specific table if the confirmation is correct; The method of using semantic distribution to expand the business keyword library includes: collecting a large number of texts related to the bidding field to form a text corpus, preprocessing the collected texts; constructing a vocabulary co-occurrence matrix, each element of the matrix represents a vocabulary and The number of co-occurrences in the same context window, where the context window is pre-defined, calculates the number of co-occurrences of each pair of words in the context window, and calculates the probability of each pair of words appearing as follows: The vocabulary is calculated by the following formula and of value: Calculate and of After the value is calculated, it is compared with the preset threshold. If it exceeds the threshold, the pair of words is identified as synonyms. For each keyword in the business keyword library, the semantic distribution method is used to calculate and filter out its synonyms in the text corpus, and the synonyms are added to the business keyword library to obtain an extended business keyword library. If the table data is a digital table, convert the table data into structured data, and add annotations to the structured data to obtain a text description of the digital table; If the table data is a text-type table, detect and split the merged cells in the table data, use the OCR recognition method to recognize the text in the table data, organize the recognized text into a text format according to rows, columns and headers, and obtain a text description of the text-type table.

2. The table data processing method for large model pre-training in the bidding field according to claim 1 is characterized in that: The method of using semantic distribution to expand the business keyword library also includes: Use the weighted mutual information value calculation method to expand the keyword library and introduce word frequency weights to calculate the weighted mutual information value , the specific formula is as follows: in and Vocabulary and Calculate the word frequency of and After calculating the weighted mutual information value of the business keyword library, compare it with the preset threshold value. If it exceeds the threshold value, the pair of words is identified as synonyms. For each keyword in the business keyword library, use the above method to calculate and filter out its synonyms in the text corpus, add the synonyms to the business keyword library, and obtain an extended business keyword library. After expanding the keyword library using the point mutual information value and weighted mutual information value calculation methods respectively, the union of the two expanded keyword libraries is taken as the final expanded keyword library.

3. The table data processing method for large model pre-training in the bidding field according to claim 1 is characterized in that: The determining, based on a preset heuristic rule, whether the table data is a digital table or a text table includes: The heuristic rules are as follows: 1) Tables with an average cell length of less than 10; 2) Tables containing more than 10 special characters [#,$,%,^,&,*@]; 3) Tables with a alphanumeric ratio of more than half; 4) A table with 5 or more columns and no repeated elements in each row; If the table data meets any heuristic rule, the table data is determined to be a digital table; otherwise, the table data is determined to be a text table.

4. The table data processing method for pre-training large models in the bidding field according to claim 1 is characterized in that: The selecting a corresponding text template from a pre-built text template library based on the specific type of the business-specific table to which the table data belongs, extracting data from the table data and filling the data into the text template comprises: In the pre-built text template library, each specific type of business-specific form corresponds to a text template; Select the text template corresponding to the specific type of business-specific form to which the table data belongs in the text template library; A Python-based text template engine is used to extract data from the table data, and the data is filled into corresponding positions in the text template one by one.

5. The table data processing method for pre-training large models in the bidding field according to claim 1 is characterized in that: The converting the tabular data into structured data and adding annotations to the structured data includes: Convert tabular data into structured data using Python’s Pandas library or R’s data frame; Add annotations to the structured data to describe the meaning of each field.

6. The table data processing method for large model pre-training in the bidding field according to claim 1 is characterized in that: Before using the OCR recognition method to recognize the text in the table data, the method further includes: The OCR recognition method is optimized.

7. A table data processing system for pre-training large models in the bidding field, characterized in that: include: A table type determination module is used to obtain table data and determine whether the table data belongs to a business-specific table by keyword retrieval. If it does not belong to the business-specific table, it determines whether the table data belongs to a digital table or a text table based on a preset heuristic rule; The method for determining whether the table data belongs to a business-specific table by keyword retrieval includes: constructing and dynamically updating a business keyword library; expanding the business keyword library using a semantic distribution method; searching the table data using the expanded business keyword library to determine whether there are keywords in the table data, and if so, tentatively determining it as a business-specific table; manually confirming the table data tentatively determined as a business-specific table, and finally determining it as a business-specific table if the confirmation is correct; The method of using semantic distribution to expand the business keyword library includes: collecting a large number of texts related to the bidding field to form a text corpus, preprocessing the collected texts; constructing a vocabulary co-occurrence matrix, each element of the matrix represents a vocabulary and The number of co-occurrences in the same context window, where the context window is pre-defined, calculates the number of co-occurrences of each pair of words in the context window, and calculates the probability of each pair of words appearing as follows: The vocabulary is calculated by the following formula and of value: Calculate and of After the value is calculated, it is compared with the preset threshold. If it exceeds the threshold, the pair of words is identified as synonyms. For each keyword in the business keyword library, the semantic distribution method is used to calculate and filter out its synonyms in the text corpus, and the synonyms are added to the business keyword library to obtain an extended business keyword library. A business-specific table processing module is used for, if the table data belongs to a business-specific table, selecting a corresponding text template from a pre-built text template library based on the specific type of the business-specific table to which the table data belongs, extracting data from the table data and filling it into the text template, so as to obtain a text description of the business-specific table; A digital table processing module, used for converting the table data into structured data if the table data belongs to a digital table, and adding annotations to the structured data to obtain a text description of the digital table; The text-type table processing module is used to detect and split the merged cells in the table data if the table data belongs to a text-type table, use the OCR recognition method to recognize the text in the table data, organize the recognized text into a text format according to rows, columns and headers, and obtain a text description of the text-type table.

8. An electronic device, characterized in that: The device includes a processor and a memory; a program is stored in the memory, and the program is loaded and executed by the processor to implement a tabular data processing method for pre-training of large models in the bidding field as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The storage medium stores a program, which, when executed by a processor, is used to implement a tabular data processing method for pre-training a large model in the bidding field as described in any one of claims 1 to 6.