Data reconditioning method and apparatus for retrieval enhancement generation in field of integrated traffic planning

Abstract

The invention provides a data reorganizing method and device for retrieval enhancement generation in the field of comprehensive traffic planning, in the data reorganizing method, statistical yearbook data in a PDF format can be subjected to format conversion to obtain statistical yearbook data in an Excel table form, the statistical yearbook data in the Excel table form is standard in format, and the statistical yearbook data in the Excel table form can be stored in the data reorganizing method. According to the method, the statistical yearbook data in the Excel table form can be conveniently and subsequently subjected to automatic multi-dimensional row and column hierarchical dimension reduction, then the statistical yearbook data in the one-dimensional Excel table form which is more convenient to understand and good in readability is obtained, then the statistical yearbook data in the one-dimensional Excel table form is subjected to information optimization processing, the statistical yearbook data in the target one-dimensional Excel table form is obtained, and finally, the statistical yearbook data in the target one-dimensional Excel table form is obtained. In addition, the index knowledge graph is constructed according to the statistical yearbook data in the target one-dimensional Excel table form, modeling of the index entities in the statistical yearbook data is achieved through the index knowledge graph, the relation between the index entities is extracted, and the effect of subsequent knowledge retrieval can be enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of data preparation, and in particular to a data preparation method and device for retrieval enhancement generation in the field of comprehensive transportation planning. Background Art

[0002] At present, in the process of data preparation, the original data is generally only simply preprocessed, such as data cleaning and removal of stop words, and then the preprocessed data is saved to the database so that subsequent users can enter keywords and perform knowledge retrieval in the database based on the search terms.

[0003] For the statistical data in the field of comprehensive transportation planning, the data sources are wide and the data structure is diversified, that is, it has the characteristics of multi-source heterogeneity. If the above data preparation process is still used, the final pre-processed data is not standardized enough and the correlation between data is poor. The data results obtained by subsequent retrieval based on the search terms are inaccurate and the retrieval efficiency is low.

[0004] In summary, how to improve the effect of data preparation in the field of comprehensive transportation planning in order to improve the performance of data retrieval has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a data preparation method and device for retrieval enhancement generation in the field of comprehensive transportation planning, so as to alleviate the technical problem that the existing data preparation methods in the field of comprehensive transportation planning are poor in effect, which leads to poor data retrieval performance.

[0006] In a first aspect, an embodiment of the present invention provides a data preparation method for retrieval enhancement generation in the field of comprehensive transportation planning, comprising:

[0007] Obtain statistical yearbook data in the field of comprehensive transportation planning in PDF format, and convert the statistical yearbook data in PDF format to obtain the statistical yearbook data in Excel table format;

[0008] Performing multi-dimensional row and column level dimensionality reduction on the statistical yearbook data in the Excel table form to obtain the statistical yearbook data in the one-dimensional Excel table form;

[0009] Performing information optimization processing on the statistical yearbook data in the one-dimensional Excel table format to obtain the statistical yearbook data in the target one-dimensional Excel table format;

[0010] Entity-relationship entity triples are extracted from the target statistical yearbook data in the form of a one-dimensional Excel table, and an indicator knowledge graph of the statistical yearbook data in the PDF format is constructed based on the extracted entity-relationship entity triples, so as to perform statistical data retrieval based on the indicator knowledge graph.

[0011] Furthermore, the statistical yearbook data in PDF format is converted into a format, including:

[0012] The statistical yearbook data in PDF format is converted into an Excel table using OCR software to obtain the original statistical yearbook data in Excel table form;

[0013] Manual processing is performed on the original statistical yearbook data in the form of Excel tables to obtain the statistical yearbook data in the form of Excel tables, wherein the manual processing includes: yearbook table classification and renaming, multi-worksheet splitting and continuation table merging, multi-level indicator adjustment, and time and space information specification.

[0014] Furthermore, multi-dimensional row and column level dimensionality reduction is performed on the statistical yearbook data in the Excel table format, including:

[0015] Identify the format and information of the statistical yearbook data in Excel table form;

[0016] The hierarchical relationship of the row and column headers is integrated and rewritten according to the recognized format and information to obtain the statistical yearbook data in the one-dimensional Excel table form.

[0017] Further, identifying the format and information of the statistical yearbook data in Excel table form includes:

[0018] Performing semantic understanding on the row and column headers of the statistical yearbook data in the Excel table format to obtain basic information, row and column header information, and table type information of the statistical yearbook data in the Excel table format;

[0019] Identify the time field and space field of the statistical yearbook data in the Excel table format to obtain the yearbook time and space information of the statistical yearbook data in the Excel table format;

[0020] Identifying subheadings in the statistical yearbook data in the Excel table format and hierarchical relationships between the subheadings, wherein the subheadings are bold content in a data area in the statistical yearbook data in the Excel table format;

[0021] Identifying outliers in the statistical yearbook data in the Excel table format and processing the outliers;

[0022] The basic information, the row and column header information, the table type information, the yearbook time, the space information, the subheadings and the hierarchical relationship of the subheadings are used as the format and information of the statistical yearbook data in the Excel table form.

[0023] Furthermore, semantic understanding is performed on the row and column headers of the statistical yearbook data in the Excel table format, including:

[0024] Extracting and locating the title, unit and annotation information of the statistical yearbook data in the Excel table format, and obtaining the title, unit, annotation information of the statistical yearbook data in the Excel table format and the position of the annotation information in the statistical yearbook data in the Excel table format;

[0025] Determine the chapter number and section number corresponding to the statistical yearbook data in Excel table format;

[0026] Determine the row header range and column header range of the statistical yearbook data in the Excel table format;

[0027] Determine the table type of the statistical yearbook data in Excel table format, wherein the table type includes at least one of the following: a single-row header, a multi-row header, a single-column header, and a multi-column header.

[0028] Furthermore, the time field and the space field of the statistical yearbook data in the Excel table format are identified, including:

[0029] Determine the yearbook time of the statistical yearbook data in the Excel table format;

[0030] Determine the spatial fields in the statistical yearbook data in the Excel table format and the position of each of the spatial fields in the statistical yearbook data in the Excel table format.

[0031] Furthermore, the row and column header hierarchical relationship is integrated and rewritten according to the recognized format and information, including:

[0032] Based on the table type, the row header range, and the column header range, the merged cells are split to obtain a plurality of empty cells, and the hierarchical contents of the corresponding merged cells are filled in the empty cells;

[0033] Use the level separator to concatenate the row-level content, and delete duplicate content from the concatenated row-level content to obtain a row concatenation value;

[0034] Use the level separator to splice the column-level content, and delete duplicate content from the spliced ​​column-level content to obtain the column splicing value;

[0035] A one-dimensional Excel table with column names is constructed based on the title, the unit, the annotation information, the chapter number, the section number, and the yearbook time, and based on the spatial fields and the positions of the spatial fields in the statistical yearbook data in the Excel table form, the values ​​of the data areas, the row splicing values, and the column splicing values ​​in the statistical yearbook data in the Excel table form are filled into the corresponding column names to obtain the statistical yearbook data in the one-dimensional Excel table form.

[0036] Further, the statistical yearbook data in the one-dimensional Excel table form is subjected to information optimization processing, including:

[0037] The data time and unit information are determined based on the statistical yearbook data in the one-dimensional Excel table format.

[0038] Furthermore, the method further comprises:

[0039] Obtaining natural language input by a user for statistical data retrieval;

[0040] Using a natural language big model to convert the natural language used for statistical data retrieval into a graph query statement;

[0041] Query the indicator knowledge graph according to the graph query statement to obtain a query result;

[0042] The natural language big model is used to perform natural language conversion on the query result to obtain a natural language query result.

[0043] In a second aspect, an embodiment of the present invention further provides a data preparation device for retrieval enhancement generation in the field of comprehensive transportation planning, comprising:

[0044] A format conversion unit, used for obtaining statistical yearbook data in PDF format in the field of comprehensive transportation planning, and performing format conversion on the statistical yearbook data in PDF format to obtain statistical yearbook data in Excel table form;

[0045] A multi-dimensional row and column level dimension reduction unit is used to perform multi-dimensional row and column level dimension reduction on the statistical yearbook data in the Excel table form to obtain the statistical yearbook data in the one-dimensional Excel table form;

[0046] An information optimization processing unit, used for performing information optimization processing on the statistical yearbook data in the one-dimensional Excel table format to obtain the statistical yearbook data in the target one-dimensional Excel table format;

[0047] The extraction and construction unit extracts entity relationship entity triples from the target statistical yearbook data in the form of a one-dimensional Excel table, and constructs an indicator knowledge graph of the statistical yearbook data in the PDF format based on the extracted entity relationship entity triples, so as to perform statistical data retrieval based on the indicator knowledge graph.

[0048] In an embodiment of the present invention, a data preparation method for retrieval enhanced generation in the field of comprehensive transportation planning is provided, comprising: obtaining statistical yearbook data in PDF format in the field of comprehensive transportation planning, and converting the format of the statistical yearbook data in PDF format to obtain statistical yearbook data in Excel table format; performing multi-dimensional row and column level dimensionality reduction on the statistical yearbook data in Excel table format to obtain statistical yearbook data in one-dimensional Excel table format; performing information optimization processing on the statistical yearbook data in one-dimensional Excel table format to obtain statistical yearbook data in target one-dimensional Excel table format; extracting entity relationship entity triples in the target one-dimensional Excel table format of the statistical yearbook data, and constructing an indicator knowledge graph of the statistical yearbook data in PDF format according to the extracted entity relationship entity triples, so as to perform statistical data retrieval based on the indicator knowledge graph. From the above description, it can be seen that in the data preparation method for retrieval enhanced generation in the field of comprehensive transportation planning of the present invention, the statistical yearbook data in PDF format can be format converted to obtain statistical yearbook data in Excel table form. The statistical yearbook data in Excel table form has a standardized format, which is convenient for the subsequent automatic multi-dimensional row and column level dimensionality reduction of the statistical yearbook data in Excel table form, so as to obtain statistical yearbook data in one-dimensional Excel table form that is easier to understand and has good readability. The statistical yearbook data in one-dimensional Excel table form is then subjected to information optimization processing to obtain the target statistical yearbook data in one-dimensional Excel table form. Finally, an indicator knowledge graph is constructed based on the target statistical yearbook data in one-dimensional Excel table form. The indicator knowledge graph realizes the modeling of indicator entities in the statistical yearbook data, extracts the relationship between indicator entities, and can enhance the effect of subsequent knowledge retrieval. That is, the data preparation method of the present invention improves the effect of data preparation in the field of comprehensive transportation planning, thereby improving the performance of subsequent data retrieval, and alleviating the poor effect of the existing data preparation method in the field of comprehensive transportation planning, thereby leading to the technical problem of poor data retrieval performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0050] Figure 1 A flowchart of a data preparation method for retrieval enhancement generation in the field of comprehensive transportation planning provided by an embodiment of the present invention;

[0051] Figure 2 A schematic diagram of statistical yearbook data in PDF format provided by an embodiment of the present invention;

[0052] Figure 3 A schematic diagram of statistical yearbook data in Excel table format provided by an embodiment of the present invention;

[0053] Figure 4 A schematic diagram of the target one-dimensional Excel table statistical yearbook data provided by an embodiment of the present invention;

[0054] Figure 5 A schematic diagram of nine types of templates provided in an embodiment of the present invention;

[0055] Figure 6 A schematic diagram of a data preparation device for retrieval enhancement generation in the field of comprehensive transportation planning provided by an embodiment of the present invention;

[0056] Figure 7 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] Traditional data preparation methods in the field of integrated transportation planning are ineffective, which in turn leads to poor data retrieval performance.

[0059] Based on this, in the data preparation method for retrieval enhanced generation in the field of comprehensive transportation planning of the present invention, the statistical yearbook data in PDF format can be format converted to obtain the statistical yearbook data in Excel table form. The statistical yearbook data in Excel table form has a standardized format, which is convenient for the subsequent automatic multi-dimensional row and column level dimensionality reduction of the statistical yearbook data in Excel table form, thereby obtaining the statistical yearbook data in one-dimensional Excel table form that is easier to understand and has good readability. The statistical yearbook data in one-dimensional Excel table form is then subjected to information optimization processing to obtain the target one-dimensional Excel table form of the statistical yearbook data. Finally, an indicator knowledge graph is constructed based on the target one-dimensional Excel table form of the statistical yearbook data. The indicator knowledge graph realizes the modeling of indicator entities in the statistical yearbook data, extracts the relationship between indicator entities, and can enhance the effect of subsequent knowledge retrieval. That is, the data preparation method of the present invention improves the effect of data preparation in the field of comprehensive transportation planning, thereby improving the performance of subsequent data retrieval.

[0060] To facilitate understanding of this embodiment, a data preparation method for retrieval enhancement generation in the field of comprehensive transportation planning disclosed in an embodiment of the present invention is first introduced in detail.

[0061] Embodiment 1:

[0062] According to an embodiment of the present invention, an embodiment of a data preparation method for retrieving enhanced generation in the field of integrated transportation planning is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0063] Figure 1 is a flow chart of a data preparation method for retrieving enhanced generation in the field of comprehensive transportation planning according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0064] Step S102, obtaining statistical yearbook data in the field of comprehensive transportation planning in PDF format, and converting the statistical yearbook data in PDF format to obtain the statistical yearbook data in Excel table format;

[0065] like Figure 2 As shown in the figure, the statistical yearbook data in PDF format is obtained ( Figure 2Only part of the content is shown in the figure), first convert the statistical yearbook data in PDF format into statistical yearbook data in Excel table format, the data format of the statistical yearbook data in Excel table format is standardized and unified, and the subsequent algorithm can automatically perform multi-dimensional row and column level dimensionality reduction. Figure 3 As shown, Figure 2 The statistical yearbook data in PDF format corresponds to the statistical yearbook data in Excel table format.

[0066] Step S104, performing multi-dimensional row and column level dimensionality reduction on the statistical yearbook data in the Excel table format to obtain the statistical yearbook data in the one-dimensional Excel table format;

[0067] Specifically, Figure 3 As shown, in the statistical yearbook data in the form of an Excel table, the row header is a multidimensional row header (or a multi-row header, or a multi-dimensional row hierarchy (each row is a hierarchy)), and the dimensions included include: the primary industry, the entire city and the municipal districts below the primary industry hierarchy, the secondary industry, the mining industry and the manufacturing industry below the secondary industry hierarchy, etc., the entire city and the municipal districts below the mining industry and the manufacturing industry hierarchy, respectively. The column header is also a multidimensional column header (or a multi-column header, or a multi-dimensional column hierarchy (each column is a hierarchy)), and the dimensions included include: Hebei Province, Shijiazhuang City, Tangshan City, Qinhuangdao City, etc. below the Hebei Province hierarchy. The statistical yearbook data in the form of this Excel table has poor readability and is not easy to understand. The computer cannot interpret the semantics it represents. For example, it is impossible to obtain the information that the number of people in the primary industry of Beijing is 3,585. Therefore, it is necessary to perform multi-dimensional row and column hierarchy dimensionality reduction to obtain the statistical yearbook data in the form of a one-dimensional Excel table. The form of the statistical yearbook data in the form of a one-dimensional Excel table is as follows: Figure 4 As shown, there is only one row of column names (so it is called a one-dimensional Excel table format, there is no second dimension), and each column name in the row of column names can be an indicator entity in the statistical yearbook data.

[0068] Step S106, performing information optimization processing on the statistical yearbook data in the one-dimensional Excel table format to obtain the statistical yearbook data in the target one-dimensional Excel table format;

[0069] Specifically, the process from step S102 to step S106 completes the preparation and conversion of unstructured data (i.e., statistical yearbook data in PDF format) to semi-structured data (i.e., statistical yearbook data in target one-dimensional Excel table format, CSV). Figure 4 What is shown in the figure is the statistical yearbook data in the target one-dimensional Excel table format, which is easy to read and more standardized.

[0070] Step S108, extracting entity-relationship entity triples from the target one-dimensional Excel table-form statistical yearbook data, and constructing an indicator knowledge graph of the statistical yearbook data in PDF format based on the extracted entity-relationship entity triples, so as to perform statistical data retrieval based on the indicator knowledge graph.

[0071] Specifically, after obtaining the statistical yearbook data (CSV file) in the form of a target one-dimensional Excel table, the "indicator name" can be used as an entity, and the Neo4j Data Importer tool can be used to extract the "entity-relationship-entity" triples in the statistical yearbook data in the form of a target one-dimensional Excel table, thereby forming an indicator knowledge graph. For example, the extracted entity-relationship-entity triples are:

[0072] Entity: Primary industry: Citywide;

[0073] Relation 1: Region;

[0074] Entity: Beijing;

[0075] Relation 2: Yearbook time;

[0076] Entity: 2020;

[0077] Relation 3: indicator value;

[0078] Entities: 3585;

[0079] Relation 4: Chapter name;

[0080] Entity: Statistics of cities at prefecture level and above;

[0081] Relation 5: Yearbook name;

[0082] Entity: China City Statistical Yearbook.

[0083] By connecting them in this way, an indicator knowledge graph can be formed, in which entities serve as nodes in the graph and relationships serve as edges in the graph.

[0084] The indicator knowledge graph can mine the relationship between indicator entities, and subsequently, when performing data retrieval based on the indicator knowledge graph, more accurate retrieval results can be efficiently mined.

[0085] In an embodiment of the present invention, a data preparation method for retrieval enhanced generation in the field of comprehensive transportation planning is provided, comprising: obtaining statistical yearbook data in PDF format in the field of comprehensive transportation planning, and converting the format of the statistical yearbook data in PDF format to obtain statistical yearbook data in Excel table format; performing multi-dimensional row and column level dimensionality reduction on the statistical yearbook data in Excel table format to obtain statistical yearbook data in one-dimensional Excel table format; performing information optimization processing on the statistical yearbook data in one-dimensional Excel table format to obtain statistical yearbook data in target one-dimensional Excel table format; extracting entity relationship entity triples in the target one-dimensional Excel table format of the statistical yearbook data, and constructing an indicator knowledge graph of the statistical yearbook data in PDF format according to the extracted entity relationship entity triples, so as to perform statistical data retrieval based on the indicator knowledge graph. From the above description, it can be seen that in the data preparation method for retrieval enhanced generation in the field of comprehensive transportation planning of the present invention, the statistical yearbook data in PDF format can be format converted to obtain statistical yearbook data in Excel table form. The statistical yearbook data in Excel table form has a standardized format, which is convenient for the subsequent automatic multi-dimensional row and column level dimensionality reduction of the statistical yearbook data in Excel table form, so as to obtain statistical yearbook data in one-dimensional Excel table form that is easier to understand and has good readability. The statistical yearbook data in one-dimensional Excel table form is then subjected to information optimization processing to obtain the target statistical yearbook data in one-dimensional Excel table form. Finally, an indicator knowledge graph is constructed based on the target statistical yearbook data in one-dimensional Excel table form. The indicator knowledge graph realizes the modeling of indicator entities in the statistical yearbook data, extracts the relationship between indicator entities, and can enhance the effect of subsequent knowledge retrieval. That is, the data preparation method of the present invention improves the effect of data preparation in the field of comprehensive transportation planning, thereby improving the performance of subsequent data retrieval, and alleviating the poor effect of the existing data preparation method in the field of comprehensive transportation planning, thereby leading to the technical problem of poor data retrieval performance.

[0086] The above content briefly introduces the data preparation method for retrieval enhancement generation in the field of comprehensive transportation planning of the present invention. The specific contents involved are described in detail below.

[0087] In an optional embodiment of the present invention, the statistical yearbook data in PDF format is converted into a format, which specifically includes the following steps:

[0088] (1) Use OCR software to convert statistical yearbook data in PDF format into Excel tables to obtain the original statistical yearbook data in Excel table format;

[0089] Specifically, open source OCR software is used to convert the statistical yearbook data in PDF format (ie, scanned PDF) into an Excel table to obtain the original statistical yearbook data in the form of an Excel table.

[0090] The format of the original statistical yearbook data in the Excel table is not standardized and also contains some other information that requires further manual processing.

[0091] (2) Manually process the original statistical yearbook data in the form of Excel tables to obtain statistical yearbook data in the form of Excel tables, where the manual processing includes: yearbook table classification and renaming, multi-worksheet splitting and continuation table merging, multi-level indicator adjustment, and time and space information specification.

[0092] Specifically, filter out other information except the original statistical yearbook data in Excel form, and merge cells (such as Figure 3 Hebei Province in the PDF format) represents the hierarchical relationship and extracts basic information such as chapter name, section name, and publication time from the statistical yearbook data in PDF format.

[0093] The formats of statistical yearbook data are diverse and the information they contain is relatively complex. The positions and formats of parameters such as titles, indicators, and units cannot form a unified paradigm. The manual processing process mainly classifies different data tables into the following 9 standard format templates based on the different presentation forms of horizontal and vertical content and indicator hierarchical relationships, such as Figure 5 The present invention can solve all the data preparation tasks of these nine types of templates.

[0094] Manual processing includes: yearbook table classification and renaming, multi-worksheet splitting and continuation table merging, multi-level indicator adjustment, time and space information designation. Specifically, it includes the following contents: folder and file naming preprocessing; directory information sorting; data table name preprocessing; unit preprocessing; multi-level indicator processing; indicator name preprocessing; time information preprocessing; space information preprocessing; annotation information preprocessing (put at the end of the table); other and special cases, etc.

[0095] In an optional embodiment of the present invention, multi-dimensional row and column level dimensionality reduction is performed on the statistical yearbook data in Excel table format, specifically comprising the following steps:

[0096] (1) Identify the format and information of statistical yearbook data in Excel form;

[0097] Specifically, identifying the format and information of statistical yearbook data in Excel table format includes the following steps:

[0098] (11) Perform semantic understanding on the row and column headers of the statistical yearbook data in the Excel table format to obtain basic information, row and column header information, and table type information of the statistical yearbook data in the Excel table format;

[0099] Specifically, the specific process of semantic understanding includes the following steps:

[0100] (111) extracting and locating the title, unit and annotation information of the statistical yearbook data in the Excel table format, and obtaining the title, unit, annotation information of the statistical yearbook data in the Excel table format and the position of the annotation information in the statistical yearbook data in the Excel table format;

[0101] Specifically, from the designated location of the statistical yearbook data in Excel table form (usually located in cells A1 and A2 of the table, because the statistical yearbook data in Excel table form is manually processed and its format is standardized. When manually processed, the content in cell A1 is the title, and the content in cell A2 is the unit, such as Figure 3 As shown in the figure, extract the title and unit respectively. If A2 is not empty, it means that the data units in the statistical yearbook data in Excel table format are unified, and this unified unit is stored in the global variable of the current table. Otherwise, it means that the data units in the current table are not unified. Position the data cursor to the end of the statistical yearbook data in Excel table format, check whether there is any comment information starting with "Note:", and locate it. If the comment information is in row 500, then rows 3 to 499 are the header area and data area to be reduced in dimension.

[0102] (112) Determine the chapter and section numbers corresponding to the statistical yearbook data in Excel form;

[0103] Specifically, according to the catalog information and the title of the statistical yearbook data in Excel table format, the chapter number and section number corresponding to the statistical yearbook data in Excel table format are determined. Figure 3 As shown, the worksheet at the bottom contains the table of contents. In addition, in cell A1, 2-25 represents Chapter 2, Section 25, indicating that the chapter number is 2 and the section number is 25.

[0104] (113) Determine the row header range and column header range of statistical yearbook data in Excel table format;

[0105] Specifically, Figure 3 As shown, the row header range is rows 3-5, and the column header range is columns A and B.

[0106] (114) Determine the table type of the statistical yearbook data in Excel table format, wherein the table type includes at least one of the following: a single-row header, a multi-row header, a single-column header, and a multi-column header.

[0107] Specifically, a single-row header means that the row header does not have a hierarchical relationship and has only one row; a single-column header means that the column header does not have a hierarchical relationship and has only one column; a multi-row header means that the row header contains a hierarchical relationship and has at least two rows; a multi-column header means that the column header contains a hierarchical relationship and has at least two columns.

[0108] (12) identifying the time field and space field of the statistical yearbook data in the Excel table format to obtain the yearbook time and space information of the statistical yearbook data in the Excel table format;

[0109] Specifically, the identification of the time field and the space field includes the following steps:

[0110] (121) Determine the yearbook time of statistical yearbook data in Excel table format;

[0111] Specifically, determine whether the file name of the statistical yearbook data in Excel table format meets the format of "4-digit year number + Chinese character or underscore or letter + any character except line break + .xls or .xlsx" (when processed manually, it has been standardized, so the format of the file name is fixed). If it meets the format, recognize the "4-digit year number" as the yearbook time; otherwise, report an error and record the error.

[0112] (122) Determine the spatial fields in the statistical yearbook data in the Excel table format and the positions of the spatial fields in the statistical yearbook data in the Excel table format.

[0113] Specifically, the field name of the spatial field is usually "region" and is located in a row or column. When implementing, you can query the spatial information dictionary maintained by yourself to determine whether the data in the statistical yearbook data in the Excel table format is a spatial field, and determine the position of each spatial field in the statistical yearbook data in the Excel table format. For example, if the spatial field is Beijing, its position is the 6th row, that is, the 6th row is all data for Beijing.

[0114] (13) Identifying subheadings and hierarchical relationships of subheadings in the statistical yearbook data in the Excel table format, wherein the subheadings are bold content in the data area of ​​the statistical yearbook data in the Excel table format;

[0115] Specifically, Figure 3As shown, the data area is formed below the indicator area (gray area, also called the header area) formed by the indicator rows and indicator columns. In the data area, check row by row whether there is bold content, identify it as a subheading, and insert a hierarchical relationship for it, that is, the subheading is regarded as part of the indicator. For example, below the city and districts in row 6 is not a numerical value, but 2018, in bold format, so 2018 is the subheading, and its hierarchical relationship is Beijing: 2018.

[0116] (14) Identify and process outliers in statistical yearbook data in Excel format;

[0117] Specifically, outlier identification and conversion are performed. All data areas are checked. If a cell contains special characters (non-digits, non-Chinese and English characters, non-common characters such as "￥", "$", "%", escape characters such as line breaks), and the cell contains other legal characters, the special characters are replaced with empty strings; if a cell contains only special characters, the cell content is marked as missing values;

[0118] Irregular text identification and conversion. Check all indicator areas. If a cell contains irregular text such as "where:" or "within:" defined by the user, replace the irregular text with an empty string. If a cell is entirely irregular text, mark the cell content as a missing value.

[0119] (15) The basic information, row and column header information, table type information, yearbook time, spatial information, subheadings and the hierarchical relationship of subheadings are used as the format and information of statistical yearbook data in the form of Excel tables.

[0120] (2) Integrate and rewrite the hierarchical relationship of row and column headers based on the identified format and information to obtain the statistical yearbook data in the form of a one-dimensional Excel table.

[0121] Specifically, the hierarchical relationship of the row and column headers is integrated and rewritten according to the recognized format and information, which specifically includes the following steps:

[0122] (21) Split the merged cells based on the table type, row header range, and column header range to obtain multiple empty cells, and fill the empty cells with the hierarchical content of the corresponding merged cells;

[0123] Specifically, Figure 3 The Hebei Province cell shown is a merged cell. Split it to get multiple empty cells A8-A18, and fill the empty cells with Hebei Province.

[0124] Split the row and column headers and merge the cells separately, and align the multiple empty cells. Alignment operation: Assuming that the maximum level of the data table after splitting is n, fill the data rows that are less than n levels with empty levels; then, for multi-row headers, first fill the missing values ​​downward within the row header range, and then fill the missing values ​​to the right; for multi-column headers, first fill the missing values ​​to the "right" within the column header range, and then fill the missing values ​​"downward". For example, after splitting the Hebei Province cell and aligning it, you will get 10 empty cells (i.e. empty levels) from A8 to A18, and then fill the Hebei Province "downward".

[0125] (22) Use the level separator to splice the row-level content, and delete duplicate content from the spliced ​​row-level content to obtain a row splicing value;

[0126] Specifically, the level separator can be ":", such as Figure 3 As shown, after the row-level contents of the 8th to 18th rows are spliced, the spliced ​​row-level contents are: Hebei Province: Shijiazhuang City, Hebei Province: Tangshan City, Hebei Province: Qinhuangdao City, etc., which are not listed here one by one. If there are continuous repeated contents in the spliced ​​row-level contents, the repeated contents are deleted to obtain the row splicing value. For example, if the spliced ​​row-level contents are Hebei Province: Shijiazhuang City: Shijiazhuang City: Qiaoxi District, then delete one Shijiazhuang City, and the final row splicing value is: Hebei Province: Shijiazhuang City: Qiaoxi District.

[0127] (23) Use the level separator to splice the column-level content, and delete duplicate content from the spliced ​​column-level content to obtain a column splicing value;

[0128] (24) A one-dimensional Excel table with column names is constructed based on the title, unit, annotation information, chapter number, section number, and yearbook time. Based on the spatial fields and the positions of the spatial fields in the statistical yearbook data in the Excel table, the values ​​of the data areas, row splicing values, and column splicing values ​​in the statistical yearbook data in the Excel table are filled into the corresponding column names to obtain the statistical yearbook data in the one-dimensional Excel table.

[0129] Specifically, a one-dimensional Excel table with column names is as follows Figure 4 As shown, based on the spatial fields and the positions of the spatial fields in the statistical yearbook data in the Excel table form, the values ​​of the data areas, the row splicing values ​​and the column splicing values ​​in the statistical yearbook data in the Excel table form are filled into the corresponding column names.

[0130] The above content obtains the unit, time information and space information, which are replaced with unified standard names, such as: unit, yearbook time, region. Then, the column name and the corresponding column name value are stacked, that is, for the data area, a dictionary is constructed according to the column name: {"Indicator name": corresponding indicator name, "Indicator value": corresponding indicator value, "Region": corresponding region name, "Unit": corresponding unit name, "Yearbook name": corresponding yearbook name, "Yearbook time": corresponding yearbook time, "Data time": corresponding data time, ...}, and the data area is filled to obtain the statistical yearbook data in the form of a one-dimensional Excel table.

[0131] In an optional embodiment of the present invention, information optimization processing is performed on statistical yearbook data in a one-dimensional Excel table format, specifically comprising the following steps:

[0132] Determine the data time and unit information based on the statistical yearbook data in the form of a one-dimensional Excel table.

[0133] Specifically, the data time structure. Check whether there is data time field information in the row header (the field name is usually "year"). If it exists, use the corresponding column of the field information as the data time column; if it does not exist, find out whether there is time information in the "indicator name" column. If there is time information, use the time information as the data time of the data area, and delete the corresponding time information in the indicator name. Otherwise, use the yearbook time information as the data time information.

[0134] Unit information identification. If there is a unified unit in this worksheet, use this unit information as the unit information for each row in the data area. Otherwise, check whether there is unit information in the "Indicator Name" column of each row according to the self-maintained unit information dictionary. If so, use the unit information as the unit information for each row in the data area and delete the corresponding unit information in the indicator name. Otherwise, an error message will be prompted.

[0135] Indicator name and unit concatenation. To facilitate user viewing, add the unit information of each row in the data area to the end of the corresponding indicator name.

[0136] Dimensionality reduction result file output. Save the dimension reduction result as a CSV format file, which is the statistical yearbook data in the target one-dimensional Excel table format.

[0137] The above dimensionality reduction process can be summarized as the process in the following table:

[0138]

[0139] In an optional embodiment of the present invention, the method further comprises the following steps:

[0140] (1) Obtaining natural language input by a user for statistical data retrieval;

[0141] (2) Using a large natural language model to convert natural language used for statistical data retrieval into graph query statements;

[0142] (3) Query the indicator knowledge graph according to the graph query statement to obtain the query results;

[0143] (4) A natural language big model is used to convert the query results into natural language to obtain the query results in natural language.

[0144] Specifically, the user inputs natural language for statistical data retrieval (such as "How many national highways are there in Beijing in 2019, divided by administrative level?"), uses an open source natural language big model (such as Qwen27B) to convert the natural language for statistical data retrieval into a graph query Cypher statement, uses the Cypher statement to query in the Neo4j graph database, obtains the query results, and then returns the query results to the natural language big model for statistical data retrieval answer generation.

[0145] The data preparation method for retrieval enhancement generation in the field of comprehensive transportation planning of the present invention includes: preparation and conversion of unstructured data to semi-structured data, indicator knowledge graph generation and indicator knowledge graph retrieval for retrieval enhancement generation, modeling of indicator entities in statistical yearbook data, extraction of relational feature information between indicator entities to enhance knowledge retrieval effect, and completion of the knowledge acquisition technology route of "natural language knowledge retrieval, graph retrieval and answer generation", which significantly improves the quality and efficiency of data preparation and knowledge retrieval in a semi-automatic manner.

[0146] Embodiment 2:

[0147] An embodiment of the present invention also provides a data preparation device for retrieval and enhancement generation in the field of comprehensive transportation planning. The data preparation device for retrieval and enhancement generation in the field of comprehensive transportation planning is mainly used to execute the data preparation method for retrieval and enhancement generation in the field of comprehensive transportation planning provided in the first embodiment of the present invention. The following is a specific introduction to the data preparation device for retrieval and enhancement generation in the field of comprehensive transportation planning provided in the embodiment of the present invention.

[0148] Figure 6 is a schematic diagram of a data preparation device for retrieving enhanced generation in the field of comprehensive traffic planning according to an embodiment of the present invention, such as Figure 6 As shown, the device mainly includes: a format conversion unit 10, a multi-dimensional row and column level dimension reduction unit 20, an information optimization processing unit 30, and an extraction and construction unit 40, wherein:

[0149] A format conversion unit is used to obtain statistical yearbook data in PDF format in the field of comprehensive transportation planning, and to convert the statistical yearbook data in PDF format to obtain the statistical yearbook data in Excel table form;

[0150] A multi-dimensional row and column level dimension reduction unit is used to perform multi-dimensional row and column level dimension reduction on the statistical yearbook data in the form of an Excel table to obtain the statistical yearbook data in the form of a one-dimensional Excel table;

[0151] An information optimization processing unit is used to perform information optimization processing on the statistical yearbook data in the form of a one-dimensional Excel table to obtain the statistical yearbook data in the form of a target one-dimensional Excel table;

[0152] Extraction and construction units are used to extract entity-relationship entity triples from the target statistical yearbook data in the form of a one-dimensional Excel table, and an indicator knowledge graph of the statistical yearbook data in PDF format is constructed based on the extracted entity-relationship entity triples to perform statistical data retrieval based on the indicator knowledge graph.

[0153] In an embodiment of the present invention, a data preparation device for retrieval enhancement generation in the field of comprehensive transportation planning is provided, including: obtaining statistical yearbook data in PDF format in the field of comprehensive transportation planning, and converting the format of the statistical yearbook data in PDF format to obtain statistical yearbook data in Excel table format; performing multi-dimensional row and column level dimensionality reduction on the statistical yearbook data in Excel table format to obtain statistical yearbook data in one-dimensional Excel table format; performing information optimization processing on the statistical yearbook data in one-dimensional Excel table format to obtain statistical yearbook data in target one-dimensional Excel table format; extracting entity relationship entity triples in the target one-dimensional Excel table format of the statistical yearbook data, and constructing an indicator knowledge graph of the statistical yearbook data in PDF format according to the extracted entity relationship entity triples, so as to perform statistical data retrieval based on the indicator knowledge graph. From the above description, it can be known that in the data preparation device for retrieval enhancement generation in the field of comprehensive transportation planning of the present invention, the statistical yearbook data in PDF format can be format converted to obtain statistical yearbook data in Excel table form. The statistical yearbook data in Excel table form has a standardized format, which is convenient for the subsequent automatic multi-dimensional row and column level dimensionality reduction of the statistical yearbook data in Excel table form, so as to obtain statistical yearbook data in one-dimensional Excel table form that is easier to understand and has good readability. The statistical yearbook data in one-dimensional Excel table form is then subjected to information optimization processing to obtain the target statistical yearbook data in one-dimensional Excel table form. Finally, an indicator knowledge graph is constructed according to the target statistical yearbook data in one-dimensional Excel table form. The indicator knowledge graph realizes the modeling of indicator entities in the statistical yearbook data, extracts the relationship between indicator entities, and can enhance the effect of subsequent knowledge retrieval. That is, the data preparation method of the present invention improves the effect of data preparation in the field of comprehensive transportation planning, thereby improving the performance of subsequent data retrieval, and alleviating the poor effect of the existing data preparation method in the field of comprehensive transportation planning, thereby leading to the technical problem of poor data retrieval performance.

[0154] Optionally, the format conversion unit is also used to: use OCR software to convert the statistical yearbook data in PDF format into an Excel table to obtain the original statistical yearbook data in the Excel table format; manually process the original statistical yearbook data in the Excel table format to obtain the statistical yearbook data in the Excel table format, wherein the manual processing includes: classification and renaming of yearbook tables, splitting of multiple worksheets and merging of continued tables, adjustment of multi-level indicators, and specification of time and space information.

[0155] Optionally, the multi-dimensional row and column level dimension reduction unit is also used to: identify the format and information of statistical yearbook data in Excel table form; integrate and rewrite the row and column header level relationship according to the identified format and information to obtain the statistical yearbook data in one-dimensional Excel table form.

[0156] Optionally, the multi-dimensional row and column hierarchical dimensionality reduction unit is also used to: perform semantic understanding on the row and column headers of the statistical yearbook data in the form of Excel tables to obtain basic information, row and column header information and table type information of the statistical yearbook data in the form of Excel tables; identify the time field and space field of the statistical yearbook data in the form of Excel tables to obtain yearbook time and space information of the statistical yearbook data in the form of Excel tables; identify subheadings and hierarchical relationships of subheadings in the statistical yearbook data in the form of Excel tables, wherein the subheadings are bold content in the data area of ​​the statistical yearbook data in the form of Excel tables; identify outliers in the statistical yearbook data in the form of Excel tables and process the outliers; and use the basic information, row and column header information, table type information, yearbook time, space information, subheadings and hierarchical relationships of subheadings as the format and information of the statistical yearbook data in the form of Excel tables.

[0157] Optionally, the multi-dimensional row and column level dimensionality reduction unit is also used to: extract and locate the title, unit and annotation information of the statistical yearbook data in the Excel table form, obtain the title, unit, annotation information of the statistical yearbook data in the Excel table form and the position of the annotation information in the statistical yearbook data in the Excel table form; determine the chapter number and section number corresponding to the statistical yearbook data in the Excel table form; determine the row header range and column header range of the statistical yearbook data in the Excel table form; determine the table type of the statistical yearbook data in the Excel table form, wherein the table type includes at least one of the following: a single-row header, multiple-row headers, a single-column header and multiple-column headers.

[0158] Optionally, the multidimensional row and column level dimensionality reduction unit is also used to: determine the yearbook time of the statistical yearbook data in the Excel table format; determine the spatial fields in the statistical yearbook data in the Excel table format and the positions of each spatial field in the statistical yearbook data in the Excel table format.

[0159] Optionally, the multi-dimensional row and column level dimension reduction unit is also used to: split the merged cells based on the table type, row header range and column header range to obtain multiple empty cells, and fill the corresponding hierarchical content of the merged cells in the empty cells; use the level separator to splice the row level content, and delete the duplicate content of the spliced ​​row level content to obtain the row splicing value; use the level separator to splice the column level content, and delete the duplicate content of the spliced ​​column level content to obtain the column splicing value; construct a one-dimensional Excel table with column names based on the title, unit, annotation information, chapter number, section number, and yearbook time, and fill the values ​​of the data area, row splicing values ​​and column splicing values ​​in the statistical yearbook data in the Excel table form into the corresponding column names based on the spatial fields and the positions of each spatial field in the statistical yearbook data in the Excel table form to obtain the statistical yearbook data in the one-dimensional Excel table form.

[0160] Optionally, the information optimization processing unit is further used to determine data time and unit information based on the statistical yearbook data in the form of a one-dimensional Excel table.

[0161] Optionally, the device is also used to: obtain natural language input by a user for statistical data retrieval; use a natural language big model to convert the natural language used for statistical data retrieval into a graph query statement; query in the indicator knowledge graph according to the graph query statement to obtain query results; use the natural language big model to convert the query results into natural language to obtain natural language query results.

[0162] The device provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0163] like Figure 7 As shown, an electronic device 600 provided in an embodiment of the present application includes: a processor 601, a memory 602 and a bus, wherein the memory 602 stores machine-readable instructions executable by the processor 601. When the electronic device is running, the processor 601 communicates with the memory 602 through the bus, and the processor 601 executes the machine-readable instructions to perform the steps of the data preparation method for retrieving enhanced generation in the field of comprehensive transportation planning as mentioned above.

[0164] Specifically, the above-mentioned memory 602 and processor 601 can be general-purpose memories and processors, which are not specifically limited here. When the processor 601 runs the computer program stored in the memory 602, it can execute the data preparation method for retrieval and enhanced generation in the above-mentioned field of comprehensive transportation planning.

[0165] The processor 601 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 601. The above processor 601 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 602, and the processor 601 reads the information in the memory 602 and completes the steps of the above method in combination with its hardware.

[0166] Corresponding to the data preparation method for retrieving enhanced generation in the field of comprehensive transportation planning mentioned above, an embodiment of the present application also provides a computer-readable storage medium, which stores machine-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to execute the steps of the data preparation method for retrieving enhanced generation in the field of comprehensive transportation planning mentioned above.

[0167] The data preparation device for retrieving enhanced generation in the field of comprehensive transportation planning provided in the embodiment of the present application can be specific hardware on the device or software or firmware installed on the device. The device provided in the embodiment of the present application, its implementation principle and the technical effect produced are the same as those in the aforementioned method embodiment. For the sake of brief description, the parts not mentioned in the device embodiment can refer to the corresponding contents in the aforementioned method embodiment. Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working processes of the systems, devices and units described above can all refer to the corresponding processes in the aforementioned method embodiment, and will not be repeated here.

[0168] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0169] For another example, the flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the device, method and computer program product according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of the boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0170] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0171] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0172] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including a number of instructions to enable an electronic device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the data preparation method for retrieving enhanced generation in the field of comprehensive transportation planning described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0173] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0174] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solution of the present application, rather than to limit it. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the aforementioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solution recorded in the aforementioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solution deviate from the scope of the technical solution of the embodiment of the present application. They should all be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A data preparation method for retrieval enhancement generation in the field of comprehensive transportation planning, characterized in that: include: Obtain statistical yearbook data in the field of comprehensive transportation planning in PDF format, and convert the statistical yearbook data in PDF format to obtain the statistical yearbook data in Excel table form; Performing multi-dimensional row and column level dimensionality reduction on the statistical yearbook data in the Excel table form to obtain the statistical yearbook data in the one-dimensional Excel table form; Performing information optimization processing on the statistical yearbook data in the one-dimensional Excel table format to obtain the statistical yearbook data in the target one-dimensional Excel table format; Entity-relationship-entity triples are extracted from the target statistical yearbook data in the form of a one-dimensional Excel table, and an indicator knowledge graph of the statistical yearbook data in the PDF format is constructed based on the extracted entity-relationship-entity triples, so as to perform statistical data retrieval based on the indicator knowledge graph.

2. The method according to claim 1, characterized in that: The statistical yearbook data in PDF format is converted into a format, including: The statistical yearbook data in PDF format is converted into an Excel table using OCR software to obtain the original statistical yearbook data in Excel table form; Manual processing is performed on the original statistical yearbook data in the form of Excel tables to obtain the statistical yearbook data in the form of Excel tables, wherein the manual processing includes: yearbook table classification and renaming, multi-worksheet splitting and continuation table merging, multi-level indicator adjustment, and time and space information specification.

3. The method according to claim 1, characterized in that Performing multi-dimensional row and column level dimensionality reduction on the statistical yearbook data in the Excel table format, including: Identify the format and information of the statistical yearbook data in Excel table form; The hierarchical relationship of the row and column headers is integrated and rewritten according to the recognized format and information to obtain the statistical yearbook data in the one-dimensional Excel table form.

4. The method according to claim 3, characterized in that Identify the format and information of the statistical yearbook data in Excel table format, including: Performing semantic understanding on the row and column headers of the statistical yearbook data in the Excel table format to obtain basic information, row and column header information, and table type information of the statistical yearbook data in the Excel table format; Identify the time field and space field of the statistical yearbook data in the Excel table format to obtain the yearbook time and space information of the statistical yearbook data in the Excel table format; Identifying subheadings in the statistical yearbook data in the Excel table format and hierarchical relationships between the subheadings, wherein the subheadings are bold content in a data area in the statistical yearbook data in the Excel table format; Identifying outliers in the statistical yearbook data in the Excel table format and processing the outliers; The basic information, the row and column header information, the table type information, the yearbook time, the space information, the subheadings and the hierarchical relationship of the subheadings are used as the format and information of the statistical yearbook data in the Excel table form.

5. The method according to claim 4, characterized in that The semantic understanding of the row and column headers of the statistical yearbook data in the Excel table format includes: Extracting and locating the title, unit and annotation information of the statistical yearbook data in the Excel table format, and obtaining the title, unit, annotation information of the statistical yearbook data in the Excel table format and the position of the annotation information in the statistical yearbook data in the Excel table format; Determine the chapter number and section number corresponding to the statistical yearbook data in Excel table format; Determine the row header range and column header range of the statistical yearbook data in the Excel table format; Determine the table type of the statistical yearbook data in Excel table format, wherein the table type includes at least one of the following: a single-row header, a multi-row header, a single-column header, and a multi-column header.

6. The method according to claim 5, characterized in that Identifying the time field and space field of the statistical yearbook data in the Excel table format includes: Determine the yearbook time of the statistical yearbook data in the Excel table format; Determine the spatial fields in the statistical yearbook data in the Excel table format and the position of each of the spatial fields in the statistical yearbook data in the Excel table format.

7. The method according to claim 6, characterized in that Integrate and rewrite the row and column header hierarchical relationship based on the recognized format and information, including: Based on the table type, the row header range, and the column header range, the merged cells are split to obtain a plurality of empty cells, and the hierarchical contents of the corresponding merged cells are filled in the empty cells; Use the level separator to concatenate the row-level content, and delete duplicate content from the concatenated row-level content to obtain a row concatenation value; Use the level separator to splice the column-level content, and delete duplicate content from the spliced ​​column-level content to obtain the column splicing value; A one-dimensional Excel table with column names is constructed based on the title, the unit, the annotation information, the chapter number, the section number, and the yearbook time, and based on the spatial fields and the positions of the spatial fields in the statistical yearbook data in the Excel table form, the values ​​of the data areas, the row splicing values, and the column splicing values ​​in the statistical yearbook data in the Excel table form are filled into the corresponding column names to obtain the statistical yearbook data in the one-dimensional Excel table form.

8. The method according to claim 1, characterized in that: The statistical yearbook data in the one-dimensional Excel table form is subjected to information optimization processing, including: The data time and unit information are determined based on the statistical yearbook data in the one-dimensional Excel table format.

9. The method according to claim 1, characterized in that: The method further comprises: Obtaining natural language input by a user for statistical data retrieval; Using a natural language big model to convert the natural language used for statistical data retrieval into a graph query statement; Query the indicator knowledge graph according to the graph query statement to obtain a query result; The natural language big model is used to perform natural language conversion on the query result to obtain a natural language query result.

10. A data preparation device for retrieval enhancement generation in the field of comprehensive transportation planning, characterized in that: include: A format conversion unit, used for obtaining statistical yearbook data in PDF format in the field of comprehensive transportation planning, and performing format conversion on the statistical yearbook data in PDF format to obtain statistical yearbook data in Excel table form; A multi-dimensional row and column level dimension reduction unit is used to perform multi-dimensional row and column level dimension reduction on the statistical yearbook data in the Excel table form to obtain the statistical yearbook data in the one-dimensional Excel table form; An information optimization processing unit, used for performing information optimization processing on the statistical yearbook data in the one-dimensional Excel table format to obtain the statistical yearbook data in the target one-dimensional Excel table format; An extraction and construction unit extracts entity-relationship-entity triples from the target statistical yearbook data in the form of a one-dimensional Excel table, and constructs an indicator knowledge graph of the statistical yearbook data in the PDF format based on the extracted entity-relationship-entity triples, so as to perform statistical data retrieval based on the indicator knowledge graph.

Images