A structured data conversion system and method

Abstract

The present invention discloses a structured data conversion system, comprising: a text feature definition module for providing defined text rules; an unstructured data preprocessing module for reading a specified unstructured table file, extracting text features of the table file, matching corresponding text data templates and text data maps according to the text features, and creating a conversion data object copy; a structured data conversion module for collecting information items and extracting structured data units from the conversion data object copy; a structured data generation module for normalizing structured data units and generating structured data objects. According to the above technical solution, the text feature definition modules such as data templates and data maps of the three-dimensional prevention and control system of social security can be utilized to batch process unstructured table files that meet the specifications, extract and convert data according to text features, and store structured and content-standardized data in a database for subsequent intelligence analysis.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a system and method for converting structured data. Background Art

[0002] Integrating intelligence thinking into the development of a three-dimensional public security prevention and control system, adhering to the concept of intelligence-led policing, and conducting public security intelligence information analysis and assessment are key priorities in the current development of public security systems. Multi-source data is the foundation of intelligence analysis. Historical case files and other documentary materials contain a wealth of knowledge and are a crucial source of data for multi-dimensional intelligence analysis. Due to the relatively slow progress of public security informatization in some regions, prior to the establishment of law enforcement and case handling information systems, many documentary materials, such as case files, were compiled by grassroots police officers. This resulted in a large amount of textual data that was not yet integrated into the structured databases of the system. Historical case files and other documentary materials also need to be converted into structured data to enable data correlation. To achieve multi-dimensional data correlation and collision analysis, and to support higher-level public security operations, large amounts of documentary materials must be structured extracted, standardized, and stored in a database for subsequent intelligence analysis.

[0003] Most of the files involved in public security informatization are tabular files with specific formatting rules. Due to the merging and decomposition of cells and the redefinition of cell correspondences within the tables, these tabular files cannot be easily structured. While machine learning can be used to classify text, it requires massive amounts of data to support the data processing models. Furthermore, even with standardized forms, manually completing them can hardly achieve uniform content standards. Therefore, a solution is needed to convert the unstructured tabular files in the files of the three-dimensional public security prevention and control system into structured data while also standardizing the data content. Summary of the Invention

[0004] In a first aspect, to achieve the above-mentioned objectives, the present application provides a structured data conversion system, comprising:

[0005] Text feature definition module: used to provide defined text rules, wherein the text rules include text feature relationships, text data templates and text data maps; wherein the text feature relationships and text data maps have a one-to-one correspondence; the text data templates are generated by the text data maps;

[0006] Unstructured data preprocessing module: used to read the specified unstructured table file, extract the text features of the table file, match the corresponding text data template and text data map according to the text features, and create a copy of the converted data object;

[0007] The structured data conversion module is configured to collect information items from the converted data object copy and extract data values ​​corresponding to the information items to form structured data units; wherein the structured data conversion module includes a collection unit configured to extract data values ​​corresponding to the information items;

[0008] Structured data generation module: used to normalize structured data units and generate structured data objects.

[0009] Furthermore, the unstructured data preprocessing module includes:

[0010] Text reader: used to extract unstructured table files and output file content objects;

[0011] Text feature extraction unit: used to obtain feature text of unstructured table files and determine text features corresponding to document content objects; wherein the feature text includes text character content, font, and font size;

[0012] Data object copy creation unit: obtains the corresponding text data template and text data map according to the text features, and generates a data object copy of the unstructured table file.

[0013] Furthermore, the collection unit includes a strong verification mode data item collection unit; the data source of the strong verification mode data item collection unit is the text data map corresponding to the unstructured table file; the collection method is to traverse the information item nodes in the text data map one by one, and extract the information item title feature information and the information item content information; it also includes a validity verifier.

[0014] Furthermore, the collection unit also includes an adaptive mode data item collection unit; the data source supported by the adaptive mode data item collection unit is the cells corresponding to the unstructured table file; the collection method supported by the adaptive mode data item collection unit is to traverse all cells of the table, extract the cell content, and match it with the information item title feature information and the information item content information; at the same time, it also supports the status marking function.

[0015] Furthermore, the structured data generation module includes a data normalization processing module for formatting data values ​​of a specific type in the data object copy.

[0016] Among them, text rules also include standard title word dictionaries, value mapping tables, and information item warehouses.

[0017] On the other hand, the present application provides a method for converting structured data, comprising the following steps:

[0018] Determine text rules, including determining text feature relationships, text data templates and text data maps; text feature relationships and text data maps Figure 1One-to-one correspondence; text data template is generated by text data map;

[0019] Read the specified unstructured table file, extract its text features, match the corresponding text data template and text data map according to the text features, and create a copy of the converted data object;

[0020] Collecting information items from the converted data object copy and extracting data values ​​corresponding to the information items to form structured data;

[0021] Further normalize structured data and generate structured data objects.

[0022] Among them, extracting text features of a table file refers to obtaining feature text of an unstructured table file and determining text features corresponding to a document content object; wherein the feature text includes text character content, font, and font size.

[0023] Furthermore, collecting information items from the converted data object copy and extracting data values ​​corresponding to the information items to form structured data includes: strong verification mode data item collection type and adaptive mode data item collection type:

[0024] Among them, the data source of the strong verification mode data item collection type is the text data map corresponding to the unstructured table file; the collection method provided by the strong verification mode data item collection type is to traverse the information item nodes in the text data map one by one, and extract the information item title feature information and information item content information;

[0025] The data source supported by the adaptive mode data item collection type is the cells corresponding to the unstructured table file; the collection method supported by the adaptive mode data item collection type is to traverse the cells of all tables, extract the cell content, and match it with the information item title feature information and the information item content information.

[0026] According to the present invention, in some specific environments, such as historical files required for the construction of a three-dimensional social security prevention and control system, and file contents filled in by grassroots personnel, unstructured table files with uniform formats and unstructured files with similar content meanings but not unified standards can be converted into structured data that conforms to unified standards in terms of data content, and stored in the form of databases and structured files, which is convenient for access to the information construction system and full utilization in subsequent intelligence analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a structural diagram of a structured data conversion system provided according to an embodiment of the present invention;

[0028] Figure 2 1 is a flowchart of a strong verification mode data item acquisition unit according to an embodiment of the present invention;

[0029] Figure 3 4 is a flow chart of a method for converting structured data provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The specific implementation of the present invention is described in detail below with reference to the accompanying drawings.

[0031] The present invention summarizes the table structure features, table content, and content text features of the three-dimensional social security prevention and control system to form text feature definition modules such as data templates and data maps. For the unstructured table files to be processed, data is extracted and converted according to the text features to form structured data.

[0032] Figure 1 A structured data conversion system structure diagram is provided, as shown in the figure, including the following parts:

[0033] P100: Text feature definition module: used to provide defined text rules, where the text rules are the feature constraints of the converted structured data, including standard title word dictionary, value mapping table, information item warehouse, text feature relationship, text data template and text data map; the text feature relationship and the text data map are in a one-to-one correspondence; the text data template is generated by the text data map.

[0034] 1. The standard title word dictionary title-dictionary includes the basic data units identified from the table files of all files, that is, the title content corresponding to the information items, including standard titles, which are characteristic titles selected from multiple titles with the same meaning, and also includes synonyms and abbreviations for expanding each title word.

[0035] 2. The item library includes the data structures of all information items found in the tabular files of all dossiers. These include attributes such as the standard title, value data type, value range, and value validity check rules. This includes the value map, which maps data types to value sets. This map is a value set that aggregates data types with fixed value ranges and all standard and equivalent values.

[0036] For example, taking the standard title of the information item "Case Source" in the field of public security as an example, the information item value type is a string, and its fixed value range includes: 110 instructions, discovery during work, report, surrender, transfer, arrest, and others.

[0037] 3. The text feature relationship refers to the text specifications of the file documents and materials, and extracts the unique text content features of each text type through features such as content paragraph position, text character content, and font format. It is used to establish a mapping relationship table feature-map between each social security text type and text content features. The text type docType can be reversely inferred through the text content features.

[0038] Taking the text content title feature as an example, when comparing features, pure string matching or regular expression matching is used. The feature acquisition process includes the following steps:

[0039] A. Remove the blank string in the title text at the top of the title cell to get title t1;

[0040] B. Remove the punctuation marks from the title text t1 processed in step A to obtain the pure string title t2;

[0041] C. Extract the font name and font size of the top title text as one of the features.

[0042] D. Concatenate t2, the font name, and the font size into a string, perform MD5 calculation, and obtain the text content feature value feature1.

[0043] 4. Text data maps are data relationships constructed using structured data formats (such as JSON and XML). Data maps use a tree structure, with information items as the smallest unit. They define the hierarchical relationships, standard titles, title locations, value locations, data types, data lengths, and the order of information items within the text. Each data map is associated one-to-one with a text type.

[0044] For example, if there is a data table as follows, it contains three information items: reporter (aggregate combination), name and contact information.

[0045]

[0046] The corresponding data map format is as follows:

[0047]

[0048]

[0049] 5. The text data template is a data map based on the text content. By cutting out descriptive fields such as the title location, value location, and data length, adding value data fields, and establishing a structured data format (such as JSON or XML) for the text content, each data template is associated one-to-one with the text type.

[0050] In the above example, the text data template generated by the data map conversion is as follows:

[0051]

[0052] P110: Unstructured data preprocessing module: used to read the specified unstructured table file, extract the text features of the table file, match the corresponding text data template and text data map according to the text features, and create a copy of the converted data object; including:

[0053] Text reader: used to extract unstructured table files and output file content objects, that is, to obtain a document content object file; taking Java language as an example, Apache POI components can be used to read electronic documents;

[0054] Text feature extraction unit: used to obtain the feature text of the unstructured table file and determine the text features corresponding to the document content object; the feature text includes text character content, font, and font size; the text features are combined with the mapping relationship table to reversely determine the text type of the table file;

[0055] For example, get the title cell of a table, remove blank characters to get t1, remove punctuation marks to get the pure string title t2, and combine the font name and font size to calculate the feature value feature1.

[0056] The data object replica creation unit: Based on the text features obtained by the text feature extraction unit, the corresponding text data template and text data map are obtained through text feature relationships to generate a data object replica for the unstructured table file. The data object replica is obtained by converting the data template file using a data parsing component (applicable to JSON or XML) to obtain a model object replica, which is used to store and update the reading results of social security text data.

[0057] P120: Structured data conversion module: used to collect information items from the data object replica and extract data values ​​corresponding to the information items to form structured data units; wherein the structured data conversion module includes a collection unit, used to extract data values ​​corresponding to specified information items;

[0058] The collection unit includes a strong verification mode data item collection unit:

[0059] The data source of the strong verification mode data item collection unit is the text data map corresponding to the unstructured table file; the collection method provided is to traverse the information item nodes in the text data map one by one, extracting the information item title feature information and information item content information, as described in detail below:

[0060] When collecting information, first obtain the table collection object in the document content object file, then obtain the attribute value from the data map object, and extract data for the selected cell. The extraction process uses the method of traversing the information item nodes in the data map one by one to find the information item attributes (title, title position, value position, type, child node) that need to be read from the map structure. The strong verification mode data item collection unit also includes a validity verifier to determine whether the attribute value of each information item is valid.

[0061] Its workflow is as follows Figure 2 As shown:

[0062] Step S210: Obtain a node object from the data map;

[0063] Step S211: read the value of the title attribute of the node object, title';

[0064] Step S212: Find all words related to title' in the standard title word dictionary title-dictionary of the text feature definition module to form a title character set title-array;

[0065] Step S213: Read the title location attribute under the node object, obtain the table number, row position and column position of the cell to be read, and read the content title_value of the corresponding position cell from the target text table collection object tables;

[0066] Step S220: The validity checker compares title_value with the title character set title-array generated in step S212 item by item. If the title character set title-array contains the cell content title_value, it means that the node title verification has passed, and the process proceeds to the next step.

[0067] Step S221: Read the value location attribute under dNode, obtain the table number, row position and column position of the cell to be read, and read the content value2 and the value type attribute value type' of the corresponding position cell from the target text table collection object tables;

[0068] Step S222: Search the value set that matches type' in the value mapping table in the text feature definition module, and use the validity checker to compare value2 with the value set item by item to verify its validity; if the value validity verification passes, proceed to the next step;

[0069] Step S223: The information item data titled "title" is retrieved from the item-library in the text feature definition module. The validity checker verifies the validity of value2 by comparing it with the value range and value verification rules of the information item. If the validity verification passes, the process proceeds to the next step.

[0070] Step S224: Save the valid result value value2 to the value attribute of the node object with the same position offset in the data object dTemplate;

[0071] If an information item has multiple values, step S221 is executed again;

[0072] If the data type type' of the node object is a parent node, each node object in its child item list is read cyclically, and step S221 is executed again.

[0073] Take the following table as an example:

[0074]

[0075] Traverse the data map, read the value of each information item and fill it into the data object. The result is as follows:

[0076]

[0077]

[0078] The collection unit provided by this application also includes an adaptive mode data item collection unit: the data source supported by the adaptive mode data item collection unit is the cells of the unstructured table file itself; the supported collection method is to traverse all the cells of the table, extract the cell content, and match it with the information item title feature information and information item content information. Specifically, it includes the following contents:

[0079] A. Read the cell content to obtain value3, and compare the read value3 with the standard title word dictionary title-dictionary (including synonyms and abbreviations) item by item. If the comparison is successful, the cell is confirmed to be the information item title, and the standard title word title' is obtained; obtain the information item data with the title title' from the information item warehouse, and determine whether the data value needs to be extracted based on the information item type. For information items that only represent parent relationships, jump to the next cell and repeat step A;

[0080] B. Remember the current cell position and try to shift one cell to the right within the current table. Repeat step A to determine whether it is an information item title. After excluding the right cell as an information item title, use the cell content value4 as the data value and mark the status of the right cell as processed.

[0081] C. If the data value cannot be found by shifting right in step B, shift one cell below the current cell and repeat step A to determine whether it is the information item title. After excluding the lower cell as the information item title, use the cell content value5 as the data value and mark the status of the lower cell as processed.

[0082] D. Repeat steps A, B, and C until the table in the target document is read. All extracted data is stored in the form of key-value pairs to form an information item data map structure object vMap.

[0083] E. Traverse each node of the data object copy dTemplate one by one, find the data value from the vMap object in step D by using the node title for full matching, save it to the value attribute of the node, and remove the used data from the vMap object;

[0084] F. Convert the remaining data in the vMap object into a format compatible with the data object copy and add it to the end of the dTemplate object.

[0085] It can be seen from the above description that the adaptive mode data item acquisition unit supports the status marking function.

[0086] P130: Structured data generation module: used to normalize structured data units and generate structured data objects. This includes a data normalization processing module, which is used to format data values ​​of specific types in data object copies.

[0087] For example, after the content of the document object is extracted, the data values ​​of specific types in the data object copy dTemplate are formatted and normalized, such as automatically adding the international area code based on the mobile phone number, and automatically adding "province, city, county, district" to the address.

[0088] Normalization uses a value mapping table to compare data values ​​in a data object copy with equivalent values ​​in the value mapping. If a match is found, the data value is replaced with the standard value. For example, if the standard value is "Male" and the equivalent value is "Male", then the gender data value "Male" in the data object copy will be normalized to the standard value "Male".

[0089] At this point, the conversion of unstructured table files into structured data is completed, that is, the data in the table files that meet certain specifications but cannot be directly structured are extracted, and a format that meets the structuring requirements in terms of data structure (such as database table format, XML format, JSON) and meets the industry standard requirements in terms of numerical values ​​(for example, the telephone number is 11 digits) is generated and stored in the database or saved as a file for subsequent business use.

[0090] Figure 3 A flow chart of the structured data conversion method is provided. As shown in the figure, converting an unstructured table file into structured data includes the following steps:

[0091] Step S300: Determine text rules. Text rules are feature constraints of the converted structured data, including a standard title word dictionary, a value mapping table, an information item repository, text feature relationships, a text data template, and a text data map. The text feature relationships and the text data map are in a one-to-one correspondence. The text data template is generated from the text data map.

[0092] The standard title word dictionary title-dictionary includes basic data units identified from the table files of all files, that is, the title content corresponding to the information items, including standard titles, which are characteristic titles selected from multiple titles with the same meaning, and also includes synonyms and abbreviations for expanding each title word.

[0093] The item library contains the data structures of all information items found in the tabular files of all dossiers, including attributes such as standard title, value data type, value range, and value validity verification rules. This includes a value map table, called a value map, which maps data types to value sets. This map is a value set that aggregates data types with fixed value ranges and all standard and equivalent values.

[0094] The text feature relationship refers to the text specifications of the file document materials, and extracts the unique text content features of each text type through features such as content paragraph position, text character content, and font format. It is used to establish a mapping relationship table feature-map between each social security text type and text content features. The text type docType can be reversely inferred through the text content features.

[0095] A text data map is a data relationship constructed using structured data formats (such as JSON and XML). The data map uses a tree structure, with the smallest unit being the information item. It defines the hierarchical relationship, standard title, title location, value location, data type, data length, and the order of each information item in the text. Each data map is associated one-to-one with a text type.

[0096] A text data template is a data map based on text content. By cutting out descriptive fields such as the title location, value location, and data length, and adding value data fields, a structured data format (such as JSON or XML) for the text content is established, and each data template is associated one-to-one with a text type.

[0097] Step S310: Read the specified unstructured table file, extract the text features of the table file, match the corresponding text data template and text data map according to the text features, and create a conversion data object copy;

[0098] In this step, the unstructured table file is extracted, the file content object is output, and a document content object file is obtained. The corresponding feature text is obtained and the text features corresponding to the document content object are determined. Then, the corresponding text data template and text data map are obtained through the text feature relationship, and a data object copy of the unstructured table file is generated. The data object copy is obtained by converting the data template file using a data parsing component (compatible with JSON or XML) to obtain a model object copy, which is used to store and update the reading results of the social security text data.

[0099] Among them, the feature text includes text character content, font, and font size; the text features combined with the mapping relationship table can reversely determine the text type to which the table file belongs;

[0100] Step S320: collecting information items from the converted data object copy and extracting data values ​​corresponding to the information items to form structured data;

[0101] Collect information items from the converted data object copy and extract the data values ​​corresponding to the information items to form structured data. The collection process includes two types: strong verification mode data item collection type and adaptive mode data item collection type:

[0102] Among them, the data source of the strong verification mode data item collection type is the text data map corresponding to the unstructured table file; the collection method provided by the strong verification mode data item collection type is to traverse the information item nodes in the text data map one by one, and extract the information item title feature information and information item content information;

[0103] The data source supported by the adaptive mode data item collection type is the cells of the unstructured table file itself; the collection method it supports is to traverse the cells of all tables, extract the cell content, and match it with the information item title feature information and information item content information.

[0104] Step S330: Normalize the structured data and generate a structured data object.

[0105] In this step, the data values ​​of a specific type in the data object copy are formatted, including formatting and normalizing the data values ​​of a specific type in the data object copy dTemplate. The data values ​​in the data object copy are compared with the equivalent values ​​in the value mapping using the value mapping table. If a match is successful, the data values ​​are replaced with standard values.

[0106] Through the above steps, the conversion of an unstructured table file into structured data is completed.

[0107] In the present invention, the text feature definition modules such as data templates and data maps of the three-dimensional social security prevention and control system are utilized to batch process non-structured table files that meet the specifications, extract and convert data according to text features, and store structured and standardized data in the database for subsequent intelligence analysis services.

[0108] The above disclosures are only a few specific embodiments of the present invention. However, the present invention is not limited thereto. Any changes that can be conceived by those skilled in the art should fall within the scope of protection of the present invention.

Claims

1. A structured data conversion system, characterized in that: include: Text feature definition module: used to provide defined text rules, the text rules include text feature relationships, text data templates and text data maps; wherein the text feature relationship is a text specification, and the text content features unique to each text type are extracted through text features, and are used to establish a mapping relationship table between each text type and text content features; the text data template is a data map based on the text content, and a structured data form of the text content is established by cutting descriptive fields and adding value data fields, and each data template is associated with a text type one-to-one; the text data map is a data relationship constructed in a structured data form, and the data map uses a tree structure, with the smallest unit being an information item, and defines the hierarchical relationship, standard title, title location, value location, data type, data length, and the order between information items of each information item in the text; each data map is associated with a text type one-to-one; the text feature relationship corresponds one-to-one to the text data map; the text data template is generated by the text data map; Unstructured data preprocessing module: used to read the specified unstructured table file, extract the text features of the table file, match the corresponding text data template and text data map according to the text features, and create a conversion data object copy; A structured data conversion module is used to collect information items from the converted data object copy and extract the data values ​​corresponding to the information items to form a structured data unit; wherein the structured data conversion module includes a collection unit for extracting the data values ​​corresponding to the information items; a structured data generation module is used to normalize the structured data unit and generate a structured data object.

2. The structured data conversion system according to claim 1, characterized in that: The unstructured data preprocessing module includes: Text reader: used to extract unstructured table files and output file content objects; A text feature extraction unit is configured to obtain feature text of the unstructured table file and determine text features corresponding to the document content object; wherein the feature text includes text character content, font, and font size; A data object copy creating unit is configured to obtain a corresponding text data template and a text data map according to the text feature, and generate a data object copy of the unstructured table file.

3. The structured data conversion system according to claim 2, characterized in that: The acquisition unit includes a strong verification mode data item acquisition unit; Wherein, the data source of the strong verification mode data item acquisition unit is the text data map corresponding to the unstructured table file; The strong verification mode data item collection unit provides a collection method of traversing the information item nodes in the text data map one by one, and extracting the title feature information and content information of the information item nodes; The strong verification mode data item acquisition unit further includes a validity verifier.

4. The structured data conversion system according to claim 2, characterized in that: The acquisition unit further includes an adaptive mode data item acquisition unit; Wherein, the data source supported by the adaptive mode data item acquisition unit is the cell corresponding to the unstructured table file; The adaptive mode data item acquisition unit supports an acquisition method of traversing cells of all tables, extracting cell contents, and matching them to information item title feature information and information item content information.

5. The structured data conversion system according to claim 4, characterized in that: The adaptive mode data item acquisition unit supports the status marking function.

6. The structured data conversion system according to claim 1, characterized in that: The structured data generation module includes a data normalization processing module for formatting data values ​​of a specific type in the data object copy.

7. The structured data conversion system according to claim 1, characterized in that: The text rules also include a standard title word dictionary, a value mapping table, and an information item warehouse.

8. A method for converting structured data, characterized in that: include: Determining text rules includes determining text feature relationships, text data templates, and text data maps; wherein the text feature relationship is a text specification, and the text content features unique to each text type are extracted through text features, which are used to establish a mapping relationship table between each text type and text content features; the text data template is a data map based on the text content, and a structured data form of the text content is established by cutting descriptive fields and adding value data fields, and each data template is associated with a text type one-to-one; the text data map is a data relationship constructed in a structured data form, and the data map uses a tree structure, with the smallest unit being an information item, and defines the hierarchical relationship, standard title, title location, value location, data type, data length, and the order between information items of each information item in the text; each data map is associated with a text type one-to-one; the text feature relationship corresponds one-to-one to the text data map; and the text data template is generated by the text data map; Reading a specified unstructured table file, extracting text features of the table file, matching a corresponding text data template and a text data map according to the text features, and creating a conversion data object copy; Collecting information items from the converted data object copy, extracting data values ​​corresponding to the information items to form structured data; The structured data is normalized to generate a structured data object.

9. The method for converting structured data according to claim 8, wherein: The extracting of text features of the table file refers to obtaining feature text of the unstructured table file and determining text features corresponding to the document content object; wherein the feature text includes text character content, font, and font size.

10. The method for converting structured data according to claim 8, wherein: The collecting of information items from the converted data object copy and extracting data values ​​corresponding to the information items to form structured data includes: strong verification mode data item collection type and adaptive mode data item collection type: The data source of the strong verification mode data item collection type is the text data map corresponding to the unstructured table file; the collection method provided by the strong verification mode data item collection type is to traverse the information item nodes in the text data map one by one to extract the information item title feature information and the information item content information; The data source supported by the adaptive mode data item collection type is the cell corresponding to the unstructured table file; the collection method supported by the adaptive mode data item collection type is to traverse all cells of the table, extract cell content, and match it with information item title feature information and information item content information.

Images