A metadata identification method, storage medium and system based on fragmented documents

Abstract

The present invention provides a metadata identification method, storage medium and system based on fragmented documents. The method includes the following steps: reading a preset document to obtain an element body iterator containing all elements in the preset document; traversing the element body iterator to identify titles and tables among all elements contained therein, and using a data structure of a title stack to establish a mapping relationship between titles and tables; identifying relational metadata for the titles and tables with the established mapping relationship based on a preset meta-model; identifying relational metadata between tables and fields, and generating a metadata update script based on the relational metadata to update metadata information, so that the updated metadata information includes the relationship between titles and tables, making it complete.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a metadata identification method, storage medium and system based on fragmented documents. Background Art

[0002] Currently, to identify metadata in docx documents and update the metadata information in the metadata management system accordingly, the XWPFDocument interface in Apache POI (Apache POI is a free, open-source, cross-platform Java API written in Java. Apache POI provides an API for Java programs to read and write Microsoft Office format documents) is usually used to read all elements in the docx document. Then, the metadata in all elements is identified using a preset metamodel. Finally, the metadata information in the metadata management system is updated based on the identified metadata.

[0003] docx documents usually contain tables and titles used to describe the tables, that is, documents usually contain field metadata corresponding to the table content and table metadata corresponding to the title used to describe the table. For a table, the field metadata corresponding to its table content and the table metadata corresponding to the title used to describe the table are mutually associated. However, this association relationship has not been established in the process of identifying document metadata, resulting in incomplete metadata information updates in the metadata management system. Summary of the Invention

[0004] The technical problem to be solved by the present invention is how to establish an association relationship between field metadata and table metadata during the process of identifying document metadata.

[0005] To solve the above technical problems, the present invention provides a metadata identification method based on fragmented documents, comprising the following steps:

[0006] A. Read the preset document and obtain an element body iterator containing all elements in the preset document;

[0007] B. Traverse the element body iterator to identify the titles and tables among all the elements it contains, and use the data structure of the title stack to build a mapping relationship between the titles and the tables, specifically including the following steps B1 to B8:

[0008] ——B1. Create the current title and initialize its outline level to 0;

[0009] ——B2. Initialize the title stack;

[0010] ——B3. Perform element traversal access on the element body iterator. If the element is accessed, proceed to step B4 below. If the element is not accessed, the process ends.

[0011] ——B4. Determine whether the current element instance is a table. If the current element instance is a table, construct a mapping relationship between the current title and the table, then add the current title to the title stack, and then return to step B3. If the current element instance is not a table, proceed to step B5 below;

[0012] ——B5. Determine whether the current element instance is a title. If the current element instance is a title, proceed to step B6 below. If the current element instance is not a title, return to step B3 above.

[0013] B6. Get the outline level of the current element and determine whether the outline level of the current element is greater than the outline level of the current title. If the outline level of the current element is greater than the outline level of the current title, proceed to step B7 below. If the outline level of the current element is not greater than the outline level of the current title, proceed to step B8 below.

[0014] B7. Add the current title to the title stack and use the current element as the new current title, then return to step B3 above.

[0015] B8. Pop the top title from the title stack and then determine whether the outline level of the current element is greater than the outline level of the newly popped title. If the outline level of the current element is greater than the outline level of the newly popped title, the popped title is placed back into the title stack, with the current element as the new current title. The process then returns to step B3. If the outline level of the current element is not greater than the outline level of the newly popped title, step B8 is repeated.

[0016] C. Based on the preset meta-model, the relationship metadata of the titles and tables with established mapping relationships is identified, specifically including the following steps C1 to C6:

[0017] C1. Traverse each title in the title stack and obtain a table with a mapping relationship between the title and the title.

[0018] ——C2. For the obtained table, extract the first row of table content as the title list;

[0019] C3. For each title list, calculate the text similarity between it and the field metadata in the preset field metamodel. Obtain the number of title lists that meet the preset text similarity level. Calculate the ratio of this number of title lists to the total number of title lists to generate a table confidence score. If the table confidence score exceeds a preset threshold, the current table is considered a metadata table. If the confidence score is not greater than the preset threshold, the current table is considered a non-metadata table.

[0020] C4. Obtaining the field metadata of the metadata table according to the title list of the metadata table;

[0021] - C5 obtains the title that has a mapping relationship with the metadata table, and uses a regular expression to parse the title into table metadata;

[0022] ——C6. Merge the table metadata obtained in step C5 and the field metadata obtained in step C4 to generate relationship metadata between the table and the field;

[0023] D. Identify the relational metadata of tables and fields, and generate metadata update scripts based on this to update metadata information.

[0024] Preferably, in step B2, a stack structure is first established, and then the stack structure is initialized to become an empty title stack.

[0025] Preferably, in step B4, the current element is first parsed according to the row and column format of the table to obtain the table content, and then a mapping relationship between the current title and the table content is constructed, thereby constructing a mapping relationship between the current title and the table.

[0026] Preferably, in step C, the data objects of the preset meta-model include tables and fields.

[0027] Preferably, in step C3, a text edit distance algorithm is used to calculate the text similarity between the title list and the field metadata in the preset field meta-model.

[0028] Preferably, in step C4, the field metadata includes name, age and gender, and in step C5, the table metadata includes the Chinese name and English name of the table.

[0029] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps in the metadata identification method described above when executed by a processor.

[0030] The present invention also provides a metadata identification system based on fragmented documents, comprising a computer-readable storage medium and a processor connected to each other, wherein the computer-readable storage medium is as described above.

[0031] The present invention has the following beneficial effects: after reading and obtaining an element body iterator containing all elements in a preset document, the element body iterator is traversed to identify the titles and tables therein, and then a data structure of a title stack is used to construct a mapping relationship between the titles and the tables, and then the relational metadata of the titles and tables with the mapping relationship constructed is identified based on a preset meta-model, and the field metadata belonging to the metadata table in all tables and the table metadata of the titles with the mapping relationship constructed with the metadata table are obtained, so that the table metadata and the field metadata are merged to generate the relational metadata of the table and the field, and then the relational metadata of the table and the field is identified and the metadata information in the metadata management system is updated accordingly to make it perfect. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a flowchart of a metadata identification method based on fragmented documents;

[0033] Figure 2 It is a flowchart for building the mapping relationship between titles and tables;

[0034] Figure 3 is a schematic diagram of the docx document content in the embodiment;

[0035] Figure 4 It is a flowchart of identifying relational metadata for titles and tables with established mapping relationships. DETAILED DESCRIPTION

[0036] The present invention is further described in detail below in conjunction with specific embodiments.

[0037] This embodiment provides a metadata identification system based on fragmented documents. The system includes a computer-readable storage medium and a processor connected to each other. The computer-readable storage medium stores a computer program. When the computer program is executed by the processor, the system implements the following Figure 1 The metadata identification method based on fragmented documents shown in the figure specifically includes the following steps A, B, C, and D.

[0038] A. Read the preset document and obtain an element body iterator containing all elements in the preset document.

[0039] In this embodiment, the XWPFDocument interface in Apache POI is used to read the preset document. Apache POI is a free, open-source, cross-platform Java API written in Java. Apache POI provides an API interface for Java programs to read and write documents in Microsoft Office formats. In this embodiment, the preset document is a .docx document. Apache POI uses the XWPF module to read and write .docx documents. The core of the XWPF module is the XWPFDocument interface. One XWPFDocument interface corresponds to one .docx document and can be used to read or write .docx documents. The system uses the XWPFDocument interface in Apache POI to read a .docx document and obtains an element body iterator containing all elements in the .docx document. An element body iterator is a collection of all elements in the .docx document.

[0040] B. Traverse the element body iterator, identify the titles and tables among all the elements it contains, and use the data structure of the title stack to build a mapping relationship between the titles and the tables.

[0041] In this embodiment, the types of all elements in the element body iterator include titles, tables, and texts. In order to establish the association between the field metadata corresponding to the table content and the table metadata corresponding to the title used to describe the table, the system first traverses the element body iterator, identifies the titles and tables among all the elements it contains, and then uses the data structure of the title stack to construct the mapping relationship between the title and the table, that is, first fragment the document in elements, and then construct the mapping relationship between the title and the table based on the fragmented elements. Figure 2 As shown, it includes the following steps B1 to B8:

[0042] B1. Create the current heading and initialize its outline level to 0.

[0043] After obtaining the element body iterator, the system creates an empty root heading and initializes its outline level to 0. This root heading is then used as the current heading, making the current heading null and its outline level 0. It should be noted that each chapter in a docx document has an outline level, such as level 1, 2, or 3. Each element in a chapter has an outline level. When reading a docx document using the XWPFDocument interface in Apache POI, all elements in the docx document and their corresponding outline levels can be retrieved. The outline level of the current heading "null" is 0, which is lower than the outline levels of each element in the element body iterator.

[0044] B2. Initialize the title stack.

[0045] After creating the current title, the system creates a stack structure and initializes the stack structure to make it an empty title stack.

[0046] B3. Perform an element traversal on the element body iterator. If an element is found, proceed to step B4 below. If no element is found, terminate.

[0047] After establishing an empty current title and initializing the title stack, the system performs element traversal access on the element body iterator, that is, accesses each element in the element body iterator in sequence. If the element is accessed, it enters the following step B4 to analyze the current element. If the element is not accessed, it means that all elements of the element body iterator have been accessed, so the element traversal access ends.

[0048] B4. Determine whether the current element instance is a table. If the current element instance is a table, a mapping relationship is constructed between the current title and the table. The current title is then added to the title stack and the process returns to step B3. If the current element instance is not a table, the process proceeds to step B5.

[0049] When accessing an element, the system first determines whether the instance of the current element is a table based on the formatting style of the current element. If the current element instance is determined to be a table, a mapping relationship between the current title and the table must be established. Therefore, the current element is parsed according to the table's row and column format to obtain the table content. Then, a mapping relationship between the current title and the table content is established. That is, after the mapping relationship between the current title and the table is established, the current title is added to the title stack and the system returns to step B3 to continue traversing the element body iterator to access the next element. If the current element instance is determined not to be a table, there is no need to establish a mapping relationship between the current title and the current element. Therefore, the system proceeds to step B5 below to further analyze the current element. It should be noted that each element in a docx document has a corresponding formatting style, such as title, subtitle, body, caption, table, etc. When reading a docx document using the XWPFDocument interface in Apache POI, all elements of the docx document and the formatting styles corresponding to each element can be read. In this way, the system can determine whether the current element instance is a table based on the element's formatting style.

[0050] B5. Determine whether the current element instance is a title. If the current element instance is a title, proceed to step B6 below. If the current element instance is not a title, return to step B3 above.

[0051] After the system determines that the current element instance is not a table based on the format style of the current element, it then determines whether the current element instance is a title based on the format style of the current element. If it is determined that the current element instance is a title, it is necessary to determine whether the current element belongs to the chapter corresponding to the current title, so the system enters the following step B6 for specific judgment; if it is determined that the current element instance is not a title, it means that the current element belongs to the chapter corresponding to the current title, so the system returns to the above step B3 to continue to traverse the element body iterator, that is, access the next element.

[0052] B6. Obtain the outline level of the current element and determine whether the outline level of the current element is greater than the outline level of the current title. If the outline level of the current element is greater than the outline level of the current title, proceed to step B7 below. If the outline level of the current element is not greater than the outline level of the current title, proceed to step B8 below.

[0053] When the current element is determined to be a title, the system obtains the outline level of the current element and then determines whether the outline level of the current element is greater than the outline level of the current title to determine whether the current element belongs to the chapter corresponding to the current title. If the outline level of the current element is greater than the outline level of the current title, it means that the current element belongs to the chapter corresponding to the current title, and the title corresponding to the current element is more specific than the current title, and the process proceeds to step B7 below. If the outline level of the current element is not greater than the outline level of the current title, it means that the current element belongs to the chapter corresponding to the current title, and the title corresponding to the current element is more specific than the current title, and the process proceeds to step B8 below.

[0054] B7. Add the current title to the title stack and use the current element as the new current title, then return to step B3 above.

[0055] When it is determined that the outline level of the current element is greater than the outline level of the current title, it means that the current element belongs to the chapter corresponding to the current title and the title corresponding to the current element is more specific than the current title. Therefore, the system puts the current title into the title stack and uses the current element as the new current title. Then, the system returns to the above step B3 to continue traversing the element body iterator, that is, accessing the next element. In this way, if a table is accessed subsequently, a new mapping relationship between the current title and the table can be established to ensure that a mapping relationship is established between the table and a more specific title.

[0056] B8. Pop the top title from the title stack and then determine whether the outline level of the current element is greater than the outline level of the newly popped title. If the outline level of the current element is greater than the outline level of the newly popped title, the popped title is placed back into the title stack, with the current element as the new current title. The process then returns to step B3. If the outline level of the current element is not greater than the outline level of the newly popped title, step B8 is repeated.

[0057] When it is determined that the outline level of the current element is not greater than the outline level of the current title, it means that the current element does not belong to the chapter corresponding to the current title and the title corresponding to the current element is not more specific than the current title. Therefore, the top title in the title stack is popped out, and then it is determined whether the outline level of the current element is greater than the outline level of the newly popped title to determine whether the current element belongs to the chapter corresponding to the newly popped title. If it is determined that the outline level of the current element is greater than the outline level of the newly popped title, it means that the current element belongs to the chapter corresponding to the newly popped title and the title corresponding to the current element is more specific than the newly popped title. Therefore, the popped title is first put back into the title stack, and then the current element is used as the new current title. Then, the process returns to step B3 to continue traversing and accessing the element body iterator. If it is determined that the outline level of the current element is not greater than the outline level of the newly popped title, it means that the current element does not belong to the chapter corresponding to the newly popped title. Therefore, step B8 is repeated to continuously pop the top title in the title stack, and then it is determined whether the outline level of the current element is greater than the outline level of the newly popped title, until the chapter corresponding to the newly popped title to which the current element belongs is found. Then, the process returns to step B3 to continue traversing and accessing the element body iterator.

[0058] by Figure 3 Take the docx document shown as an example, which includes two chapters. Specifically: the outline level of the first chapter is 1, which corresponds to a parent title "Title 1" and contains two sub-chapters. The outline level of these two sub-chapters is 2, among which the first sub-chapters contains a sub-title "Title 2", a main text "Main Text 1" and a table "Table 1", and the second sub-chapters contains a sub-title "Title 3" and a main text "Main Text 2"; the outline level of the second chapter is 1, which corresponds to a parent title "Title 4" and contains a table "Table 2".

[0059] In this case, the system executes step A and uses the XWPFDocument interface in Apache POI to read the docx document, obtaining an element body iterator containing all elements in the docx document. The element body iterator contains eight elements, specifically, Heading 1, Heading 2, Body 1, Table 1, Heading 3, Body 2, Heading 4, and Table 2. Heading 1 and Heading 4 have an outline level of 1, while Heading 2, Body 1, Table 1, Heading 3, Body 2, and Table 2 have an outline level of 2. The system then executes step B1 to create an empty current title, null, and initialize its outline level to 0. Step B2 then initializes the stack structure to obtain an empty title stack.

[0060] The system then executes step B3 to traverse the element body iterator and obtains the first element as "Title 1". In this case, the system enters step B4 to determine that the first element is not a table, and then enters step B5 to determine that the first element is a title. Then, the system enters step B6 to obtain the outline level of the first element as 1. Since the outline level of the current title null is 0, it is determined that the outline level of the first element is greater than the outline level of the current title. Therefore, the system enters step B7 to add the current title null to the title stack, that is, the title at the top of the stack is "null". Then, the first element "Title 1" is used as the new current title, and then returns to step B3.

[0061] After returning to step B3, the system continues to traverse the element body iterator and obtains the second element as "Title 2". In this case, it enters step B4 to determine that the second element is not a table, and then enters step B5 to determine that the second element is a title, and then enters step B6 to obtain the outline level of the second element as 2. Since the outline level of the current title "Title 1" is 1, it is determined that the outline level of the second element is greater than the outline level of the current title, so it enters step B7 to put the current title "Title 1" into the title stack, that is, the top title of the stack becomes "Title 1", and then the second element "Title 2" is used as the new current title, and then returns to step B3.

[0062] After the system returns to step B3, it continues to traverse the element body iterator to obtain the third element "Text One". In this case, it enters step B4 to determine that the third element is not a table, and then enters step B5 to determine that the third element is not a title. Therefore, it returns to step B3 to continue to traverse the element body iterator to obtain the fourth element "Table One". In this case, it enters step B4 to determine that the fourth element is a table. Therefore, the fourth element is parsed according to the row and column format of the table to obtain the table content "Name", "Age", and "Gender". Then, a mapping relationship is constructed between the current title and the table, that is, "Title Two" corresponds to "Name", "Age" and "Gender" respectively, thereby constructing a mapping relationship between "Title Two" and "Table One". Then, the current title "Title Two" is added to the title stack, that is, the top title of the stack becomes "Title Two", and then returns to step B3.

[0063] After the system returns to step B3, it continues to traverse the element body iterator and obtains the fifth element as "Title 3". In this case, it enters step B4 to determine that the fifth element is not a table, and then enters step B5 to determine that the fifth element is a title. Then it enters step B6 to obtain the outline level of the fifth element as 2. Since the outline level of the current title "Title 2" is 2, it is determined that the outline level of the fifth element is not greater than the outline level of the current title. Therefore, it enters step B8 to pop the top title "Title 2" in the title stack. After "Title 2" is popped, the top title on the stack becomes "Title 1". Since the newly popped title The outline level of "Title 2" is 2, so it is judged that the outline level of the fifth element is not greater than the outline level of the newly popped title, so step B8 is repeated to pop the top title "Title 1" in the title stack. After "Title 1" is popped, the top title becomes "null". Since the outline level of the newly popped title "Title 1" is 1, it is judged that the outline level of the fifth element is greater than the outline level of the newly popped title, so the popped titles "Title 1" and "Title 2" are put back into the title stack, that is, the top title becomes "Title 2" at this time, and the fifth element "Title 3" is used as the new current title, and then returns to step B3.

[0064] After the system returns to step B3, it continues to traverse the element body iterator to obtain the sixth element as "Text 2". In this case, it enters step B4 to determine that the sixth element is not a table, and then enters step B5 to determine that the sixth element is not a title. Therefore, it returns to step B3 to continue to traverse the element body iterator to obtain the seventh element as "Title 4". In this case, it enters step B4 to determine that the seventh element is not a table, and then enters step B5 to determine that the seventh element is a title. Then it enters step B6 to obtain the outline level of the seventh element as 1. Since the outline level of the current title "Title 3" is 2, it is determined that the outline level of the seventh element is not greater than the outline level of the current title. Therefore, it enters step B8 to pop the top title "Title 2" in the title stack. After "Title 2" is popped, the top title on the stack becomes "Title 1". Since the newly popped title " The outline level of "Title 2" is 2, so it is judged that the outline level of the seventh element is not greater than the outline level of the newly popped title, so step B8 is repeated to pop the top title "Title 1" in the title stack. After "Title 1" is popped, the top title becomes "null". Since the outline level of the newly popped title "Title 1" is 1, it is judged that the outline level of the seventh element is not greater than the outline level of the newly popped title, so step B8 is repeated to pop the top title "null" in the title stack. Since the outline level of the newly popped title "null" is 0, it is judged that the outline level of the seventh element is greater than the outline level of the newly popped title, so the popped titles "null", "Title 1" and "Title 2" are put back into the title stack, that is, the top title becomes "Title 2" at this time, and the seventh element "Title 4" is used as the new current title, and then the process returns to step B3.

[0065] After the system returns to step B3, it continues to traverse the element body iterator and obtains the eighth element as "Table Two". In this way, it enters step B4 to determine that the eighth element is a table, so the fourth element is parsed according to the row and column format of the table to obtain the table content, and then the mapping relationship between the current title and the table is constructed, that is, the mapping relationship between "Title Four" and "Table Two" is constructed, and then the current title "Title Four" is added to the title stack, that is, the top title of the stack becomes "Title Four", and then returns to step B3. After the system returns to step B3, it continues to traverse the element body iterator and no element can be accessed, which means that all elements of the element body iterator have been accessed, so the element traversal access is ended.

[0066] In the pair Figure 3After the above element traversal, the docx document shown constructs the mapping relationship between "Title 2" and "Table 1", as well as the mapping relationship between "Title 4" and "Table 2". The title stack retains "Title 2" and "Title 4" that have a mapping relationship with the table, and retains "Title 1" and "null" that are the parent titles of "Title 2". "Title 3" is not retained. In other words, the title stack only retains the relevant titles that have a mapping relationship with the table, and does not retain other titles that are not related to the table, such as "Title 3".

[0067] C. Based on the preset meta-model, the relationship metadata of the titles and tables with mapping relationships are identified, as follows: Figure 3 As shown, it includes the following steps C1 to C6.

[0068] After constructing the mapping relationship between the title and the table, and making the title stack retain the relevant titles that have the mapping relationship with the table, the system performs relational metadata recognition on the titles and tables with the mapping relationship based on the preset meta-model.

[0069] It should be noted that the preset metamodel refers to a metamodel for global data structure types constructed by obtaining user-input attribute information and metamodel relationship information according to the Common Warehouse Metamodel (CWM) specification to define the metamodel.

[0070] The Common Warehouse Metamodel (CWM) is an interoperability standard developed by the Object Management Group (OMG). It defines a common language and exchange mechanism for metadata used in data warehouses and business analysis. Its specifications include: the Unified Modeling Language (UML) specification, which is an OMG modeling standard; the Meta Object Facility (MOF) specification, which is an OMG standard for metamodels and metadata libraries; and the XML Metadata Interchange (XMI) specification, which is an OMG standard for metadata exchange.

[0071] In the process of defining the metamodel, the metamodel layer can be expanded according to the above-mentioned specifications of the public warehouse metamodel, and unstructured data metamodels such as text, pictures, audio, and time series data and semi-structured data metamodels can be added on the basis of the structured data metamodel. That is, after obtaining the attribute information and metamodel relationship information input by the user, a metamodel for the global data structure type can be constructed. Among these multiple metamodels are structured data metamodel, unstructured data metamodel and semi-structured data metamodel.

[0072] The main data objects of the structured data metamodel include systems, databases, tables, fields, jobs, indicators, etc. Different objects form metamodels with different model attributes (also called metamodel elements), and metamodel attributes describe the metadata concepts of different objects. The main data objects of the unstructured data metamodel include documents, pictures, videos, audio, etc. Different objects form metamodels with different model attributes, such as document metamodels and picture metamodels. The main data objects of the semi-structured data metamodel are time series data generated by IoT devices (Internet of Things devices). Different metamodels are formed by describing different time series data objects.

[0073] The preset meta-model in this embodiment mainly performs relational metadata recognition on titles and tables with constructed mapping relationships, that is, the data objects of the preset meta-model are tables and fields.

[0074] C1. Traverse and access each title in the title stack, and obtain a table with a mapping relationship established with the accessed title.

[0075] After obtaining the preset metamodel, the system traverses each title in the title stack and then retrieves the table mapped to each title. It should be noted that the titles in the title stack that are mapped to tables are "Title 2" and "Title 4," while "null" and "Title 1" are not directly mapped to tables. Therefore, only "Table 1" mapped to "Title 2" and "Table 2" mapped to "Title 4" are retrieved.

[0076] C2. Extract the first row of the obtained table as the title list;

[0077] Take the obtained "Table 1" as an example, extract the first row of table content of "Table 1" as the title list, see Figure 2 The first row of "Table 1" contains three items, namely "Name", "Age" and "Gender". In this way, the three title lists of "Name", "Age" and "Gender" are extracted.

[0078] C3. For each title list, calculate the text similarity between it and the field metadata in the preset meta-model, obtain the number of title lists that have passed and whose text similarity reaches the preset level, calculate the ratio of the number of title lists that have passed to the total number of title lists to generate the table confidence. If the table confidence is greater than the preset threshold, the current table is judged as a metadata table. If the confidence is not greater than the preset threshold, the current table is judged as a non-metadata table.

[0079] After extracting the title list of "Table 1", the system first uses the text edit distance algorithm to calculate the text similarity between each title list and the field metadata in the preset meta-model. Specifically, the preset meta-model contains fields that can be summarized as metadata, namely field metadata. For example, "name", "age", "gender", "height", "weight" and so on are all metadata that can be used to describe people, so they are summarized as field metadata, that is, the preset meta-model contains field metadata such as "name", "age", "gender", "height", "weight", etc. Therefore, the system first uses the text edit distance algorithm to calculate the text similarity between each title list "name", "age" and "gender" and the field metadata. In this way, the text similarity between the title list "name" and the field metadata "name" is 100%, the text similarity between the title list "age" and the field metadata "age" is 100%, and the text similarity between the title list "gender" and the field metadata "gender" is 100%; then, the system obtains The system calculates the number of title lists that have passed through and have reached a preset level of text similarity (e.g., 80%), and then calculates the ratio of the number of title lists that have passed through to the total number of title lists to generate a table confidence. It then determines whether the table confidence is greater than a preset threshold (60%). If the table confidence is greater than the preset threshold, the current table is determined to be a metadata table. If the confidence is not greater than the preset threshold, the table currently being traversed is determined to be a non-metadata table. Specifically, the text similarity between the three title lists and the field metadata is all 100%, which reaches the preset level. Therefore, the system calculates that the ratio of the number of title lists that have passed through to the total number of title lists is 100%, thereby generating a table confidence of 100%. It then determines that the table confidence is greater than the preset threshold, which means that the similarity between the table content of "Table 1" and the field metadata meets the standard, so the current table "Table 1" is determined to be a metadata table. If the table confidence is not greater than the preset threshold, it means that the similarity between the table content of the current table and the field metadata does not meet the standard, so the current table is determined to be a non-metadata table.

[0080] In this embodiment, the specific process of determining whether "Table 2" is a metadata table is the same as that of determining "Table 1". The result obtained is that "Table 2" is a non-metadata table, and the determination process will not be repeated here.

[0081] C4. Get the metadata field metadata of the metadata table according to the list of each title of the metadata table;

[0082] As can be seen from the above, the data objects of the preset meta-model are tables and fields. Therefore, after determining that "Table One" is a metadata table, the system can obtain the field metadata of the metadata table as "Name", "Age", and "Gender" based on the title lists "Name", "Age", and "Gender" of the metadata table "Table One".

[0083] C5. Obtain the title that has a mapping relationship with the metadata table, and use a regular expression to parse the title into table metadata.

[0084] As can be seen above, the title "Title 2" is mapped to the metadata table "Table 1". Therefore, the system obtains "Title 2" that is mapped to "Table 1" and then uses regular expressions to parse the title into table metadata. For example, if the content of Title 2 is "Student Information Sheet", then the content of Title 2 is used to describe the Chinese and English names of Table 1. Since the data objects of the preset metamodel are tables and fields, the regular expression can be used to parse Title 2 into two table metadata, namely "Student Information Sheet" and "StudentInformationSheet".

[0085] It's important to note that a regular expression (often abbreviated as regex, regexp, or RE in code) is a text pattern that includes common characters (such as letters from a to z) and special characters (called metacharacters). It's a concept from computer science. A regular expression uses a single string to describe and match a series of strings that match a certain syntactic rule. It's typically used to search for and replace text that matches a certain pattern (rule).

[0086] C6. Merge the table metadata obtained in step C5 and the field metadata obtained in step C4 to generate relational metadata for the table and field.

[0087] After executing step C5 to obtain table metadata and executing step C4 to obtain field metadata, the system merges all field metadata with one of the table metadata to generate the relational metadata of the table and fields. For example, by selecting all field metadata and the table metadata "Student Information Sheet" and merging them, you can obtain the relational metadata of the table and fields including "Student Information Sheet", "Name", "Age" and "Gender".

[0088] D. Identify the relational metadata of tables and fields, and generate metadata update scripts based on this to update metadata information.

[0089] After obtaining the relational metadata of the table and fields including "Student Information Sheet", "Name", "Age" and "Gender", the system identifies the relational metadata of the table and fields and generates a corresponding metadata update script, and then uses the metadata update script to update the metadata information in the metadata management system. In this way, the updated metadata information includes the relationship between the title and the table, making it complete.

[0090] The above is only an embodiment of the present invention and does not limit the scope of patent protection. Those skilled in the art can make non-substantial changes or substitutions based on the present invention and still fall within the scope of patent protection.

Claims

1. A metadata identification method based on fragmented documents, characterized in that: The steps include: A. Read the preset document and obtain an element body iterator containing all elements in the preset document; B. Traverse the element body iterator to identify the titles and tables among all the elements it contains, and use the data structure of the title stack to build a mapping relationship between the titles and the tables, specifically including the following steps B1 to B8: ——B1. Create the current title and initialize its outline level to 0; ——B2. Initialize the title stack; ——B3. Perform element traversal access on the element body iterator. If the element is accessed, proceed to step B4 below. If the element is not accessed, the process ends. ——B4. Determine whether the current element instance is a table. If the current element instance is a table, construct a mapping relationship between the current title and the table, then add the current title to the title stack, and then return to step B3. If the current element instance is not a table, proceed to step B5 below; ——B5. Determine whether the current element instance is a title. If the current element instance is a title, proceed to step B6 below. If the current element instance is not a title, return to step B3 above. B6. Get the outline level of the current element and determine whether the outline level of the current element is greater than the outline level of the current title. If the outline level of the current element is greater than the outline level of the current title, proceed to step B7 below. If the outline level of the current element is not greater than the outline level of the current title, proceed to step B8 below. B7. Add the current title to the title stack and use the current element as the new current title, then return to step B3. B8. Pop the top title from the title stack and determine whether the outline level of the current element is greater than the outline level of the newly popped title. If the outline level of the current element is greater than the outline level of the newly popped title, put the popped title back into the title stack, with the current element as the new current title. The process then returns to step B3. If the outline level of the current element is not greater than the outline level of the newly popped title, repeat step B8. C. Based on the preset meta-model, the relationship metadata of the titles and tables with established mapping relationships is identified, specifically including the following steps C1 to C6: C1. Traverse each title in the title stack and obtain a table with a mapping relationship between the title and the title. ——C2. For the obtained table, extract the first row of table content as the title list; C3. For each title list, calculate the text similarity between it and the field metadata in the preset field metamodel. Obtain the number of title lists that meet the preset text similarity level. Calculate the ratio of this number of title lists to the total number of title lists to generate a table confidence score. If the table confidence score exceeds a preset threshold, the current table is considered a metadata table. If the confidence score is not greater than the preset threshold, the current table is considered a non-metadata table. C4. Obtaining the field metadata of the metadata table according to the title list of the metadata table; - C5 obtains the title that has a mapping relationship with the metadata table, and uses a regular expression to parse the title into table metadata; ——C6. Merge the table metadata obtained in step C5 and the field metadata obtained in step C4 to generate relationship metadata between the table and the field; D. Identify the relational metadata of tables and fields, and generate metadata update scripts based on this to update metadata information.

2. The metadata identification method based on fragmented documents according to claim 1 is characterized in that: In step B2, a stack structure is first established, and then the stack structure is initialized to become an empty title stack.

3. The metadata identification method based on fragmented documents according to claim 1 is characterized in that: In step B4, the current element is first parsed according to the row and column format of the table to obtain the table content, and then a mapping relationship between the current title and the table content is constructed, thereby constructing a mapping relationship between the current title and the table.

4. The metadata identification method based on fragmented documents according to claim 1 is characterized in that: In the step C, the data objects of the preset meta-model include tables and fields.

5. The metadata identification method based on fragmented documents according to claim 1 is characterized in that: In step C3, a text edit distance algorithm is used to calculate the text similarity between the title list and the field metadata in the preset field meta-model.

6. The metadata identification method based on fragmented documents according to claim 1 is characterized in that: In step C4, the field metadata includes name, age and gender. In step C5, the table metadata includes the Chinese name and English name of the table.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the metadata identification method according to any one of claims 1 to 6 are implemented.

8. A metadata identification system based on fragmented documents, comprising a computer-readable storage medium and a processor connected to each other, characterized in that: The computer-readable storage medium as claimed in claim 7.

Images