Document automatic cataloguing method and system based on knowledge graph

Abstract

The invention relates to the technical field of automatic document cataloguing methods, in particular to an automatic document cataloguing method and system based on a knowledge graph. According to the method, a directory structure is automatically generated and documents are associated by constructing a knowledge graph, defining a directory construction scheme (based on a directory structure semantic description script DSSDS) and utilizing semantic reasoning and association capabilities of the knowledge graph. The method specifically comprises the following steps: constructing a knowledge graph: forming a semantic network through entity recognition and relation extraction; a directory construction scheme is defined, wherein directory levels, node constraint conditions and document screening conditions are designed based on dimension analysis of the knowledge graph; script analysis and execution: according to the DSSDS script, generating a directory structure and associating a document; the pre-processing comprises format conversion, text cleaning, content analysis and the like; the method can improve cataloguing accuracy and efficiency, reduce labor cost and flexibly adapt to business changes.

Description

Technical Field

[0001] The present invention relates to the technical field of automatic document cataloging methods, and in particular to a knowledge graph-based automatic document cataloging method and system. Background Art

[0002] In the era of information explosion, various industries have accumulated massive amounts of documents. Whether it is a business, an organization or an individual, they all face the challenge of how to efficiently catalog and manage these documents so that they can be quickly and accurately retrieved and utilized.

[0003] Document cataloging refers to defining a directory structure based on business needs and management requirements, and associating documents with specific directory nodes in the directory structure.

[0004] Although text-based automatic classification technology has been introduced in the cataloging of document materials, the system cannot truly understand the internal logic of the directory structure, nor can it truly realize the automatic construction of the directory structure on demand. Not only does the change of the directory structure require manual intervention, but in order to adapt to the change of the directory structure, manual intervention and retraining of the classification model are still required. As a result, the cataloging management and maintenance of document materials are too complicated and difficult to expand flexibly. It is not only time-consuming, labor-intensive and inefficient, but also prone to errors and inconsistencies. It is difficult to meet document management needs and it is difficult to quickly adapt to business changes. It is even more difficult to cope with a large number of document materials.

[0005] With the development of artificial intelligence, knowledge graphs, and natural language processing technologies, efficient and adaptive automatic cataloging of documents has become possible. By forming a knowledge graph through knowledge governance and assigning metadata such as classification and keywords to documents, users can flexibly define and execute directory construction schemes (i.e., document cataloging rules) based on dimensional combinations according to business needs and management requirements. This allows for the rapid and accurate generation of directory structures, the establishment of associations between documents and directory nodes (also known as populating the directory structure), and the generation of directory views (also known as document directory association views), enabling automatic cataloging of documents. This provides strong support for convenient document retrieval and utilization, and meets users' needs for efficient management of massive amounts of documents.

[0006] While it's currently possible to use automatic text classification technology to associate documents with directory nodes based on a pre-designed directory structure, the main problem is that editing and maintaining the directory structure itself still requires manual effort, which is not only time-consuming and error-prone, but also lacks the ability to flexibly adapt to changes in the directory structure because the system doesn't understand the underlying logic of the directory structure. This requires manual intervention and a complex process to adjust the automatic classification mechanism to accommodate changes in the directory structure and classification standards.

[0007] For example, a company builds a directory to manage project documents according to the structure of "project name-project phase-document type". When developing a new software project, it is initially divided into four phases: requirements analysis, design, development, and testing. As the project progresses, it is found that the testing phase needs to be subdivided into three sub-phases: unit testing, integration testing, and system testing, and the directory structure changes accordingly. However, the automatic classification system cannot recognize this change, and documents originally classified by "test phase" cannot be automatically classified into the new sub-directory. The company can only assign dedicated personnel to check the documents in the testing phase one by one, and manually move them to the corresponding unit test, integration test, or system test sub-directory based on the document content. In addition, the automatic classification rules need to be reconfigured and debugged to ensure that the subsequently added test documents can be correctly classified.

[0008] For example, a local government department managed documents by categorizing them solely by the issuing unit, such as the Finance Bureau and the Education Bureau. For some jointly issued documents, they only categorized them by the primary issuing unit, without comprehensively considering metadata features such as the document's subject, document number, and confidentiality level. When searching for jointly issued documents with a specific subject and confidentiality level, staff struggled to quickly locate the document using the existing classification mechanism.

[0009] In view of the problems in existing document cataloging methods, such as manual reliance on directory structure maintenance, inability to flexibly adapt to changes, and low classification accuracy, a flexible and efficient document automatic cataloging method is urgently needed. Summary of the Invention

[0010] The problem to be solved by the present invention is to provide a document automatic cataloging method and system based on knowledge graph. By defining a directory construction scheme and utilizing the semantic reasoning and knowledge association capabilities of the knowledge graph, the directory structure can be automatically generated and the document data can be automatically associated with the corresponding directory nodes.

[0011] In view of the shortcomings of the existing technology, the present invention solves the technical problems thereof by adopting a technical solution: a method for automatic document cataloging based on a knowledge graph, comprising the following steps:

[0012] S1: Build a knowledge graph: Through entity recognition and relationship extraction, the document topic-related entities and their semantic relationships are stored as a knowledge graph;

[0013] S2: Define the directory construction plan: Based on the dimensional analysis of the knowledge graph, design the directory hierarchy and define the node constraints and document screening conditions; the directory construction plan is written using the Directory Structure Semantic Description Script (DSSDS) specification;

[0014] S3: Parse and execute the directory construction plan: Generate the directory structure according to the DSSDS script, and associate the documents to the corresponding directory nodes through knowledge graph semantic reasoning;

[0015] S4: Preprocessing of new documents: Perform format conversion, text cleaning, content analysis on new documents, and extract key entities and relationships to match directory screening conditions.

[0016] Preferably, the Directory Structure Semantic Description Script (DSSDS) specification in S2 includes: the directory expression consists of node name, node constraints, and document screening conditions; the node constraints define the semantic relationship between directory levels through constraint clauses connected by logical operators; the document screening conditions define the conditions for documents to be included in the directory nodes through logical expressions.

[0017] Preferably, the node constraint condition is formed by connecting one or more constraint clauses through logical operators, and the constraint clauses are in the form of functions or comparison conditions.

[0018] Preferably, the new document preprocessing in S4 includes:

[0019] (a) Convert documents in different formats into a unified text format;

[0020] (b) Remove noise data through text cleaning and perform normalization;

[0021] (c) According to the directory construction plan, the required document content metadata, such as product name and design parameters, is extracted, and then substituted into the document screening conditions of the directory construction plan to determine which directory node the document should be associated with. The extracted results are saved in the knowledge graph so that when the directory construction plan is subsequently executed, the knowledge graph can be used to reason and supplement the missing information, or the user can be prompted to supplement the information.

[0022] Preferably, the directory construction scheme in S2 supports reusing defined directory generation schemes and combining multiple node expressions through logical operators, wherein the logical operators include at least one of AND, OR, and NOT.

[0023] Preferably, in the node constraint conditions described in S2, if the node name does not use double quotes, the actual directory name is dynamically generated based on the node constraint conditions; if the node name uses double quotes, the node name is directly used as the directory name.

[0024] Preferably, constructing the knowledge graph in S1 includes the following specific steps:

[0025] (i) Collecting internal and external knowledge data related to the document topic from multiple data sources;

[0026] (ii) extracting entities through named entity recognition algorithms and determining semantic relationships between entities through relation extraction algorithms;

[0027] (iii) Entities are treated as nodes and semantic relationships as edges, and stored in graph form to form a knowledge graph.

[0028] A document automatic cataloging system based on knowledge graph, including:

[0029] Data layer: used to store knowledge graphs, document data, and directory construction schemes;

[0030] Processing layer: used to implement knowledge graph construction, directory construction plan analysis, document preprocessing and association;

[0031] Interaction layer: used to provide directory construction scheme editing interface and document management functions;

[0032] Interface layer: used to connect to external systems and provide RESTful API interfaces.

[0033] Preferably, the processing layer comprises:

[0034] Document format conversion module, used to convert documents in PDF, Word, and Excel formats into a unified text format;

[0035] Text cleaning module, used to remove noise data from documents and perform normalization processing;

[0036] Script parser, used to parse the Directory Structure Semantic Description Script (DSSDS) and generate the directory structure.

[0037] Preferably, the interactive layer supports users to dynamically adjust the directory construction plan, including editing node constraints, document screening conditions and directory hierarchical structure, and update the knowledge graph and directory view in real time.

[0038] The beneficial effects of the present invention are as follows:

[0039] (1) Improving cataloging accuracy

[0040] The semantic reasoning and knowledge association capabilities of the knowledge graph enable a deep understanding of the document's content and semantic relationships, accurately determining the directory into which the document should be placed, and avoiding the problems of misclassification and omission caused by insufficient semantic understanding in traditional cataloging methods. For example, for a complex document involving knowledge in multiple fields, traditional keyword matching or rule-based cataloging methods may not be able to accurately classify it. However, the present invention, through analysis of the knowledge graph, can accurately identify the entities and relationships in the document and place it in the appropriate directory.

[0041] (2) Reducing labor costs

[0042] This system automates document cataloging, significantly reducing the manpower and time required for manual cataloging. Users simply submit their documents, and the system automatically completes the cataloging process, improving cataloging efficiency and reducing operational costs. For example, in an enterprise with a large volume of documents, the system can save significant labor costs, freeing catalogers from tedious manual cataloging tasks and allowing them to focus on more valuable work.

[0043] (3) Ensuring cataloging consistency

[0044] Cataloging is based on a unified directory construction scheme and directory structure semantic description script, avoiding the inconsistent cataloging results caused by subjectivity in manual cataloging. Regardless of time and location, the system catalogs the same type of documents according to established rules and logic, ensuring consistency and standardization of cataloging results. For example, different catalogers may use different directory classification methods when cataloging the same type of contract documents due to differences in personal understanding. However, the system of the present invention ensures that all contract documents are classified into the corresponding directory according to unified rules.

[0045] (4) Quickly adapt to business changes: When business needs or management requirements change, users only need to update the knowledge graph based on the existing document data, and then create a new directory structure plan, or edit the existing directory structure plan, and then re-execute the directory structure plan on the existing document data. A new directory structure that meets the requirements can be regenerated, and the document data can be associated with the directory node to form a document classification view (document directory association view). BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a flow chart of the method of the present invention;

[0047] Figure 2 It is a system block diagram of the present invention;

[0048] Figure 3 It is a schematic diagram of the system workflow of the present invention. DETAILED DESCRIPTION

[0049] The present invention will be described in further detail below with reference to the accompanying drawings and specific embodiments. The embodiments of the present invention are provided for purposes of illustration and description and are not intended to be exhaustive or to limit the invention to the disclosed forms. Many modifications and variations will be apparent to those skilled in the art. The embodiments are chosen and described to better illustrate the principles of the invention and its practical application, and to enable those skilled in the art to understand the invention and design various embodiments with various modifications suitable for specific applications.

[0050] In order to solve the above technical problems, the present invention provides a document automatic cataloging method based on knowledge graph, such as Figure 1 As shown, the following steps are included:

[0051] S1: Build a knowledge graph: Through entity recognition and relationship extraction, the document topic-related entities and their semantic relationships are stored as a knowledge graph; specifically, data collection → entity recognition → relationship extraction → storage graph;

[0052] Building a knowledge graph includes specific steps:

[0053] (i) Collecting internal and external knowledge data related to the document topic from multiple data sources;

[0054] (ii) extracting entities through named entity recognition algorithms and determining semantic relationships between entities through relation extraction algorithms;

[0055] (iii) Entities are treated as nodes and semantic relationships as edges, and stored in graph form to form a knowledge graph.

[0056] S2: Define the directory construction plan: Specifically, the steps are dimensional analysis → directory hierarchy design → DSSDS script writing. Based on the dimensional analysis of the knowledge graph, the directory hierarchy is designed, and node constraints and document screening conditions are defined. The directory construction plan is written using the Directory Structure Semantic Description Script (DSSDS) specification. The DSSDS specification includes: a directory expression consists of a node name, node constraints, and document screening conditions. Node constraints define the semantic relationship between directory hierarchies through constraint clauses connected by logical operators. Document screening conditions define the conditions for document inclusion in directory nodes through logical expressions. Node constraints consist of one or more constraint clauses connected by logical operators. These constraint clauses can be in the form of functions, such as "contains (topic, product line)", or comparison conditions, such as "document type == contract".

[0057] The directory construction scheme supports reusing pre-defined directory generation schemes and combining multiple node expressions using logical operators, including at least one of AND, OR, and NOT. If the node name in the node constraint is not enclosed in double quotes, the actual directory name is dynamically generated based on the node constraint. If the node name is enclosed in double quotes, the node name is used directly as the directory name.

[0058] S3: Parse and execute the directory construction plan: Specifically, parse the script → generate the directory structure → associate documents; generate the directory structure according to the DSSDS script, and associate the documents to the corresponding directory nodes through knowledge graph semantic reasoning;

[0059] S4: Preprocessing of new documents: specifically format conversion → text cleaning → content analysis → knowledge graph reasoning; perform format conversion, text cleaning, and content analysis on new documents, and extract key entities and relationships to match directory screening conditions.

[0060] New document preprocessing includes:

[0061] (a) Convert documents in different formats into a unified text format;

[0062] (b) Remove noise data through text cleaning and perform normalization;

[0063] (c) According to the directory construction plan, the required document content metadata, such as product name, design parameters, etc., is extracted, and then substituted into the document screening conditions of the directory construction plan to determine which directory node the document should be associated with. The extracted results are saved in the knowledge graph so that when the directory construction plan is subsequently executed, the knowledge graph can be used to reason and supplement the missing information, or the user can be prompted to supplement the information.

[0064] To support the implementation of the above method, the present invention also provides a document automatic cataloging system based on a knowledge graph, including: a data layer, a processing layer, an interaction layer, and an interface layer. The data layer is used to store the knowledge graph, document data, and directory construction schemes; the processing layer includes: a document format conversion module, a text cleaning module, and a script parser. The document format conversion module is used to convert documents in PDF, Word, and Excel formats into a unified text format; the text cleaning module is used to remove noise data in the document and perform normalization processing; the script parser is used to parse the directory structure semantic description script (DSSDS) and generate the directory structure. The processing layer is used to implement knowledge graph construction, directory construction scheme parsing, document preprocessing and association; the interaction layer is used to provide a directory construction scheme editing interface and document management functions; the interface layer is used to connect to external systems and provide a RESTful API interface. The interaction layer supports users to dynamically adjust the directory construction scheme, including editing node constraints, document screening conditions, and directory hierarchy structure, and updates the knowledge graph and directory view in real time.

[0065] The following example illustrates the above method in detail:

[0066] (1) Building a knowledge graph

[0067] 1. Data Collection: Collect internal and external knowledge data related to the document topic from multiple data sources. External data includes but is not limited to industry standards, professional literature, and domain knowledge bases. Internal data includes but is not limited to business process specifications, product manuals, market research reports, etc.

[0068] 2. Entity Recognition and Relationship Extraction: Natural language processing (NLP) techniques are used to perform entity recognition and relationship extraction on the collected data. Named entity recognition (NER) algorithms are used to identify entities in the data, such as people, organizations, concepts, and events. Relationship extraction algorithms are then used to determine semantic relationships between entities, such as "belongs to," "contains," and "associated with." For example, in a product manual, entities such as "product name," "components," and "function" are identified, as well as relationships such as "product contains components" and "components implement functions."

[0069] 3. Graph Storage: Identified entities and relationships are stored in a graph format to build a knowledge graph. Each entity is represented as a node, and the relationships between entities are represented as edges, forming a semantic network. For example, entities such as "product," "component," and "function" are represented as nodes, connected by edges such as "include" and "implement," to build a product-related knowledge graph.

[0070] (2) Define the directory structure plan

[0071] 1. Knowledge Graph-Based Dimensional Analysis: Structural analysis is performed on the constructed knowledge graph. Based on business needs and management requirements, key entity categories and their interrelationships are identified as candidate classification dimensions. For example, within an enterprise knowledge graph, analysis reveals entity categories such as "business process," "business module," and "document type," as well as relationships between them, such as "business process contains business module" and "business module is associated with document type."

[0072] 2. Directory Hierarchy Design: Based on business needs and management requirements, select an appropriate combination of classification dimensions to construct the desired directory structure. Specifically, key entity categories serve as directory hierarchies, and relationship paths serve as the basis for linking directory hierarchies. For example, design a first-level directory for "Business Process," a second-level directory for "Business Module," and a third-level directory for "Document Type," creating a clearly defined directory structure.

[0073] 3. Node Constraints: The directory context is formed by the upper and lower directories in the directory structure. Defining node constraints for each directory level determines which entities are allowed to appear as directory nodes within a specific directory context.

[0074] 4. Document Filtering Conditions: Indicate the conditions that a document must meet to be categorized into the corresponding directory node. Document filtering conditions can be formulated based on the matching relationship between entities in the document and entities in the knowledge graph, as well as the document's metadata. For example, if the primary entity contained in a document belongs to a "business module" entity in the knowledge graph, the document will be categorized into the corresponding "business module" directory.

[0075] 5. Node constraints and document screening conditions are logical expressions connected by logical operators such as AND, NOT, and OR. Based on these logical expressions, a variety of classification methods can be combined, such as keywords, machine learning, neural networks, and topics, and combined with knowledge graphs to achieve a flexible, scalable, and adaptable cataloging and classification mechanism.

[0076] You can define a directory structure by writing a directory structure plan. This should follow the Directory Structure Semantic Description Script (DSSDS). See the following sections for details.

[0077] (3) Directory Structure Semantic Description Script (DSSDS) Specification

[0078] Directory construction plans can be written in accordance with the Directory Structure Semantic Description Script specification. The Directory Structure Semantic Description Script (DSSDS) is a scripting language that describes the inherent logical semantics of the target directory structure based on the knowledge graph. This allows the corresponding directory structure to be constructed based on the knowledge graph according to this logical semantics, and the knowledge contained in the document data is used to associate the document with the directory structure (i.e., populate the directory structure).

[0079] 1. Grammatical rules and structural norms

[0080] <Directory construction scheme>::=<Directory name>”:”<Directory expression list>

[0081] <Directory name>::=[\u4e00-\u9fa5a-zA-Z_][\u4e00-\u9fa5a-zA-Z0-9_]*

[0082] <Directory expression list>::=<Directory expression>|<Directory expression>";"<Directory expression list>

[0083] <Directory expression>::="{"<node definition list>"}"|<defined directory generation scheme>

[0084] <node definition list>::=<node definition>|<node definition>" / "<node definition list>

[0085] <node definition>::=<node expression>|<node expression>","<node definition>

[0086] <node expression>::=<node name>"|""{"<node constraint condition>"}""{"<document filter condition>"}"

[0087] <node name>::=[\u4e00-\u9fa5a-zA-Z0-9_]+|"\""[\u4e00-\u9fa5a-zA-Z0-9_]+"\""

[0088] <node constraint>::=<constraint clause>|<constraint clause>("AND"|"OR"|"NOT")<node constraint>

[0089] <Constraint clause>::=[\u4e00-\u9fa5a-zA-Z0-9_]+

[0090] <Document filter condition>::=<Condition clause>|<Condition clause>("AND"|"OR"|"NOT")<Document filter condition>

[0091] <Conditional clause>::="~"|[\u4e00-\u9fa5a-zA-Z_][\u4e00-\u9fa5a-zA-Z0-9_=><()]*

[0092] <Defined directory generation scheme>::=[\u4e00-\u9fa5a-zA-Z_][\u4e00-\u9fa5a-zA-Z0-9_]*

[0093] 2. Specifications

[0094] (1) Directory construction scheme: consists of a directory name and a directory expression list.

[0095] (2) Directory name: A character string that complies with the identifier naming rules and is used as the root directory name.

[0096] (3) Directory expression list: contains one or more directory expressions, and the expressions are separated by semicolons.

[0097] (4) Directory expression: Contains a node definition or a list of node definitions connected by " / ". It is used to describe the internal logic of the directory structure; it can also be a defined directory generation scheme to achieve the reuse of directory construction schemes. Each directory expression can generate one or a group of directory structures.

[0098] (5) Node definition: It can contain one or more node expressions. Multiple node expressions are separated by commas, indicating that the same level of directory can be based on multiple logical structures.

[0099] (6) Node expression: It contains the node name, node constraint, document filter condition and static subdirectory list in sequence. The node name and node constraint are separated by “|”.

[0100] 1) Node Name: This is typically an abstract class ontology node, consisting of letters, numbers, and underscores. It cannot begin with a number. If the node name is not enclosed in double quotes, the actual node name is calculated based on the node constraints and the parent directory node. If the node name is enclosed in double quotes, the node name is used directly as the directory name.

[0101] 2) Node Constraints: These consist of one or more constraint clauses connected by logical operators, and are used to describe the semantic relationship between the current node and its parent node. Constraint clauses are strings consisting of letters, numbers, underscores, and spaces.

[0102] 3) Document filter conditions: Logical expressions based on document attributes and logical functions, also enclosed in curly braces. If there are no document filter conditions, empty curly braces are used.

[0103] 4) The constraint clause can be a natural language query, such as "potential customers of the product" (if it is a natural language query, it must be converted into an executable script). It can also be a comparison expression or logical function, for example:

[0104] "Document Type == "Contract"" indicates that the document type is "Contract";

[0105] "Contains (topic, product line)" means that the topic of the document contains a certain product line (the specific value of the product line needs to be determined in the specific context)

[0106] 5) The conditional clause can also be a natural language query, such as "creation time is this year", or a comparison expression or logical function, such as:

[0107] "Document Type == "Contract"" indicates that the document type is "Contract";

[0108] "Document Type == "Contract"" indicates that the document type should be "Contract".

[0109] Additionally, if "~" appears in a conditional clause, it indicates that the document filtering conditions from the parent directory are inherited. For example, "~AND contains (topic, production stage)" means that the current document filtering conditions inherit the document filtering conditions from the parent directory and also extend the current document filtering conditions, forming an AND relationship between the two.

[0110] (6) Chinese characters are allowed in <directory name>, <node name>, <constraint clause>, and <document filtering condition>.

[0111] 3. Example

[0112] A possible directory structure is as follows:

[0113] Business:

[0114] Products|{}{} /

[0115] Customers|{Customers who are already using the product or potential customers of the product}{} /

[0116] "Scheme"|{}{contains (content, customer) AND document type == "Scheme"},

[0117] "Bidding Document"|{}{Contains(knowledge,Party A==Customer)AND document type="Bidding Document"},

[0118] "Contract"|{}{contains(knowledge,Party A==customer)AND document type="contract"}

[0119] express:

[0120] The root directory is named "Business";

[0121] A first-level directory is a hyponym of a product. A node constraint of "empty" indicates that there are as many first-level directories as there are products. A document filter condition of "empty" indicates that a first-level directory does not contain any associated documents.

[0122] The secondary directory contains the names of customers who are already using a product or the names of potential customers of a product. Since the document filtering criteria is empty, there will be no associated documents under the secondary directory.

[0123] The third-level directory contains three node expressions, which means that there are three subdirectories under each second-level directory, namely:

[0124] "Solution": Notice that the node name is enclosed in double quotes, so the node name is used directly as the directory name (the same below). The document filtering condition "contains (content, customer) AND document type == "Solution"" means that if the document content contains a customer name and the document type is "Solution", the document should belong to this subdirectory. The customer is the name of the second-level directory (the same below).

[0125] "Bidding Documents": The document filter condition "Contains (knowledge, Party A == Customer) AND Type = "Bidding Documents"" indicates that if a document contains knowledge (knowledge graph) that "Party A" is a customer and the document type is "Bidding Documents," the document should belong to this subdirectory. In other words, this subdirectory should contain bidding documents provided to a specific customer.

[0126] "Contract": The document filter condition "Contains (knowledge, Party A == Customer) AND Document Type = "Contract"" indicates that if a document contains the knowledge (knowledge graph) that "Party A" is a customer and the document type is "Contract," the document should belong to this subdirectory. In other words, this subdirectory should contain contract documents signed with a customer.

[0127] (4) Script parsing and execution directory structure scheme

[0128] Develop a DSSDS script parser to parse and execute the written directory structure semantic description script (i.e., directory structure scheme). During the document cataloging process, according to the rules and logic in the script, the parser first generates the directory structure using the knowledge graph, and then analyzes and judges the document using the knowledge graph, document attributes, and content data to find related documents; determines the directory to which the document should be assigned, thereby establishing the association between the document and the directory node.

[0129] For example, when a new document enters the cataloging system, the script parser reads the script, analyzes the document according to the rules defined in the script, and assigns the document that meets the screening conditions of the "sales business process" directory to this directory.

[0130] (5) On-demand preprocessing of newly added document materials

[0131] On-demand preprocessing refers to identifying and processing newly added document materials according to the document attributes and content data used in the directory structure scheme, and only ensuring that they can be used for parsing and calculating document screening conditions. This can greatly save the processing overhead of newly added document materials, reduce waiting time, and thus significantly improve the user experience.

[0132] On-demand preprocessing specifically includes the following tasks:

[0133] 1. Format conversion: For newly added documents in different formats, such as PDF, Word, Excel, etc., use document format conversion tools to convert them into a unified text format for subsequent processing. For example, use an open-source document conversion library to convert PDF documents into plain text format.

[0134] 2. Text cleaning and preprocessing: Clean the converted text to remove noise data, such as garbled characters, special characters, irrelevant advertising information, etc. At the same time, perform text normalization processing, such as converting the text to lowercase, removing stop words, and performing stemming. For example, use an NLP toolkit to remove stop words such as "de", "shi", "zai" in the text to improve the efficiency and accuracy of text processing.

[0135] 3. Content analysis based on the directory construction scheme: Based on the defined directory construction scheme and directory structure semantic description script, the pre-processed document content is analyzed to identify and extract document attribute data and content data, including key entities and relationships, to determine the directory hierarchy to which the document should be categorized. For example, in a new market research report document, entities such as "market analysis" and "product sales" are extracted. By matching these with entities such as "market business process" and "sales business module" in the knowledge graph, it is determined that the document should be categorized under the "market business process-sales business module" directory.

[0136] 4. Knowledge Graph Inference: If a document lacks necessary information, preventing accurate cataloging, the system can infer the required knowledge based on relevant knowledge in the knowledge graph or prompt the user for additional information to supplement and improve the document content. For example, if a document only mentions a product's abbreviation, but the knowledge graph uses the full name, the system will prompt the user to provide the full product name for more accurate cataloging.

[0137] (6) System Architecture

[0138] The document automatic cataloging system described in the present invention is divided into a data layer, a processing layer, an interaction layer, and an interface layer:

[0139] 1. Data layer: Stores knowledge graph data, document data, and directory structure semantic description scripts. A graph database (such as Neo4j) can be used to store knowledge graphs, a relational database (such as MySQL) to store document metadata, and a file system to store document content data.

[0140] 2. Processing Layer: This layer implements core processing functions such as document format conversion, text cleaning and preprocessing, entity recognition and relationship extraction, knowledge graph construction, directory structure definition, and directory structure semantic description script parsing and execution. This layer uses various NLP frameworks (such as NLTK and SpaCy) and machine learning algorithm libraries (such as Scikit-learn) to implement these functions.

[0141] 3. Interaction Layer: This layer provides users with an interface and functionality for defining, editing, managing, and maintaining directory structure plans. It allows users to flexibly adjust directory structures, node constraints, and document filtering criteria based on business needs. It also enables the import of existing documents and the ingestion of incremental documents. It also offers knowledge graph query and retrieval, as well as the ability to view generated directory structures and document classification views.

[0142] 4. Interface layer: This layer provides user and system interfaces. The user interface allows users to interact with the system, such as submitting documents for cataloging and querying cataloging results. The system interface is used for integration with other external systems, such as the enterprise's document management system and office automation system. For example, integration with an enterprise document management system can be achieved through a RESTful API, allowing users to directly access the system's cataloging functions within the document management system.

[0143] The following is a detailed explanation using a manufacturing enterprise as an example:

[0144] 1. Knowledge graph construction and implementation

[0145] For example, a manufacturing company collects internal product design documents, production process documents, quality control documents, and other relevant information. NLP tools are used to identify entities and extract relationships within these documents. For example, the Stanford Core NLP toolkit identifies entities such as product name, component name, process step, and quality standard, as well as relationships such as "product contains component," "process step applies to product," and "quality standard applies to component." These entities and relationships are then stored in a Neo4j graph database to build the company's knowledge graph.

[0146] 2. Directory construction plan definition and implementation

[0147] Taking the knowledge graph constructed by a certain enterprise and the set "product series-production link-document type" directory structure as an example, the directory structure semantic description script is implemented.

[0148] First, write a directory structure plan according to the Directory Structure Semantic Description Script (DSSDS) specification. Assuming the root directory name is "Production Management", write the following directory structure plan:

[0149] Production Management:

[0150] Product Line|{}{contains(topic,product line)} /

[0151] Production link|{Production link involved in the product series}{~AND includes (topic, production link)} /

[0152] Document type | {Document type related to production} {~AND document type = document type}

[0153] In the above script:

[0154] (1) “Production Management” is the directory name and the root node of the entire directory structure.

[0155] (2) The first-level directory is defined by "product series|{}{contains (topic, product series)}". Among them:

[0156] "Product Series" is used as the node name. Since double quotes are not used, the actual product series name must be obtained as the directory node name;

[0157] The node constraint is empty, which means that all hyponyms of the product series will be used as the first-level directory;

[0158] The document filter condition "Contains (Subject, Product Series)" means that if a document's subject contains the name of a product series, the document will belong to that directory. For example, if the current first-level directory is "Product A," that directory will contain documents with the subject containing "Product A."

[0159] (3) Secondary directory “Production links|{Production links involved in the product series}{}”, where:

[0160] The node name is "Production Link" and the node constraint is "Production Links Involved in Product Series", indicating that this production link must be associated with the product series in the parent directory.

[0161] The document filtering condition is "~AND includes (subject, production link)", where "~" means inheriting the document filtering condition of the upper directory (i.e., "includes (subject, product series)"), and forming an AND relationship with the document filtering condition of the current directory "includes (subject, production link)". Therefore, the second-level directory should contain documents whose subject includes a certain product series and whose subject includes a certain production link.

[0162] (4) The third-level directory “Document Type|{Document Type Related to Production Process}{Document Type = Document Type}”

[0163] in:

[0164] The node name is "Document Type" and the node constraint is "Document Type Related to Production Links", indicating that the document types related to the specific production links as the second-level directories should be used as the third-level directories.

[0165] The document filtering condition is "~ AND Document Type = Document Type," where "~" inherits the document filtering condition from the parent directory (i.e., "includes (topic, product series) AND includes (topic, production process)"), forming an AND relationship with the document filtering condition of the current directory, "Document Type = Document Type." In "Document Type = Document Type," the "Document Type" on the left is the attribute name, while the "Document Type" on the right is not in quotation marks, indicating that it must be determined based on the directory context. Therefore, a third-level directory should include documents whose topics include a certain product series, whose topics include a certain production process, and whose document types are related to the specific production process in the second-level directory.

[0166] For example, if the document type is "Process Flow Chart" and the document meets the screening conditions of the first-level directory "Product Series" and the second-level directory "Production Link", and the document type "Process Flow Chart" is associated with the corresponding production link in the knowledge graph, the document will be classified under this third-level directory.

[0167] After writing a directory construction plan, save it as a standard script file. During system operation, the developed DSSDS script parser reads this script file. Based on the grammatical rules and logic in the script and the data in the knowledge graph, the script parser first generates a directory structure. It then determines whether existing or incremental documents meet the document screening criteria layer by layer based on their content and metadata, thus associating documents with directory nodes.

[0168] 3. Script Parser

[0169] Python can be used to develop script parsers. When cataloging documents, the parser reads these scripts, generates a directory structure based on the knowledge graph according to the rules in the script, and associates existing or incremental document materials with corresponding directory nodes.

[0170] 4. Implementation of on-demand pre-processing of new documents

[0171] When a new product design document (in Word format) enters the system, it is first converted to text using the pandoc tool. The NLTK toolkit is then used for text cleaning and preprocessing, removing stop words and converting the text to lowercase. Next, based on the directory construction plan and script, the document's content metadata, such as the product name and design parameters, is extracted. These metadata are then substituted into the document filtering criteria of the directory construction plan to determine which directory node the document should be associated with. If the document lacks certain key information, such as if the product series to which the product belongs is not explicitly mentioned, the system uses knowledge graph reasoning to obtain the required information or prompts the user with possible product series options, asking the user to confirm before cataloging.

[0172] 5. System architecture implementation

[0173] The system architecture was constructed. The data layer used Neo4j to store the knowledge graph, MySQL to store document metadata such as document name, creation time, and author, and the file system to store document content. The processing layer used Python to develop various processing modules, invoking NLP frameworks such as NLTK and SpaCy and the Scikit-learn machine learning library to implement related functions. The interface layer used the Flask framework to develop a RESTful API, which was exposed to users and other system calls. For example, users could access the system's web interface through a browser and upload documents for cataloging. The system would receive the documents through the interface layer, call the processing layer for processing, and return the cataloging results to the user.

[0174] The present invention provides a method and system for automatic document cataloging based on a knowledge graph. By defining a directory construction scheme and utilizing the semantic reasoning and knowledge association capabilities of the knowledge graph, the system automatically generates a directory structure and automatically associates document data with corresponding directory nodes. Furthermore, by introducing node constraints and document screening conditions based on logical expressions into the directory construction scheme, the present invention enables the system to comprehensively utilize the knowledge graph, various automatic classification methods, and metadata features to implement a flexible and scalable classification mechanism based on logical expressions. Furthermore, the present invention can also implement on-demand preprocessing of newly added document data based on node constraints and document screening conditions, saving system overhead and automatically completing the classification of newly added document data.

[0175] In summary, the present invention can effectively solve the defects and shortcomings of the aforementioned existing technologies, significantly improve the accuracy, efficiency and adaptability of document cataloging, while greatly reducing labor costs and management costs, and ensuring the consistency and reliability of cataloging results.

Claims

1. A method for automatic document cataloging based on knowledge graph, characterized by: The following steps are involved: S1: Build a knowledge graph: Through entity recognition and relationship extraction, the document topic-related entities and their semantic relationships are stored as a knowledge graph; S2: Define the directory construction plan: Based on the dimensional analysis of the knowledge graph, design the directory hierarchy structure and define the node constraints and document screening conditions; the directory construction plan is written through the directory structure semantic description DSSDS script specification; S3: Parse and execute the directory construction plan: Generate the directory structure according to the DSSDS script, and associate the documents to the corresponding directory nodes through knowledge graph semantic reasoning; S4: Preprocessing of new documents: Perform format conversion, text cleaning, content analysis on new documents, and extract key entities and relationships to match directory screening conditions.

2. The method for automatic document cataloging based on knowledge graph according to claim 1, characterized in that: The DSSDS script specification for the semantic description of directory structure in S2 includes: directory expressions are composed of node names, node constraints, and document filtering conditions; node constraints define the semantic relationship between directory levels through constraint clauses connected by logical operators; document filtering conditions define the conditions for documents to be included in directory nodes through logical expressions.

3. The method for automatic document cataloging based on knowledge graph according to claim 2, characterized in that: The node constraint condition is formed by connecting one or more constraint clauses through logical operators, and the constraint clauses are in the form of functions or comparison conditions.

4. The method for automatic document cataloging based on knowledge graph according to claim 1, characterized in that: New document preprocessing in S4 includes: (a) Convert documents in different formats into a unified text format; (b) Remove noise data through text cleaning and perform normalization; (c) According to the directory construction plan, the required document content metadata, such as product name and design parameters, is extracted, and then substituted into the document screening conditions of the directory construction plan to determine which directory node the document should be associated with. The extracted results are saved in the knowledge graph so that when the directory construction plan is subsequently executed, the knowledge graph can be used to reason and supplement the missing information, or the user can be prompted to supplement the information.

5. The method for automatic document cataloging based on knowledge graph according to claim 1, characterized in that: The directory construction scheme in S2 supports reusing defined directory generation schemes and combining multiple node expressions through logical operators, wherein the logical operators include at least one of AND, OR, and NOT.

6. The method for automatic document cataloging based on knowledge graph according to claim 1, characterized in that: In the node constraint conditions described in S2, if the node name does not use double quotes, the actual directory name is dynamically generated based on the node constraint conditions; if the node name uses double quotes, the node name is directly used as the directory name.

7. The method for automatic document cataloging based on knowledge graph according to claim 1, characterized in that: Constructing the knowledge graph in S1 includes the following specific steps: (i) Collecting internal and external knowledge data related to the document topic from multiple data sources; (ii) extracting entities through named entity recognition algorithms and determining semantic relationships between entities through relation extraction algorithms; (iii) Entities are treated as nodes and semantic relationships as edges, and stored in graph form to form a knowledge graph.

8. A knowledge graph-based document automatic cataloging system, characterized by: include: Data layer: used to store knowledge graphs, document data, and directory construction schemes; Processing layer: used to implement knowledge graph construction, directory construction plan analysis, document preprocessing and association; Interaction layer: used to provide directory construction scheme editing interface and document management functions; Interface layer: used to connect to external systems and provide RESTful API interfaces.

9. The document automatic cataloging system based on knowledge graph according to claim 8, characterized in that: The processing layer includes: Document format conversion module, used to convert documents in PDF, Word, and Excel formats into a unified text format; Text cleaning module, used to remove noise data from documents and perform normalization processing; Script parser, used to parse the directory structure semantic description DSSDS script and generate the directory structure.

10. The document automatic cataloging system based on knowledge graph according to claim 8, characterized in that: The interactive layer supports users to dynamically adjust the directory construction plan, including editing node constraints, document screening conditions and directory hierarchical structure, and update the knowledge graph and directory view in real time.

Images