Process knowledge graph construction and graph enhanced retrieval method based on multi-modal data

Abstract

This invention discloses a method for constructing and enhancing a process knowledge graph based on multimodal data, relating to the field of industrial manufacturing process knowledge management. The method includes collecting, parsing, and labeling multimodal process data to establish a knowledge dataset; constructing a knowledge ontology to obtain a knowledge ontology set; generating a process knowledge graph based on the knowledge data under the guidance of the knowledge ontology; and achieving graph-enhanced retrieval based on the process knowledge graph. This application transforms dispersed process knowledge into a unified knowledge system, realizing the conversion from multi-source heterogeneous data to a structured knowledge graph. It provides a global overview of the process, revealing the intrinsic connections between different process steps through the association of nodes and edges. This supports efficient knowledge retrieval and reasoning, helps engineers quickly locate the required information, assists in production process optimization and decision support, and improves the accessibility and utilization of process knowledge, achieving a significant improvement in the level of industrial intelligent knowledge.

Description

Technical Field

[0001] This invention relates to the field of industrial manufacturing process knowledge management technology, specifically to a method for constructing and enhancing the retrieval of process knowledge graphs based on multimodal data. Background Technology

[0002] In today's rapidly developing industrial landscape, process documents serve as crucial records of a company's product manufacturing steps and details. They also embody a company's core competitiveness and accumulated technical experience, making them vital for its development. However, with technological advancements and increasingly complex production processes, the limitations of traditional process documents are becoming increasingly apparent. Their multi-source, heterogeneous nature (different sources, varying formats, and a lack of effective correlation) makes the integration and rapid retrieval of process knowledge extremely difficult.

[0003] Traditional interpretation of process documents has obvious drawbacks. On the one hand, there is the problem of information silos, where documents of different formats and sources cannot form a unified view, which hinders the sharing and inheritance of process knowledge. On the other hand, due to the lack of structured data models, the process of extracting valuable information from massive amounts of unstructured text is inefficient and prone to errors. Summary of the Invention

[0004] In view of at least one of the above technical problems, this application provides a method for constructing and enhancing process knowledge graphs based on multimodal data. The aim is to transform dispersed process knowledge into a unified knowledge system, constructing a complex process knowledge graph. This achieves the conversion from multi-source heterogeneous data to a structured knowledge graph, providing a global overview of the process. The method reveals the intrinsic connections between different process steps through the association of nodes and edges, thereby supporting efficient knowledge retrieval and reasoning. This helps engineers quickly locate the required information, accelerates the problem-solving process, and ultimately enhances the innovation capabilities and market responsiveness of the entire enterprise.

[0005] The technical solution of this invention is:

[0006] A method for constructing and enhancing a process knowledge graph based on multimodal data, comprising the following steps:

[0007] Collect, analyze, and label multimodal process data to establish a knowledge dataset;

[0008] Construct a knowledge ontology, and all knowledge ontologs form a knowledge ontology set;

[0009] Based on knowledge datasets and knowledge ontology sets, and guided by the knowledge ontology, a process knowledge graph is generated from the knowledge data.

[0010] Enhanced retrieval based on process knowledge graph.

[0011] Optionally, according to the aforementioned process knowledge graph construction and graph enhancement retrieval method, the collection, parsing, and annotation of multimodal process data includes:

[0012] Receive business category tags input by users and semantically classify process data according to industrial manufacturing sub-sectors;

[0013] Collect multimodal process data from various sources and formats, and convert the raw data into a unified intermediate representation structure;

[0014] The transformed data is labeled, and the labeling types include entity type, relation type, and attribute type.

[0015] Version management is implemented for the labeled dataset, recording the timestamp of each update, the type of change operation, the range of data affected, and the verification hash value.

[0016] Optionally, according to the aforementioned process knowledge graph construction and graph enhancement retrieval method, the step of converting the original data into a unified intermediate representation structure includes:

[0017] Document-type data is converted into Markdown structure, and the data is segmented into blocks according to semantic paragraphs using a large language model.

[0018] Optionally, according to the process knowledge graph construction and graph enhancement retrieval method, the step of annotating the converted data includes: annotating the segmented data based on an industry terminology dictionary and process rule templates.

[0019] Optionally, according to the aforementioned process knowledge graph construction and graph enhancement retrieval method, the construction of the knowledge ontology includes:

[0020] Users categorize knowledge ontology into multiple categories based on different business areas and needs.

[0021] Users can create new ontologies or import existing ontologies from external sources to define the categories, attributes, and relationship rules of process knowledge entities, forming a formal knowledge model.

[0022] When it is necessary to share the knowledge ontology with other systems or to conduct further analysis, perform the ontology export operation to ensure the compatibility and interoperability of the knowledge ontology in the process of cross-system interaction;

[0023] Version control is implemented for ontology sets, changes to ontology sets are tracked, and errors are detected and corrected in a timely manner.

[0024] Optionally, according to the aforementioned process knowledge graph construction and graph enhancement retrieval method, the step of generating a process knowledge graph based on knowledge data under the guidance of a knowledge ontology includes:

[0025] Responding to user selections, a subset of process data matching business requirements is selected from the versioned dataset;

[0026] Match the business category tags of the selected data subset, automatically load the corresponding version of the knowledge ontology, use it to organize and express the selected data, and ensure that it can be correctly mapped to the knowledge graph in combination with the industry domain background;

[0027] It receives proper nouns entered by users to avoid splitting proper nouns in the subsequent entity extraction process, thus ensuring the integrity of proper nouns in the knowledge graph.

[0028] Call the large language model to perform joint entity-relation extraction on each text block and output triple data that conforms to ontology constraints;

[0029] Based on the semantic similarity between the process domain lexicon and the context, knowledge fusion is performed on triples;

[0030] The knowledge data middleware allows users to preview the fused triples in a two-dimensional table format, enabling interactive editing of entity links, relationship directions, and attribute values. After editing and confirmation, a visual knowledge graph is generated, showing the relationships and interactions between entities.

[0031] Perform spectral quality inspection;

[0032] Versioned storage is performed on knowledge graphs that pass quality inspection, and the version numbers of the datasets they depend on are associated with the version numbers of the ontology.

[0033] Optionally, according to the process knowledge graph construction and graph enhancement retrieval method, the method for performing graph quality inspection is as follows: receiving process domain questions raised by humans, calling the graph query engine to return answers, and having quality inspectors score the accuracy, completeness, and interpretability of the answers. Only when the comprehensive score is ≥ a preset threshold is the quality inspection passed.

[0034] Optionally, according to the aforementioned process knowledge graph construction and graph enhancement retrieval method, the graph enhancement retrieval based on the process knowledge graph includes:

[0035] The system receives knowledge application templates selected by the user, including intelligent question answering, root cause analysis, process parameter recommendation, decision support, and dynamic electronic process manuals.

[0036] The selected knowledge application template is bound to the knowledge graph version, GraphRAG retrieval algorithm, and a large language model fine-tuned in the process domain to build application instances;

[0037] During the application testing phase: Receive user natural language queries, first perform multi-hop semantic retrieval in the graph using GraphRAG to locate relevant entities and their neighborhood subgraphs, then concatenate the structured description of the subgraphs with the query to form a prompt which is input into the large language model to generate an answer with graph source tracing;

[0038] After testing, the application instance is released as a standardized service interface, supporting calls via both interactive entry points and program calls.

[0039] Collect user feedback, answer acceptance rate, and error correction behavior during application operation, accumulate verified and valid process cases into the knowledge case library, and inject newly discovered process mechanism rules into the knowledge ontology mechanism library to trigger ontology version updates.

[0040] This application employs the aforementioned technical solution to transform multi-source heterogeneous process data into knowledge, constructing a complex process knowledge graph. This knowledge graph is then used for enhanced retrieval, assisting in production process optimization and decision support, and improving the accessibility and utilization of process knowledge. Ultimately, this significantly enhances the level of industrial intelligent knowledge, with specific beneficial effects including:

[0041] (1) Accurately identify and extract entities and relationships in complex process data, construct a structured knowledge network, and improve the systematicness and operability of knowledge.

[0042] (2) Realize the analysis and graph transformation of multi-source heterogeneous process data, break down the information silos of traditional document interpretation, and provide a unified knowledge foundation for industrial intelligence.

[0043] (3) The introduction and optimization of the map enhancement retrieval mechanism greatly improves retrieval efficiency and accuracy, quickly locates key process information, and helps production optimization and decision support. Attached Figure Description

[0044] Figure 1 This is an architecture diagram of the knowledge platform system based on a microservice architecture in this implementation method;

[0045] Figure 2 This is a flowchart of the process knowledge graph construction and graph enhancement retrieval method based on multimodal data in this embodiment;

[0046] Figure 3 This is a technical roadmap for constructing complex process knowledge graphs based on multimodal data in this implementation method;

[0047] Figure 4 This is a technical roadmap for graph enhancement retrieval based on complex process knowledge graphs in this implementation method. Detailed Implementation

[0048] To facilitate understanding of this application, a more comprehensive description of this application will be provided below with reference to the accompanying drawings.

[0049] This implementation method relies on the process knowledge graph construction and graph enhancement retrieval method based on multimodal data. Figure 1 The implementation of the knowledge platform system based on a microservice architecture is shown below. Figure 1 As shown, the system includes a front-end service or third-party service module, an API gateway module, a service registration center module, a configuration center module, multiple functional microservice modules, a knowledge data connection pool module, and a knowledge database.

[0050] The front-end service or third-party service module is used to initiate knowledge graph construction requests or knowledge graph retrieval requests. The API gateway module interacts with external requests, providing a unified service IP address and port, and forwards the requests to the service registry module. The service registry module receives requests from the API gateway module and calls the configuration center module to obtain the service chain configuration information required for this request; the service chain configuration information includes the type, deployment address, call order, and dependencies of each functional microservice participating in this request processing. The configuration center module centrally stores and dynamically manages the runtime configuration information of all functional microservices, supporting dynamic assembly of service chains according to request type. The multiple functional microservice modules include at least two types of service groups: the first type is the graph construction service group, which includes multimodal data parsing service, graph generation service, and graph quality inspection service; the second type is the graph application service group, which includes knowledge question answering service, knowledge retrieval service, and knowledge graph enhancement service. The service registry module locates and schedules the corresponding functional microservice module to execute the request based on the service chain configuration information. Each functional microservice module establishes a connection with the knowledge database through the knowledge data connection pool module, enabling the reading, writing, verification, and management of knowledge data. After each functional microservice module completes its execution, the response result is returned to the front-end service or third-party service module sequentially via the service registration center module and the API gateway module.

[0051] This knowledge platform architecture, based on a microservices framework, offers significant flexibility and scalability. By decomposing the system into independent, loosely coupled service units, each responsible for a specific function, it makes development, deployment, and maintenance more efficient and simpler. This architecture supports dynamic scaling, easily adding or removing resources as needed, thereby improving system resilience and cost efficiency. Simultaneously, the unified entry point and automated service discovery mechanism implemented through API gateways and service registries enhance system reliability and fault tolerance, ensuring the entire system continues to operate even when some services fail. Furthermore, this architecture supports technology diversity and continuous integration / continuous deployment (CI / CD), allowing teams to adopt the technology stack best suited for specific services and achieve rapid, frequent, and reliable software updates. Overall, this architecture not only optimizes resource utilization and operational efficiency but also improves user experience and long-term system maintainability.

[0052] Figure 2 This is a flowchart of the process knowledge graph construction and graph enhancement retrieval method based on multimodal data in this implementation method. For example... Figure 2 As shown, the method for constructing and enhancing a process knowledge graph based on multimodal data includes the following steps:

[0053] Step 1: Collection, parsing, annotation, and management of multimodal process data to achieve the underlying construction of knowledge data;

[0054] like Figure 3 As shown, this step specifically includes the following steps:

[0055] Step 1.1: Receive the business category tags input by the user and perform semantic classification of the process data according to the sub-fields of industrial manufacturing; the sub-fields include, but are not limited to, machining, heat treatment, surface treatment, assembly process and quality inspection;

[0056] Users need to perform business-specific classification of knowledge data, that is, to classify the knowledge data in detail according to business needs, so as to ensure that the data processed subsequently has a clear structure and a clear purpose. This step is one of the key steps in the entire knowledge graph construction process, because only by accurately classifying the data can we ensure that subsequent processing and analysis can be more precise.

[0057] Step 1.2: Collect multimodal process data from various sources and formats, and convert the raw data into a unified intermediate representation structure;

[0058] The system receives multimodal process data from users, consisting of text, images, videos, CAD drawings, process cards, equipment logs, and sensor time-series data. It converts the raw data into a unified intermediate representation structure. Document data is converted into a Markdown structure with heading levels and table recognition markers, facilitating the subsequent recognition of headings, tables, text, and images. To ensure the completeness of data parsing, a large language model is used to segment the data into semantic blocks, ensuring that subsequent data processing follows a sequential order and small text blocks, avoiding information loss during knowledge entity relationship extraction.

[0059] Step 1.3: Perform annotation on the transformed data. The annotation types include entity type, relation type, and attribute type.

[0060] This implementation method, based on an industry terminology dictionary and process rule templates, performs manual or automatic annotation on the segmented data. Annotation types include entity types (process parameters, equipment models, material grades, process names, defect patterns), relation types ("controlled by…", "applies to…", "leads to…", "preceding process is…"), and attribute types (tolerance range, temperature range, time threshold). This provides stronger directionality for subsequent knowledge entity and relation extraction. The purpose of this step is to enable computers to better understand and process data, thereby improving the efficiency and quality of knowledge graph construction.

[0061] Step 1.4: Perform version control on the labeled dataset, and record the timestamp of each update, the type of change operation, the range of data affected, and the verification hash value;

[0062] Versioning data sets ensures continuous updates and maintenance of knowledge datasets, with each version recorded for easy tracking of historical changes and retrospection. This helps in understanding the evolution of data, enabling timely identification and resolution of problems, and also facilitates better management and utilization of data in future work.

[0063] Step 2: Knowledge Ontology Construction and Management;

[0064] like Figure 3 As shown, this step specifically includes the following steps:

[0065] Step 2.1: Users classify the knowledge ontology according to business needs, dividing it into multiple categories based on different business domains and requirements;

[0066] Users categorize knowledge ontology into business categories based on different business domains and needs, dividing the knowledge ontology into multiple categories to support knowledge representation and reasoning in specific domains. For example, based on the business needs of industrial manufacturing, the ontology business classification dimensions can be defined, including process flow dimension, equipment asset dimension, material property dimension, quality control dimension, and energy consumption and safety dimension. This step is the foundation for building a knowledge graph. By reasonably classifying the knowledge ontology, it can be ensured that business-related information can be accurately extracted in the subsequent knowledge extraction process.

[0067] Step 2.2: Users create new ontologies or import existing ontologies from external sources to define the categories, attributes, and association rules of process knowledge entities, forming a formal knowledge model;

[0068] This step is the core of building a knowledge graph. By creating or importing an ontology, it can provide clear guidance for knowledge extraction, ensuring that the extracted information is highly professional and accurate.

[0069] Step 2.3: When it is necessary to share the knowledge ontology with other systems or to conduct further analysis, perform the ontology export operation to ensure the compatibility and interoperability of the knowledge ontology in the process of cross-system interaction;

[0070] Users export the ontology to ensure cross-system compatibility and interoperability when sharing the knowledge ontology with other systems or for further analysis. This step aims to achieve knowledge sharing and reuse. By exporting the ontology, other systems can also utilize the knowledge graph we have built, thereby improving the overall informatization level of the industry.

[0071] Step 2.4: Implement version control for the ontology set, track changes to the ontology set, and promptly identify and correct errors;

[0072] Users manage the versioning of ontology sets, similar to how they manage datasets, to track changes and ensure consistency. This step is to ensure the quality of the knowledge graph; by version controlling the ontology set, errors can be detected and corrected in a timely manner, guaranteeing the accuracy and reliability of the knowledge graph.

[0073] Step 3, Knowledge Graph Construction and Management: Based on the constructed knowledge dataset and knowledge ontology set, a complex process knowledge graph is generated from the knowledge data under the guidance of the knowledge ontology, and the process knowledge graph is managed accordingly.

[0074] like Figure 3 As shown, this step specifically includes the following steps:

[0075] Step 3.1: In response to user selection, filter the subset of process data that matches business requirements from the versioned dataset;

[0076] Users select knowledge data and, based on business needs, choose relevant data from the categorized knowledge datasets to build a knowledge graph. This step is the starting point of the entire knowledge graph construction process. Users need to filter out data relevant to their business from a large amount of knowledge data to facilitate subsequent knowledge extraction and fusion.

[0077] Step 3.2: Match the business category tags of the selected data subset, automatically load the corresponding version of the knowledge ontology, and use it to organize and express the selected data to ensure that it can be correctly mapped to the knowledge graph in combination with the industry domain background;

[0078] Users select a knowledge ontology to organize and represent the chosen knowledge data, ensuring that it can be correctly mapped into the knowledge graph in conjunction with the industry domain context. This step is crucial for building a knowledge graph. By selecting an appropriate knowledge ontology, clear guidance can be provided for the extraction and fusion of knowledge data, ensuring that the generated knowledge graph is professional and accurate.

[0079] Step 3.3: Receive proper nouns entered by users to avoid splitting proper nouns in the subsequent entity extraction process and ensure the integrity of proper nouns in the knowledge graph.

[0080] Users input proper nouns into the system to avoid splitting them during subsequent entity extraction. This step ensures the integrity of proper nouns within the knowledge graph, preventing information loss or misunderstanding due to splitting.

[0081] Step 3.4: Call the large language model to perform joint entity-relation extraction on each text block and output triple data that conforms to ontology constraints;

[0082] Users perform entity / relation extraction, using large modeling techniques to extract entities and relations from each small text block, forming triplet data. This step involves deep mining of knowledge data using large modeling techniques to extract entities and relations, providing foundational data for subsequent knowledge fusion, and summarizing the extraction model based on the extraction task.

[0083] Step 3.5: Perform knowledge fusion on triples based on the semantic similarity between the process domain lexicon and the context;

[0084] Users perform knowledge fusion by leveraging large model technology to integrate the extracted entity / relation triples based on a process domain lexicon and contextual semantics, eliminating redundancy and conflicts to form a consistent knowledge representation. This step involves organizing and optimizing the extracted entities and relations, making the knowledge representation clearer and more accurate by eliminating redundancy and conflicts.

[0085] Step 3.6: Preview the fused triples in a two-dimensional table format using the knowledge data middleware. Users can interactively edit entity links, relationship directions, and attribute values. After editing and confirmation, a visual knowledge graph is finally generated, showing the relationships and interactions between entities.

[0086] Users preview the knowledge data middleware, and the knowledge platform provides a visual preview in tabular form, allowing users to view and edit the knowledge graph construction process and promptly identify and correct errors. After editing and confirmation, a visual process knowledge graph is generated, displaying the relationships and interactions between entities. This step is the ultimate goal of knowledge graph construction; by generating a visual knowledge graph, the relationships and interactions between entities can be intuitively displayed.

[0087] Step 3.7: Perform graph quality inspection: Receive process-related questions raised by manual personnel, call the graph query engine to return answers, and have quality inspectors score the accuracy, completeness, and interpretability of the answers. Only when the comprehensive score is greater than or equal to the preset threshold will the quality inspection be passed.

[0088] Users conduct graph quality checks, performing a comprehensive quality inspection of the generated knowledge graph. This primarily involves asking questions to the platform, which then answers the questions using the generated graph. Users score the answers, and only graphs that reach a pre-defined passing score pass the quality check, ensuring the graph's accuracy and completeness. This step is a crucial quality control process for the generated knowledge graph; through manual quality checks, the accuracy and completeness of the knowledge graph can be guaranteed.

[0089] Step 3.8: Perform versioned storage on the knowledge graph that has passed the quality inspection, and associate the version number of the dataset it depends on with the version number of the ontology;

[0090] Users manage knowledge graph versions, which involves storing and version-controlling the quality-checked knowledge graph to facilitate tracking changes and viewing historical versions. This step is an important means of long-term management and maintenance of knowledge graphs. Through version control, changes to the knowledge graph can be easily tracked, providing a basis for future analysis and decision-making.

[0091] The above outlines the complete process of building a knowledge graph, with knowledge data management, knowledge ontology management, and knowledge graph management each playing an indispensable role. Through detailed business classification, multimodal data access, and annotation, knowledge data management ensures the structure and clarity of the data, providing a solid foundation for subsequent processing. Knowledge ontology management, by establishing a professional knowledge concept system, imbues the generated graph with professionalism and accuracy. Finally, knowledge graph management, guided by the ontology, generates a visualized knowledge graph from this knowledge data and achieves continuous updating and maintenance of the knowledge through version control. This systematic construction method not only improves the accuracy and completeness of the knowledge graph but also enhances its cross-system compatibility and interoperability, enabling the knowledge graph to better serve various application scenarios.

[0092] Step 4: Implement graph-enhanced retrieval based on process knowledge graph;

[0093] This step leverages GraphRAG technology to construct knowledge applications for graph-enhanced retrieval. This step enables efficient querying and analysis of the knowledge graph, while also significantly improving the accuracy of graph retrieval. This allows users to quickly find the necessary process information and apply it to real-world production scenarios, such as fault diagnosis and process optimization. Figure 4 As shown, this step specifically includes the following steps:

[0094] Step 4.1: Receive the knowledge application template selected by the user. The template includes intelligent question answering, root cause analysis, process parameter recommendation, decision support, and dynamic electronic process manual.

[0095] Users select a knowledge application template, meaning they need to choose one suitable for the current task. Templates include, but are not limited to, intelligent question answering, intelligent recommendation, root cause analysis, decision support, or e-books. Each template corresponds to a specific knowledge application scenario.

[0096] Step 4.2: Bind the selected knowledge application template with the graph version generated in Step 3.8, the GraphRAG retrieval algorithm, and the large language model fine-tuned in the process domain to build an application instance;

[0097] Users can bind and associate application templates with knowledge graphs and knowledge algorithms. This allows the knowledge application templates to call on the knowledge and algorithms in the graph, thereby providing theoretical credibility support and knowledge reasoning for knowledge applications. At the same time, the platform combines GraphRAG technology to import the graph selected by the user, which serves as an information source and provides a reliable knowledge foundation for subsequent text generation.

[0098] Step 4.3: In the application testing phase: Receive the user's natural language query, first perform multi-hop semantic retrieval in the graph using GraphRAG to locate relevant entities and their neighborhood subgraphs, then concatenate the structured description of the subgraph with the query to form a Prompt input into the large language model to generate an answer with graph source tracing;

[0099] When users conduct knowledge application tests, that is, when the platform receives a user's query or task, GraphRAG will first search for relevant entities and relationships in the knowledge graph selected in step 2. At the same time, with the support of a large model, GraphRAG can understand the intent behind the query and find the most relevant knowledge fragments, thereby guiding the generation of text answers.

[0100] Step 4.4: After testing, publish the application instance as a standardized service interface, supporting calls via both interactive entry points and programmatic calls;

[0101] Users can publish knowledge applications. After testing, the knowledge application can be published to the public. The published application instance is a standardized service interface that supports two types of calls: interactive entry points and program calls. Interactive entry points include menu items, WeChat mini-program cards, and WeChat Work robots, while program calls include API methods.

[0102] Step 4.5: Collect user feedback, answer acceptance rate and error correction behavior during application operation, accumulate the verified and valid process cases into the knowledge case library, and inject the newly discovered process mechanism rules into the knowledge ontology mechanism library to trigger the ontology version update.

[0103] Users will store knowledge cases in the knowledge case library and the knowledge ontology mechanism library. After the knowledge application is released, the platform will collect user feedback and usage data, store successful cases in the knowledge case library, and update and improve the knowledge ontology mechanism library.

[0104] The knowledge application management system integrates a complex process knowledge graph and advanced GraphRAG technology to construct an efficient, flexible, and intelligent application development and deployment process. This process, starting from user needs, provides diverse application templates, ensuring a high degree of customization and applicability of knowledge applications. By tightly integrating the selected templates with the knowledge graph and algorithms, it not only strengthens the theoretical support and knowledge reasoning capabilities of the applications but also, combined with GraphRAG technology, provides a reliable information source for text generation, ensuring the accuracy and relevance of the output content. During the testing phase, it can quickly respond to user queries, accurately search for entities and relationships in the knowledge graph, and generate high-quality answers, greatly improving the user experience. Ultimately, knowledge applications can be flexibly deployed through multiple channels, achieving widespread dissemination and convenient access to knowledge. More importantly, the platform continuously collects user feedback and usage data, accumulating successful cases into a knowledge case library and continuously optimizing the knowledge ontology mechanism library, forming a closed-loop iterative optimization mechanism that drives continuous improvement and innovation in knowledge applications. This entire process not only accelerates the transformation of knowledge into practical applications but also significantly improves the intelligence level of applications and the user experience, providing enterprises and individuals with an efficient, intelligent, and sustainable knowledge service platform.

[0105] It should be understood that, inspired by the technical concept of this invention, those skilled in the art can make various improvements or modifications based on the above content without departing from the scope of this invention, and these modifications still fall within the protection scope of this invention.

Claims

1. A process knowledge graph construction and graph enhanced retrieval method based on multi-modal data, characterized in that, The method includes the following steps: Collect, analyze, and label multimodal process data to establish a knowledge dataset; Construct a knowledge ontology, and all knowledge ontologs form a knowledge ontology set; Based on knowledge datasets and knowledge ontology sets, and guided by the knowledge ontology, a process knowledge graph is generated from the knowledge data. Enhanced retrieval based on process knowledge graph. 2.The process knowledge graph construction and graph enhanced retrieval method according to claim 1, characterized in that, The collection, analysis, and annotation of multimodal process data include: Receive business category tags input by users and semantically classify process data according to industrial manufacturing sub-sectors; Collect multimodal process data from various sources and formats, and convert the raw data into a unified intermediate representation structure; The transformed data is labeled, and the labeling types include entity type, relation type, and attribute type. Version management is implemented for the labeled dataset, recording the timestamp of each update, the type of change operation, the range of data affected, and the verification hash value. 3.The process knowledge graph construction and graph enhanced retrieval method according to claim 2, characterized in that, The process of converting the raw data into a unified intermediate representation structure includes: Document-type data is converted into Markdown structure, and the data is segmented into blocks according to semantic paragraphs using a large language model.

4. The method for constructing and enhancing process knowledge graphs and for retrieval according to claim 3, characterized in that, The step of annotating the converted data includes: annotating the segmented data based on an industry terminology dictionary and process rule templates.

5. The method for constructing and enhancing a process knowledge graph according to claim 2, characterized in that, The construction of the knowledge ontology includes: Users categorize knowledge ontology into multiple categories based on different business areas and needs. Users can create new ontologies or import existing ontologies from external sources to define the categories, attributes, and relationship rules of process knowledge entities, forming a formal knowledge model. When it is necessary to share the knowledge ontology with other systems or to conduct further analysis, perform the ontology export operation to ensure the compatibility and interoperability of the knowledge ontology in the process of cross-system interaction; Version control is implemented for knowledge ontology sets, changes to knowledge ontology sets are tracked, and errors are detected and corrected in a timely manner.

6. The method for constructing and enhancing a process knowledge graph according to claim 5, characterized in that, The generation of a process knowledge graph based on knowledge data under the guidance of a knowledge ontology includes: Responding to user selections, a subset of process data matching business requirements is selected from the versioned dataset; Match the business category tags of the selected data subset, automatically load the corresponding version of the knowledge ontology, use it to organize and express the selected data, and ensure that it can be correctly mapped to the knowledge graph in combination with the industry domain background; It receives proper nouns entered by users to avoid splitting proper nouns in the subsequent entity extraction process, thus ensuring the integrity of proper nouns in the knowledge graph. Call the large language model to perform joint entity-relation extraction on each text block and output triple data that conforms to ontology constraints; Based on the semantic similarity between the process domain lexicon and the context, knowledge fusion is performed on triples; The knowledge data middleware allows users to preview the fused triples in a two-dimensional table format, enabling interactive editing of entity links, relationship directions, and attribute values. After editing and confirmation, a visual knowledge graph is generated, showing the relationships and interactions between entities. Perform spectral quality inspection; Versioned storage is performed on knowledge graphs that pass quality inspection, and the version numbers of the datasets they depend on are associated with the version numbers of the ontology.

7. The method for constructing and enhancing a process knowledge graph according to claim 6, characterized in that, The method for performing graph quality inspection is as follows: receiving process-related questions raised manually, calling the graph query engine to return answers, and having quality inspectors score the accuracy, completeness, and interpretability of the answers. Quality inspection is only passed when the comprehensive score is greater than or equal to a preset threshold.

8. The method for constructing and enhancing a process knowledge graph according to claim 1, characterized in that, The graph-enhanced retrieval based on process knowledge graph includes: The system receives knowledge application templates selected by the user, including intelligent question answering, root cause analysis, process parameter recommendation, decision support, and dynamic electronic process manuals. The selected knowledge application template is bound to the knowledge graph version, GraphRAG retrieval algorithm, and a large language model fine-tuned in the process domain to build application instances; During the application testing phase: Receive user natural language queries, first perform multi-hop semantic retrieval in the graph using GraphRAG to locate relevant entities and their neighborhood subgraphs, then concatenate the structured description of the subgraphs with the query to form a prompt which is input into the large language model to generate an answer with graph source tracing; After testing, the application instance is released as a standardized service interface, supporting calls via both interactive entry points and program calls. Collect user feedback, answer acceptance rate, and error correction behavior during application operation, accumulate verified and effective process cases into the knowledge case library, and inject newly discovered process mechanism rules into the knowledge ontology mechanism library to trigger ontology version updates.

Images