Question and answer method for customized furniture plate processing technology based on multi-modal knowledge graph

Abstract

The application discloses a customized furniture plate processing technology question and answer method based on a multi-modal knowledge graph, which comprises the following steps: according to processing requirements, different production stations are divided from a production process, different production stations are divided into multi-dimensional data, a customized furniture plate processing workshop station task beat model is constructed according to the multi-dimensional data; the multi-dimensional data is preprocessed to obtain a customized furniture plate processing knowledge graph dataset; based on the customized furniture plate processing knowledge graph dataset, a customized furniture plate processing knowledge graph and an entity set, a relationship set and an attribute set are defined; knowledge extraction, knowledge fusion and knowledge storage processing are performed on the customized furniture plate processing knowledge graph dataset to obtain a knowledge graph data layer, and a customized furniture plate processing knowledge graph is constructed; and based on the customized furniture plate processing knowledge graph, a question and answer system is constructed. The application improves the availability and value of data.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of custom furniture plate processing data management and representation, and particularly relates to a question and answer method for custom furniture plate processing technology based on a multi-modal knowledge graph. BACKGROUND

[0002] The current custom furniture production process has high mechanization, but low raw material utilization rate and long production cycle, mainly because the production and manufacturing full-cycle data utilization rate is low, and a large amount of high-dimensional multi-source heterogeneous data is generated in each production beat in the actual operation process of the production workshop, and the product production process data is crucial to product quality. Nowadays, traditional researches on production process data modeling mainly focus on four aspects: object-based modeling, metadata-based modeling, ontology-based modeling and process complex network-based modeling. The object-based modeling has good stability, but does not have a structured data description mechanism, and it is difficult to realize the interconnection and intercommunication of multi-source heterogeneous data. The metadata-based modeling uses self-description files and simpler component-based design, and has good language interoperability, but has poor flexibility in knowledge organization of process data. The ontology-based modeling has high formalization degree, strong expression ability, good support for context reasoning, easy information sharing and other characteristics, and can well express complex environments. The process complex network-based modeling can integrate production factors and processing flow. Although the above-mentioned various data modeling has advantages, it is insufficient in expressing the semantic relationship and time sequence between data, and information fragmentation and information island problems are prone to occur in the production process. Moreover, there are text, image, model and other multi-modal process data in the furniture plate production full life cycle data. Therefore, it is urgent to propose a knowledge modeling for custom furniture plate multi-modal processing data, and to construct a corresponding process question and answer system. SUMMARY

[0003] To solve the above technical problems, the application provides a question and answer method for custom furniture plate processing technology based on a multi-modal knowledge graph, which reduces the use threshold and the additional workload caused by data differences, thereby improving the overall processing speed and consistency, improving the decision-making efficiency and improving the data availability and value.

[0004] To achieve the above purpose, the application provides a question and answer method for custom furniture plate processing technology based on a multi-modal knowledge graph, which includes:

[0005] According to the processing requirements, the production process is divided into different production stations, different production stations are divided into multi-dimensional data, and a custom furniture plate processing workshop station task beat model is constructed according to the multi-dimensional data;

[0006] The multi-dimensional data is preprocessed to obtain a custom furniture plate processing knowledge graph data set;

[0007] Based on the customized furniture plate processing knowledge graph dataset, define the customized furniture plate processing knowledge graph and the entity set, the relationship set and the attribute set;

[0008] Perform knowledge extraction, knowledge fusion and knowledge storage processing on the customized furniture plate processing knowledge graph dataset to obtain a knowledge graph data layer and construct a customized furniture plate processing knowledge graph;

[0009] Based on the customized furniture plate processing knowledge graph, construct a question and answer system.

[0010] Optionally, the different production stations are divided into multi-dimensional data, which includes:

[0011] According to the classification and arrangement of people, machines, materials, methods and environment, the resource knowledge elements are constructed, and the resource knowledge elements are divided into static resource data, plan resource data and dynamic resource data according to the dimensions.

[0012] Optionally, the customized furniture plate processing workshop station task beat model is constructed according to the multi-dimensional data, which includes:

[0013] F i ={I i ,S i ,D i ,T}

[0014] Wherein, I i is a static resource data set, S i is a plan resource data set, D i is a dynamic resource data set, and T is a processing station beat time set.

[0015] Optionally, the customized furniture plate processing knowledge graph dataset includes:

[0016] Based on the customized furniture enterprise product life cycle management system PLM, the processing station is taken as a basic unit, the data is collected according to the basic data layer, the plan data layer, the processing procedure layer and the production state layer, and the processing task beat time length of each station is defined according to the production cycle of each station;

[0017] For structured data, missing values are completed, repeated values are deleted, abnormal values are processed, and standardized; for process files and quality inspection standard files, text and picture extraction is performed, and text data is segmented, special word processing is performed, image data is described and labeled; three-dimensional models are fully sorted out and exported, model features are extracted by model semantic extraction algorithm and saved, and a customized furniture plate processing knowledge graph dataset is obtained.

[0018] Optionally, the customized furniture plate processing knowledge graph and the entity set, the relationship set and the attribute set include:

[0019] The customized furniture plate processing multi-modal process knowledge graph is based on the traditional knowledge graph and adds modal knowledge;

[0020] Based on the customized furniture plate processing knowledge graph and the customized furniture plate processing real data, the entity set is defined to further divide the customized furniture plate processing data into three categories: text entity, image entity, and three-dimensional model entity;

[0021] Based on the customized furniture plate processing knowledge graph and the customized furniture plate processing real data, the relationship set is defined to be divided into structure relationship, time sequence relationship, and operation relationship according to the characteristics of the assembly process data;

[0022] Based on the customized furniture plate processing knowledge graph and the customized furniture plate processing real data, the attribute set is defined as a kind of inherent characteristic which can make semantic supplement to entities and relationships.

[0023] Optionally, the knowledge extraction of the customized furniture plate processing knowledge graph data set includes: for structured data knowledge extraction, selecting relevant data from the enterprise database as the basis for knowledge extraction; through entity, relationship, and attribute matching, the data is converted into a triple form conforming to the knowledge extraction task;

[0024] For three-dimensional model data knowledge extraction of semi-structured target format, analyze the structure and attributes of the target data, and identify the entities therein; extract the attribute information of the entities from the target data; construct knowledge triples according to the relationships between entities;

[0025] For text data knowledge extraction, use the BERT+BiLSTM+CRF model for knowledge extraction.

[0026] Optionally, the knowledge fusion of the customized furniture plate processing knowledge graph data set includes: inputting the processed text data into the text encoder of CLIP to obtain the embedding vector of the text;

[0027] Input the processed image data into the image encoder of CLIP to obtain the embedding vector of the image;

[0028] Use the contrast loss function of CLIP to calculate the similarity score between the text and the image;

[0029] According to the similarity score, determine the most matching image of the text, and perform knowledge fusion.

[0030] Optionally, based on the customized furniture plate processing knowledge graph, a question and answer system is constructed, which includes: based on the customized furniture field application scene, customizing and developing a user interface to provide an interface for user interaction with the system;

[0031] Backend logic is implemented using the Python programming language and the Node.js framework to develop backend services that receive user requests and process them;

[0032] The question input first analyzes the question, extracts the question text, classifies the question according to the extracted entities and relationship types, and assembles the classification results into a dictionary;

[0033] According to the corresponding question type and entity type in the question, the Aligner model is used for entity disambiguation and answer matching, and the searched answer is assembled and output to the user for reference or use.

[0034] Technical effects of the present application:

[0035] (1) Field data adaptation and flexible configuration: the present application can automatically adapt to data from different fields and structurally integrate them into the customized furniture plate processing knowledge graph, ensuring effective use of data and maximizing value; the present application supports flexible configuration according to user needs, reducing the use threshold, making it easy for business personnel and even technical personnel with low programming skills to get started, and completing complex process flow queries through simple interface operations.

[0036] (2) Data adaptability and standardized processing: the present application has high adaptability and can automatically adjust processing methods according to different types of data to ensure high efficiency and accuracy even when faced with diverse inputs; a standardized processing flow is provided for various types of data, reducing additional workload caused by data differences and improving overall processing speed and consistency.

[0037] (3) Knowledge sharing and data interconnection: the present application realizes knowledge sharing between departments within an enterprise by building a unified knowledge graph, promoting information flow and helping to improve decision-making efficiency; integrates workshop process data, process files, personnel information and equipment basic information into a platform to form a data interconnection network, improving data usability and value. BRIEF DESCRIPTION OF DRAWINGS

[0038] The accompanying drawings, which form a part of this application, are intended to provide further understanding of the present application, and the schematic embodiments of the present application and their descriptions are intended to explain the present application, and do not constitute an improper limitation on the present application. In the drawings:

[0039] Figure 1 The flowchart of the question and answer method of the customized furniture plate processing technology based on the multi-modal knowledge graph of the embodiments of the present application. DETAILED DESCRIPTION

[0040] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0041] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown.

[0042] As Figure 1 shown, the embodiment provides a question and answer method for custom furniture plate processing technology based on a multi-modal knowledge graph, which includes:

[0043] According to the processing requirements, the production process is divided into different production stations, and different production stations are divided into multi-dimensional data, and a custom furniture plate processing workshop station task beat model is constructed according to the multi-dimensional data;

[0044] Based on the multi-dimensional data, the custom furniture plate processing knowledge graph dataset is obtained by preprocessing;

[0045] Based on the custom furniture plate processing knowledge graph dataset, the custom furniture plate processing knowledge graph and the entity set, the relationship set, and the attribute set are defined;

[0046] The custom furniture plate processing knowledge graph dataset is subjected to knowledge extraction, knowledge fusion and knowledge storage processing to obtain a knowledge graph data layer, and a custom furniture plate processing knowledge graph is constructed;

[0047] Based on the custom furniture plate processing knowledge graph, a question and answer system is constructed.

[0048] Further, the different production stations are divided into multi-dimensional data, which includes:

[0049] The different production stations are classified and arranged according to people, machines, materials, methods and environment to construct resource knowledge elements, and the resource knowledge elements are divided into static resource data, planned resource data and dynamic resource data according to dimensions.

[0050] Further, the custom furniture plate processing workshop station task beat model is constructed according to the multi-dimensional data, which includes:

[0051] F i ={I i ,S i ,D i ,T}

[0052] Wherein, I i is a static resource data set, S i is a planned resource data set, D iT is a set of processing station tact time for dynamic resource data set.

[0053] Specifically, based on the processing demand, the production process is divided into different production stations according to the similarity of the processing technology, including main stations such as pressing, cutting, edge sealing, hole arranging, sorting, quality inspection, repairing, cleaning and warehousing, and other stations related to the main stations are defined as subordinate sub-stations. The data of different stations are classified and arranged according to man, machine, material, method and environment to classify manufacturing resource knowledge elements, and the data are divided into static resource data, planned resource data and dynamic resource data according to dimensions. In order to realize the unified representation and updating of multi-dimensional data, the processing station tact of each station is defined based on the actual working period of the processing task, and the static resource data, planning data and dynamic process data resources are integrated to establish a customized furniture plate processing workshop station task tact model F i .

[0054] Further, obtaining the customized furniture plate processing knowledge graph data set specifically includes: based on the product lifecycle management system PLM of the customized furniture enterprise, taking the processing station as the basic unit, collecting data in layers according to the basic data layer, the planning data layer, the processing procedure layer and the production state layer, and defining the processing task tact time of each station according to the production period of each station. The basic data layer embeds static resource data including equipment, tools, etc., the planning data layer embeds processing plan data of each station including processing batch, order, etc., the processing procedure layer embeds processing operation content in the process document, and the production state layer embeds related data stored in the time series database according to different collection intervals, such as real-time changing sensors, data collection instruments, etc.; different data types use different data preprocessing operations. For structured data, missing value completion, repeated value deletion, abnormal value processing, standardization, etc. are performed; for pdf, docx files such as process files and quality inspection standards, text and image extraction are performed, and text data is processed by word segmentation, sentence segmentation, special word processing, etc., and image data is processed by feature description, labeling, etc.; three-dimensional models are fully sorted out and exported as glTF files, model features are extracted by model semantic extraction algorithm and stored in JSON files to obtain the customized furniture plate processing knowledge graph data set.

[0055] Further, the customized furniture plate processing knowledge graph and the entity set, the relationship set and the attribute set are defined as follows:

[0056] The customized furniture plate processing multi-modal process knowledge graph is added with modal knowledge on the basis of the traditional knowledge graph, and is represented as follows:

[0057] MPKG={E,R,A,TW,TV,TM}

[0058] In the formula, E represents all entity sets, R represents all relationship sets, A represents all attribute sets, TW is a text type process knowledge triple set, TV is an image type process knowledge triple set, and TM is a three-dimensional model type process knowledge triple set. Each triple can be expressed as T = {(s, p, o) | Ti E (TW U TV U TM), i = 1, 2, …, n}, s, p and o represent subject, predicate and object respectively;

[0059] Based on the customized furniture plate processing knowledge graph and the customized furniture plate processing real data, the entity set is defined to include three types of customized furniture plate processing data, namely text entity EW, image entity EV and three-dimensional model entity EM. EW usually includes assembly process AC, assembly process AP, assembly process AS, assembly part AE and assembly tool AT, that is, EW = {AC U AP U AS U AE U AT}; EV usually includes assembly part graph VC and assembly sequence graph VS, that is, EV = {VC U VS}; and EM usually includes structure feature MS and geometric feature MF, that is, EM = {MS U MF};

[0060] Based on the customized furniture plate processing knowledge graph and the customized furniture plate processing real data, the relationship set is defined to include three types of assembly process data, namely structure relationship R st , time sequence relationship R tm and operation relationship R op , that is, R = {R st U R tm U R op}. The structure relationship R st represents the inherent connection between nodes, the time sequence relationship R tm represents the implicit sequence relationship between different nodes, and the operation relationship R op represents the specific operation of the process.

[0061] Based on the customized furniture plate processing knowledge graph and the customized furniture plate processing real data, the attribute set is defined as a kind of inherent characteristic, which can complement the semantics of entity and relationship, and strengthen the representation ability. The basic format is (entity, attribute name, attribute value), and the attribute value can be a common numerical type, date type or text type.

[0062] Further, the knowledge extraction of the customized furniture plate processing knowledge graph data set includes: for structured data knowledge extraction, selecting relevant data from the enterprise database as the basis for knowledge extraction; through entity, relationship and attribute matching, the data is converted into a triple form conforming to the knowledge extraction task;

[0063] For the knowledge extraction of three-dimensional model data in semi-structured json format, analyze the structure and attributes of json data, and identify entities therein; extract attribute information of entities from json data; and construct knowledge triples according to the relationship between entities.

[0064] For knowledge extraction of text data, use the BERT+BiLSTM+CRF model for knowledge extraction.

[0065] Specifically, the input layer: the pre-processed text data set is input into the model.

[0066] BERT layer: use the pre-trained BERT model to capture the semantic, syntactic and contextual information of words, and convert the text into word vector representation.

[0067] BiLSTM layer: capture long-term dependencies in sentences through BiLSTM to optimize entity boundary recognition.

[0068] CRF layer: use the CRF layer to ensure the consistency of the entire sequence of labels and improve the accuracy of entity recognition.

[0069] Entity recognition: define a span representation matrix to represent the starting position of the entity, and obtain the scoring function S(i,j)=S(s i ,s j )+S(o i ,o j )+S(s i ,o i ∣p)+S(s j ,o j ∣p)

[0070] Where i,j is used to mark the starting position of the entity, s is the first entity, o is the tail entity, and p is the relationship.

[0071] Relationship recognition: introduce relative position encoding to capture the relative position relationship between words in the text, and better handle semantic and contextual dependency relationships.

[0072] Screening triples: set the scoring function, consider evaluation indicators such as accuracy and recall rate, select appropriate threshold for screening, and only extract triples with a score higher than the set threshold.

[0073] Further, the knowledge fusion of the customized furniture plate processing knowledge graph dataset includes: inputting the processed text data into the text encoder of CLIP to obtain the embedding vector of the text;

[0074] Input the processed image data into the image encoder of CLIP to obtain the embedding vector of the image;

[0075] The similarity score between the text and the image is calculated using the contrastive loss function of CLIP.

[0076] According to the similarity score, determine the text that best matches the image, and perform knowledge fusion.

[0077] After fusion, the processing: for each pair of matched text and image, merge or associate the corresponding knowledge triples. If there are multiple text descriptions matching the same image, you can choose the highest scoring text description or decide how to integrate these descriptions through certain rules (such as majority voting).

[0078] Knowledge storage is to store the fused knowledge triples into the graph database Neo4j, and use Cypher statements to construct, edit and query the knowledge graph.

[0079] Further, based on the customized furniture plate processing knowledge graph, the question and answer system includes:

[0080] Web front-end development. Based on the application scenario of custom furniture field, customize the design and development of user interface, provide the interface for user interaction with the system. Use HTML, CSS and JavaScript front-end technologies to realize page layout and interactive functions.

[0081] Backend service development. Use Python programming language and Node.js framework to realize the backend logic, develop the backend service, receive user's request and process. Realize the connection and query of graph database, execute query statement and obtain the information in knowledge graph.

[0082] Question input first analyzes the question, extracts the question text through BERT+BiLSTM model, then classifies the question according to the extracted entities and relationship types, and assembles the classification results into a dictionary.

[0083] According to the corresponding question type and entity type in the question, use the Aligner model for disambiguation and answer matching, and finally assemble the searched answer as the answer output for user reference or use.

[0084] The present application starts from scratch, and builds a multi-modal process knowledge graph of customized furniture plate processing based on the whole cycle data of customized furniture enterprises, including workshop process data, process files, basic information of enterprise personnel and equipment, etc., and builds an automatic question answering system that can answer various types of questions on this basis. Through detailed process knowledge graph, complex processing flow and material properties, etc. Information can be standardized and standardized, thereby reducing errors caused by human factors and improving production efficiency. The question answering system based on the knowledge graph can provide instant technical support for front-line workers and help them quickly solve problems encountered in the processing process. The present application is business-oriented, realizes the connection and fusion of multi-modal data generated in the processing process, and uniformly represents the whole cycle data of the customized furniture enterprise, thereby connecting the departments of the enterprise through data, realizing the sharing of field knowledge, and being suitable for data management and knowledge reuse of customized furniture enterprises. In terms of technical implementation, the present application uses Neo4j as a storage database, and completes knowledge question answering through traditional rule mode, finally uses Cypher query statement as the language of question answering search, and supports question answering service. In addition, the present application is easy to deploy, and the required data has been pre-stored in the JSON file. In the deployment process, only the database construction according to the running steps of the project can provide search service.

[0085] The above is only the preferred specific embodiment of the present application, but the protection scope of the present application is not limited to this, any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for answering questions about customized furniture board processing technology based on a multi-modal knowledge graph, characterized in that, The method comprises the following steps: According to the processing requirements, the production process is divided into different production stations, and different production stations are divided into multi-dimensional data, and a customized furniture plate processing workshop station task beat model is constructed according to the multi-dimensional data; Based on the multi-dimensional data, the customized furniture plate processing knowledge graph dataset is obtained by preprocessing; Based on the customized furniture plate processing knowledge graph dataset, the customized furniture plate processing knowledge graph and the entity set, the relationship set and the attribute set are defined; The knowledge extraction, knowledge fusion and knowledge storage processing are performed on the customized furniture plate processing knowledge graph dataset to obtain the knowledge graph data layer and construct the customized furniture plate processing knowledge graph; Based on the customized furniture plate processing knowledge graph, a question and answer system is constructed; The customized furniture plate processing knowledge graph dataset includes: Based on the customized furniture enterprise product life cycle management system PLM, the processing station is taken as a basic unit, the data is collected in hierarchical manner according to the basic data layer, the planning data layer, the processing procedure layer and the production state layer, and the processing task beat duration of each station is defined according to the production cycle of each station; For structured data, missing value completion, repeated value reduction, abnormal value processing and standardization are performed; for process files and quality inspection standard files, text and image extraction are performed, and text data is segmented, special word processing is performed, image data is characterized and labeled; three-dimensional models are fully sorted out and exported, model features are extracted by model semantic extraction algorithm and saved, and the customized furniture plate processing knowledge graph dataset is obtained; Defining the customized furniture plate processing knowledge graph and the entity set, the relationship set and the attribute set includes: The customized furniture plate processing multi-modal process knowledge graph is added to the traditional knowledge graph based on the modal knowledge; Based on the customized furniture plate processing knowledge graph and the customized furniture plate processing real data, the entity set is defined, which is further divided into three categories: text entity, image entity and three-dimensional model entity; Based on the customized furniture plate processing knowledge graph and the customized furniture plate processing real data, the relationship set is defined, which is divided into structure relationship, time sequence relationship and operation relationship according to the characteristics of assembly process data; Based on the customized furniture plate processing knowledge graph and the customized furniture plate processing real data, the attribute set is defined as a kind of inherent characteristic which can make semantic supplement to entities and relationships; The knowledge extraction of the customized furniture plate processing knowledge graph dataset includes: for structured data knowledge extraction, relevant data is selected from the enterprise database as the basis for knowledge extraction; through entity, relationship and attribute matching, the data is converted into triple form conforming to the knowledge extraction task; For semi-structured target format three-dimensional model data knowledge extraction, the structure and attributes of the target data are analyzed, and the entities are identified; the attribute information of the entities is extracted from the target data; according to the relationship between entities, knowledge triple is constructed; For text data knowledge extraction, BERT+BiLSTM+CRF model is used for knowledge extraction. The knowledge fusion on the customized furniture plate processing knowledge graph dataset comprises: inputting the processed text data into a text encoder of the CLIP to obtain an embedding vector of the text; inputting the processed image data into an image encoder of the CLIP to obtain an embedding vector of the image; using a contrast loss function of the CLIP to calculate a similarity score between the text and the image; determining the image most matched with the text according to the similarity score, and performing knowledge fusion.

2. The multi-modal knowledge graph based question answering method for customized furniture board processing process according to claim 1, wherein, The division of different production stations into multi-dimensional data comprises: According to the classification and arrangement of resources knowledge elements, the resources knowledge elements are divided into static resource data, plan resource data and dynamic resource data.

3. The multi-modal knowledge graph based question answering method for customized furniture board processing process according to claim 1, wherein, According to the multi-dimensional data, a customized furniture plate processing workshop station task beat model is constructed, which comprises: F i ={I i ,S i ,D i ,T} where I i is the static resource dataset, S i is the planned resource dataset, D i is the dynamic resource dataset, T is the set of processing station tact times.

4. The multi-modal knowledge graph based question answering method for customized furniture board processing process according to claim 1, wherein, Based on the customized furniture plate processing knowledge graph, a question and answer system is constructed, which comprises: based on the customized furniture field application scene, a user interface is designed and developed, providing an interface for user interaction with the system; Using Python programming language and Node.js framework to realize the back-end logic, develop the back-end service, receive the user's request and process; The question input first analyzes the question, extracts the question text, classifies the question according to the extracted entities and relationship types, and assembles the classification results into a dictionary; According to the corresponding question type and entity type in the question, the Aligner model is used for entity disambiguation and answer matching, the searched answer is assembled, and the assembled answer is output to the user for reference or use.

Images