Question and answer method of customized furniture board processing technology based on multi-modal knowledge graph

Abstract

The invention discloses a question and answer method for a customized furniture board processing technology based on a multi-modal knowledge graph, and the method comprises the steps: dividing a production process into different production stations according to a processing demand, dividing the different production stations into multi-dimensional data, and constructing a customized furniture board processing workshop station task beat model according to the multi-dimensional data; performing preprocessing based on the multi-dimensional data to obtain a customized furniture board processing knowledge graph data set; based on the customized furniture board processing knowledge graph data set, defining a customized furniture board processing knowledge graph, an entity set, a relation set and an attribute set; performing knowledge extraction, knowledge fusion and knowledge storage processing on the customized furniture board processing knowledge graph data set to obtain a knowledge graph data layer, and constructing a customized furniture board processing knowledge graph; and constructing a question and answer system based on the customized furniture board processing knowledge graph. According to the invention, the availability and value of the data are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of customized furniture board processing data management and representation, and in particular to a question-answering method for customized furniture board processing technology based on a multimodal knowledge graph. Background Art

[0002] The current custom furniture production process has a high degree of mechanization, but the raw material utilization rate is low and the production cycle is long. This is mainly due to the low utilization rate of data in the entire production cycle. During the actual operation of the production workshop, each production cycle generates a large amount of high-dimensional multi-source heterogeneous data. The product production process data is crucial to product quality. Nowadays, traditional research on production process data modeling mainly focuses on four aspects: object-based modeling, metadata-based modeling, ontology-based modeling, and process complex network modeling. Object-based modeling has good stability, but does not have a structured data description mechanism, and it is difficult to achieve the interconnection and interoperability of multi-source heterogeneous data. Metadata-based modeling uses self-describing files and simpler component-based design, with good language interoperability, but poor flexibility in knowledge organization of process data. Ontology-based modeling has a high degree of formalization, strong expression ability, can better support contextual reasoning, and is easy to share information. It can better express complex environments. Process-based complex network modeling can integrate production factors and processing flows. Although the above-mentioned types of data modeling have their own advantages, they all lack the expression of semantic relationships and temporal sequences between data, and are prone to problems such as information fragmentation and information islands in the production process. Moreover, there are multimodal process data such as text, images, and models in the entire life cycle of furniture board production. Therefore, it is urgent to propose a knowledge modeling method for multimodal processing data of customized furniture boards and build a corresponding process question-answering system. Summary of the invention

[0003] In order to solve the above technical problems, the present invention proposes a question-and-answer method for customized furniture board processing technology based on a multimodal knowledge graph, which lowers the usage threshold and reduces the extra workload caused by data differences, thereby improving the overall processing speed and consistency, improving decision-making efficiency and improving the availability and value of data.

[0004] To achieve the above objectives, the present invention provides a question-answering method for customized furniture board processing technology based on a multimodal knowledge graph, comprising:

[0005] Divide the production process into different production stations according to processing requirements, divide the different production stations into multi-dimensional data, and construct a station task rhythm model for a customized furniture panel processing workshop according to the multi-dimensional data;

[0006] Preprocessing is performed based on the multidimensional data to obtain a knowledge graph dataset for customized furniture board processing;

[0007] Based on the custom furniture board processing knowledge graph dataset, define the custom furniture board processing knowledge graph and entity set, relationship set, and attribute set;

[0008] Performing knowledge extraction, knowledge fusion and knowledge storage processing on the custom furniture board processing knowledge graph data set to obtain a knowledge graph data layer and construct a custom furniture board processing knowledge graph;

[0009] A question-answering system is constructed based on the custom furniture panel processing knowledge graph.

[0010] Optionally, different production stations are divided into multi-dimensional data including:

[0011] Resource knowledge elements are constructed for different production stations according to the classification of people, machines, materials, methods, and environments, and the resource knowledge elements are divided into static resource data, planned resource data, and dynamic resource data according to dimensions.

[0012] Optionally, constructing a station task rhythm model of a customized furniture panel processing workshop according to the multi-dimensional data includes:

[0013] F i = {I i ,S i ,D i ,T}

[0014] Among them, I i is a static resource dataset, S i is the planned resource dataset, D i is a dynamic resource data set, and T is a set of processing station cycle times.

[0015] Optionally, obtain the custom furniture board processing knowledge graph dataset including:

[0016] Based on the product lifecycle management system PLM of custom furniture enterprises, the processing station is taken as the basic unit, and data is collected hierarchically according to the basic data layer, planning data layer, processing process layer and production status layer. The cycle time of each station's processing task is defined according to the production cycle of each station;

[0017] For structured data, missing values ​​are filled, duplicate values ​​are deleted, outliers are processed, and standardization is performed. For process documents and quality inspection standard documents, text and images are extracted, and text data is segmented, sentence-by-sentence, and special words are processed. Image data is described and labeled. The three-dimensional model is comprehensively combed and exported, and model features are extracted and saved through the model semantic extraction algorithm to obtain a knowledge graph dataset for customized furniture panel processing.

[0018] Optionally, define the knowledge graph and entity set, relationship set, and attribute set of customized furniture board processing, including:

[0019] The multimodal process knowledge graph for customized furniture board processing adds modal knowledge to the traditional knowledge graph;

[0020] Based on the custom furniture board processing knowledge graph and the real data of custom furniture board processing, the entity set is defined, including further dividing the custom furniture board processing data into three categories: text entity, image entity, and three-dimensional model entity;

[0021] Based on the custom furniture board processing knowledge graph and the real data of custom furniture board processing, a relationship set is defined, including structural relationship, temporal relationship and operation relationship according to the characteristics of assembly process data;

[0022] Based on the custom furniture panel processing knowledge graph and the real data of custom furniture panel processing, defining attribute sets as an inherent characteristic can make semantic supplements to entities and relationships.

[0023] Optionally, performing knowledge extraction on the custom furniture board processing knowledge graph dataset includes: for structured data knowledge extraction, selecting relevant data from the enterprise database as the basis for knowledge extraction; converting the data into a triple form that meets the knowledge extraction task by matching entities, relationships, and attributes;

[0024] For knowledge extraction of 3D model data in semi-structured target format, analyze the structure and attributes of the target data and identify the entities therein; extract the attribute information of the entities from the target data; and construct knowledge triples based on the relationships between the entities;

[0025] For knowledge extraction from text data, the BERT+BiLSTM+CRF model is used for knowledge extraction.

[0026] Optionally, performing knowledge fusion on the custom furniture board processing knowledge graph dataset includes: inputting the processed text data into a text encoder of CLIP to obtain an embedding vector of the text;

[0027] Input the processed image data into the image encoder of CLIP to obtain the image embedding vector;

[0028] Use CLIP's contrastive loss function to calculate the similarity score between text and image;

[0029] According to the similarity score, the image that best matches the text is determined to perform knowledge fusion.

[0030] Optionally, based on the custom furniture board processing knowledge graph, constructing a question-answering system includes: customizing the design and development of a user interface based on application scenarios in the custom furniture field to provide an interface for users to interact with the system;

[0031] Use Python programming language and Node.js framework to implement backend logic, develop backend services, receive user requests and process them;

[0032] The question input is first analyzed, the question text is extracted, the question is classified according to the extracted entity and relationship types, and the classification results are assembled into a dictionary;

[0033] According to the corresponding question type and the entity type in the question, the Aligner model is used to perform entity disambiguation and answer matching, and the searched answers are assembled. After the assembly is completed, it is output as the answer to the user for reference or use.

[0034] Technical effects of the present invention:

[0035] (1) Domain data adaptation and flexible configuration: The present invention can automatically adapt to data from different fields and integrate them into the knowledge graph of customized furniture board processing in a structured manner, ensuring the effective use of data and maximizing its value. The present invention supports users to flexibly configure data according to their own needs, lowering the usage threshold, so that business personnel and even technicians with low programming skills can easily get started and complete complex process flow queries through simple interface operations.

[0036] (2) Data adaptability and standardized processing: The present invention is highly adaptable and can automatically adjust the processing method according to different types of data, ensuring high efficiency and accuracy even in the face of diverse inputs. It provides a set of standardized processing procedures for various types of data, reducing the extra workload caused by data differences, thereby improving the overall processing speed and consistency.

[0037] (3) Knowledge sharing and data interconnection: The present invention realizes knowledge sharing among departments within the enterprise by constructing a unified knowledge graph, promotes information flow, and helps improve decision-making efficiency; it integrates workshop process data, process documents, personnel information, and basic equipment information onto a single platform to form a data interconnection network, thereby improving the availability and value of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0039] Figure 1 It is a flowchart of a question-answering method for customized furniture board processing technology based on a multimodal knowledge graph according to an embodiment of the present invention. DETAILED DESCRIPTION

[0040] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0041] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0042] like Figure 1 As shown, this embodiment provides a question-answering method for customized furniture board processing technology based on a multimodal knowledge graph, including:

[0043] According to the processing requirements, the production process is divided into different production stations, and different production stations are divided into multi-dimensional data. Based on the multi-dimensional data, a task rhythm model of the station in the customized furniture panel processing workshop is constructed;

[0044] Based on multi-dimensional data preprocessing, a knowledge graph dataset of customized furniture board processing is obtained;

[0045] Based on the custom furniture board processing knowledge graph dataset, define the custom furniture board processing knowledge graph and entity set, relationship set, and attribute set;

[0046] Perform knowledge extraction, knowledge fusion and knowledge storage processing on the custom furniture board processing knowledge graph dataset to obtain the knowledge graph data layer and construct the custom furniture board processing knowledge graph;

[0047] Build a question-answering system based on the knowledge graph of customized furniture panel processing.

[0048] Furthermore, different production stations are divided into multi-dimensional data including:

[0049] Resource knowledge elements are constructed by classifying and organizing different production stations according to people, machines, materials, methods, and environment, and resource knowledge elements are divided into static resource data, planned resource data, and dynamic resource data according to dimensions.

[0050] Furthermore, the task rhythm model of the customized furniture panel processing workshop is constructed based on multi-dimensional data, including:

[0051] F i = {I i ,S i ,D i ,T}

[0052] Among them, I i is a static resource dataset, S i is the planned resource dataset, D iis a dynamic resource data set, and T is a set of processing station cycle times.

[0053] Specifically, based on processing requirements, the production process is divided into different production stations according to the similarity of processing technology, including main stations such as pressing, cutting, edge banding, hole arrangement, sorting, quality inspection, modification, cleaning and warehousing. Other stations related to the main stations are defined as subordinate sub-stations. The data of different stations are classified and sorted according to the manufacturing resource knowledge elements of people, machines, materials, methods, and environments, and the data are divided into static resource data, planned resource data, and dynamic resource data according to dimensions. In order to achieve unified representation and update of multi-dimensional data, the processing station rhythm is defined based on the actual working cycle of each station processing task, and the static resource data, planned data, and dynamic process data resources are integrated to establish the station task rhythm model F of the customized furniture board processing workshop. i .

[0054] Furthermore, the knowledge graph dataset for processing customized furniture panels is obtained, which specifically includes: based on the PLM product lifecycle management system of customized furniture enterprises, taking the processing station as the basic unit, collecting data in layers according to the basic data layer, the planning data layer, the processing procedure layer and the production status layer, and defining the processing task cycle of each station according to the production cycle of each station. Among them, the basic data layer embeds static resource data including equipment and tools, the planning data layer embeds processing plan data of each station including processing batches, orders, etc., the processing procedure layer embeds the processing operation content in the process document, and the production status layer embeds real-time changing sensors, data acquisition instruments, etc. Store relevant data in the time series database according to different collection intervals; different data types use different data preprocessing operations. For structured data, missing value completion, duplicate value deletion, outlier processing, and standardization are performed; for process documents, quality inspection standards, etc., text and image extraction are performed on pdf and docx files, and text data is segmented, sentence-by-sentence, and special word processing is performed on the text data; feature description and annotation are performed on the image data; the three-dimensional model is comprehensively combed and exported as a glTF file, and the model features are extracted through the model semantic extraction algorithm and stored in the JSON file to obtain the customized furniture panel processing knowledge graph dataset.

[0055] Furthermore, the knowledge graph and entity set, relationship set, and attribute set of customized furniture board processing are defined, including:

[0056] The multimodal process knowledge graph of customized furniture board processing adds modal knowledge to the traditional knowledge graph, which is expressed as follows:

[0057] MPKG={E,R,A,TW,TV,TM}

[0058] Where E represents all entity sets, R represents all relationship sets, A represents all attribute sets, TW is a set of text-based process knowledge triples, TV is a set of image-based process knowledge triples, and TM is a set of 3D model-based process knowledge triples. Each triple can be expressed as T = {(s, p, o) | Ti∈(TW∪TV∪TM), i = 1, 2, …, n}, where s, p, and o represent the subject, predicate, and object, respectively;

[0059] Based on the knowledge graph of customized furniture board processing and the real data of customized furniture board processing, the entity set is defined, including the further division of customized furniture board processing data into three categories: text entity EW, image entity EV, and three-dimensional model entity EM; EW usually includes assembly process AC, assembly process AP, assembly step AS, assembly part AE, and assembly tool AT, that is, EW = {AC∪AP∪AS∪AE∪AT}; EV usually includes assembly part drawing VC and assembly sequence diagram VS, that is, EV = {VC∪VS}; EM usually includes structural features MS and geometric features MF, that is, EM = {MS∪MF};

[0060] Based on the knowledge graph of customized furniture board processing and the real data of customized furniture board processing, the definition of the relationship set includes structural relations R according to the characteristics of assembly process data. st , timing relationship R tm and the operation relation R op , that is, R = {R st ∪R tm ∪R op}. Structural relationship R st Represents the inherent connection between nodes, the temporal relationship R tm Represents the implicit order relationship between different nodes, the operation relationship R op Represents the specific operation of the process;

[0061] Based on the knowledge graph of customized furniture panel processing and the real data of customized furniture panel processing, the attribute set is defined as an inherent characteristic that can make semantic supplements to entities and relationships and enhance the representation ability. The basic format is (entity, attribute name, attribute value). The attribute value can be a common numeric type, date type, or text type.

[0062] Furthermore, knowledge extraction of the custom furniture board processing knowledge graph dataset includes: for structured data knowledge extraction, relevant data is selected from the enterprise database as the basis for knowledge extraction; through entity, relationship, and attribute matching, the data is converted into a triple form that meets the knowledge extraction task;

[0063] For knowledge extraction of 3D model data in semi-structured JSON format, analyze the structure and attributes of JSON data and identify the entities in it; extract the attribute information of entities from JSON data; and construct knowledge triples based on the relationship between entities;

[0064] For knowledge extraction from text data, the BERT+BiLSTM+CRF model is used for knowledge extraction.

[0065] Specifically, input layer: passes the preprocessed text dataset into the model.

[0066] BERT layer: Use the pre-trained BERT model to capture the semantic, syntactic, and contextual information of words and convert the text into word vector representation.

[0067] BiLSTM layer: BiLSTM is used to capture long-term dependencies in sentences and optimize entity boundary recognition.

[0068] CRF layer: Use the CRF layer to ensure label consistency across the entire sequence and improve the accuracy of entity recognition.

[0069] Entity recognition: Define the span representation matrix to represent the starting position of the entity, and obtain the scoring function S(i,j)=S(s i ,s j )+S(o i ,o j )+S(s i ,o i |p)+S(s j ,o j ∣p)

[0070] Among them, i and j are used to mark the starting position of the entity, s is the first entity, o is the last entity, and p is the relationship.

[0071] Relation identification: Relative position encoding is introduced to capture the relative position relationship between words in the text and better handle semantic and contextual dependencies.

[0072] Filter triplets: Set a scoring function, comprehensively consider evaluation indicators such as accuracy and recall rate, select an appropriate threshold for filtering, and only extract triplets with scores higher than the set threshold.

[0073] Furthermore, the knowledge fusion of the custom furniture board processing knowledge graph dataset includes: inputting the processed text data into the text encoder of CLIP to obtain the text embedding vector;

[0074] Input the processed image data into the image encoder of CLIP to obtain the image embedding vector;

[0075] Use CLIP's contrastive loss function to calculate the similarity score between text and image;

[0076] Based on the similarity score, the image that best matches the text is determined and knowledge fusion is performed.

[0077] Post-fusion processing: For each pair of matching text and image, merge or associate the corresponding knowledge triples. If there are multiple text descriptions matching the same image, select the text description with the highest score or use some rules (such as majority voting) to decide how to integrate these descriptions.

[0078] Knowledge storage is to store the fused knowledge triples in the graph database Neo4j, and use Cypher statements to build, edit and query the knowledge graph.

[0079] Furthermore, based on the knowledge graph of customized furniture board processing, the question-answering system is constructed including:

[0080] Web front-end development. Customized design and development of user interfaces based on application scenarios in the field of custom furniture, providing an interface for users to interact with the system. Use front-end technologies such as HTML, CSS, and JavaScript to implement page layout and interactive functions.

[0081] Backend service development. Use Python programming language and Node.js framework to implement backend logic, develop backend services, receive user requests and process them. Implement connection and query with graph database, execute query statements and obtain information in knowledge graph.

[0082] The question input first analyzes the question, extracts the question text through the BERT+BiLSTM model, then classifies the question according to the extracted entity and relationship types, and assembles the classification results into a dictionary.

[0083] According to the corresponding question type and the entity type in the question, the Aligner model is used to perform disambiguation and answer matching. Finally, the searched answers are assembled and output as answers to the user for reference or use.

[0084] The present invention starts from scratch and builds a multimodal process knowledge map for customized furniture board processing based on the full-cycle data of customized furniture enterprises, including workshop process data, process documents, basic information of enterprise personnel and equipment, etc., and builds an automatic question-and-answer system that can answer various types of questions on this basis. Through a detailed process knowledge map, complex processing flow and material properties and other information can be standardized and normalized, thereby reducing errors caused by human factors and improving production efficiency. The question-and-answer system based on the knowledge map can provide instant technical support for front-line workers to help them quickly solve problems encountered during the processing. The present invention is business-oriented, realizes the connection and integration of multimodal data generated during the processing, and uniformly characterizes the full-cycle data of customized furniture enterprises, thereby interconnecting various departments of the enterprise through data, realizing the sharing of domain knowledge, and is suitable for data management and knowledge reuse of customized furniture enterprises. In terms of technical implementation, the present invention uses Neo4j as a storage database, completes knowledge question-and-answer through traditional rule methods, and finally uses Cypher query statements as the language for question-and-answer search to support question-and-answer services. In addition, the present invention is easy to deploy quickly, and the required data has been pre-stored in a JSON file. During the deployment process, you only need to follow the project's running steps to complete the database setup and provide search services.

[0085] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A question-answering method for customized furniture board processing technology based on multimodal knowledge graph, characterized in that: include: Divide the production process into different production stations according to processing requirements, divide the different production stations into multi-dimensional data, and construct a station task rhythm model for a customized furniture panel processing workshop according to the multi-dimensional data; Preprocessing is performed based on the multidimensional data to obtain a knowledge graph dataset for customized furniture board processing; Based on the custom furniture board processing knowledge graph dataset, define the custom furniture board processing knowledge graph and entity set, relationship set, and attribute set; Performing knowledge extraction, knowledge fusion and knowledge storage processing on the custom furniture board processing knowledge graph data set to obtain a knowledge graph data layer and construct a custom furniture board processing knowledge graph; A question-answering system is constructed based on the custom furniture panel processing knowledge graph.

2. The question-answering method for customized furniture board processing technology based on multimodal knowledge graph as claimed in claim 1 is characterized in that: Divide different production stations into multi-dimensional data including: Resource knowledge elements are constructed for different production stations according to the classification of people, machines, materials, methods, and environments, and the resource knowledge elements are divided into static resource data, planned resource data, and dynamic resource data according to dimensions.

3. The question-answering method for customized furniture board processing technology based on multimodal knowledge graph as claimed in claim 1 is characterized in that: The station task rhythm model of the customized furniture panel processing workshop is constructed based on the multi-dimensional data, including: F i ={I i ,S i ,D i ,T} Among them, I i is a static resource dataset, S i is the planned resource dataset, D i is a dynamic resource data set, and T is a set of processing station cycle times.

4. The question-answering method for customized furniture board processing technology based on multimodal knowledge graph as claimed in claim 1, characterized in that: The knowledge graph dataset for custom furniture panel processing includes: Based on the PLM management system of the product lifecycle of custom furniture enterprises, the processing station is taken as the basic unit, and data is collected hierarchically according to the basic data layer, the planning data layer, the processing procedure layer and the production status layer. The cycle time of the processing task of each station is defined according to the production cycle of each station. For structured data, missing values ​​are filled, duplicate values ​​are deleted, outliers are processed, and standardization is performed. For process documents and quality inspection standard documents, text and images are extracted, and text data is segmented, sentence-by-sentence, and special words are processed. Image data is described and labeled. The three-dimensional model is comprehensively combed and exported, and model features are extracted and saved through the model semantic extraction algorithm to obtain a knowledge graph dataset for customized furniture panel processing.

5. The question-answering method for customized furniture board processing technology based on multimodal knowledge graph as claimed in claim 1, characterized in that: The knowledge graph and entity set, relationship set, and attribute set defined for custom furniture panel processing include: The multimodal process knowledge graph for customized furniture board processing adds modal knowledge to the traditional knowledge graph; Based on the custom furniture board processing knowledge graph and the real data of custom furniture board processing, the entity set is defined, including further dividing the custom furniture board processing data into three categories: text entity, image entity, and three-dimensional model entity; Based on the custom furniture board processing knowledge graph and the real data of custom furniture board processing, a relationship set is defined, including structural relationship, temporal relationship and operation relationship according to the characteristics of assembly process data; Based on the custom furniture panel processing knowledge graph and the real data of custom furniture panel processing, defining attribute sets as an inherent characteristic can make semantic supplements to entities and relationships.

6. The question-answering method for customized furniture board processing technology based on multimodal knowledge graph as claimed in claim 1, characterized in that: The knowledge extraction of the custom furniture board processing knowledge graph dataset includes: for structured data knowledge extraction, relevant data is selected from the enterprise database as the basis for knowledge extraction; by matching entities, relationships, and attributes, the data is converted into a triple form that meets the knowledge extraction task; For knowledge extraction of 3D model data in semi-structured target format, analyze the structure and attributes of the target data and identify the entities therein; extract the attribute information of the entities from the target data; and construct knowledge triples based on the relationships between the entities; For knowledge extraction from text data, the BERT+BiLSTM+CRF model is used for knowledge extraction.

7. The question-answering method for customized furniture board processing technology based on multimodal knowledge graph as claimed in claim 1, characterized in that: The knowledge fusion of the custom furniture board processing knowledge graph dataset includes: inputting the processed text data into the text encoder of CLIP to obtain the text embedding vector; Input the processed image data into the image encoder of CLIP to obtain the image embedding vector; Use CLIP's contrastive loss function to calculate the similarity score between text and image; According to the similarity score, the image that best matches the text is determined to perform knowledge fusion.

8. The question-answering method for customized furniture board processing technology based on multimodal knowledge graph as claimed in claim 1, characterized in that: Based on the custom furniture board processing knowledge graph, constructing a question-answering system includes: customizing the design and development of a user interface based on the application scenarios in the custom furniture field, and providing an interface for users to interact with the system; Use Python programming language and Node.js framework to implement backend logic, develop backend services, receive user requests and process them; The question input is first analyzed, the question text is extracted, the question is classified according to the extracted entity and relationship types, and the classification results are assembled into a dictionary; According to the corresponding question type and the entity type in the question, the Aligner model is used to perform entity disambiguation and answer matching, and the searched answers are assembled. After the assembly is completed, it is output as the answer to the user for reference or use.

Images