Manufacturing industry knowledge intelligent evaluation generation system and method based on RAG enhanced retrieval generation

Abstract

The invention belongs to the technical field of artificial intelligence, and particularly discloses a manufacturing industry knowledge intelligent evaluation generation system and method based on RAG enhanced retrieval generation, the system adopts a cloud-side-end collaborative architecture, and comprises a cloud server cluster, edge equipment, a data layer, an evaluation core layer, an intelligent model layer and an application layer, the data layer is used for storing and managing a static knowledge base, dynamic data and a benchmark test set in the manufacturing industry; and the evaluation core layer comprises a retrieval accuracy evaluation module, a generation quality evaluation module, a knowledge traceability evaluation module and a timeliness evaluation module. According to the method, the problem of insufficient retrieval accuracy is effectively solved through a manufacturing industry special benchmark test set and intelligent model optimization; quality evaluation is generated through multiple dimensions, so that the professionality and reliability of the generated content are improved; through a knowledge graph and a GraphSAGE model, precise traceability of a generation result is realized; and timely response of dynamic knowledge is guaranteed through an incremental update detection mechanism.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a manufacturing knowledge intelligent evaluation and generation system and method based on RAG enhanced retrieval. Background Technology

[0002] In the process of digital transformation in manufacturing, knowledge-enhanced intelligent applications (such as product technical Q&A systems, intelligent process guidance, and fault solution generation) have become key tools for improving production efficiency and reducing operation and maintenance costs. These applications are typically based on a Retrieval-Augmented Generation (RAG) architecture, which retrieves knowledge from specialized data sources (such as process libraries, standard libraries, and fault case libraries) to assist in generating accurate and professional responses. However, the application of existing RAG systems in manufacturing scenarios still faces the following pain points.

[0003] Insufficient search accuracy: Manufacturing knowledge is highly specialized (such as terminology standards and process logic), and traditional keyword searches struggle to capture semantic relationships (such as the causal relationship between "excessive welding current" and "excessive molten pool temperature"), resulting in poor relevance of search results.

[0004] The quality of generated content varies greatly: the generation module is prone to "hallucination," which means that the generated content contradicts the retrieved knowledge or does not conform to manufacturing standards (such as mistakenly writing "deep groove ball bearing" instead of "angular contact ball bearing"), affecting the reliability of the application.

[0005] Difficulty in knowledge traceability: The generated results cannot be clearly traced back to the source document, making it difficult to locate the root cause of the problem when errors occur (such as which version of the process specification the "maintenance steps" come from), which is not conducive to knowledge iteration.

[0006] Lagging in timeliness: Manufacturing technical standards (such as ISO and GB) and process specifications are frequently updated, and existing systems are unable to respond quickly to dynamic knowledge changes (such as the "Welding Procedure Qualification Specification" updated in 2024 not being included in the retrieval database in a timely manner), resulting in outdated generated content.

[0007] In addition, existing RAG evaluation methods are mostly aimed at general scenarios (such as open domain question answering) and lack the specificity of manufacturing industry: (1) the evaluation indicators do not cover dimensions such as "process professionalism" and "standard compliance"; (2) the benchmark test set lacks real-world scenario data in manufacturing industry (such as workshop failure cases and process change records); (3) the low latency requirements of edge devices (such as workshop smart terminals) are not considered. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a manufacturing knowledge intelligent evaluation and generation system and method based on RAG enhanced retrieval. By optimizing the manufacturing-specific benchmark test set and intelligent model, it effectively solves the problem of insufficient retrieval accuracy; by improving the professionalism and reliability of the generated content through multi-dimensional generation quality assessment; by achieving accurate traceability of the generated results through knowledge graph and GraphSAGE model; and by ensuring timely response of dynamic knowledge through incremental update detection mechanism.

[0009] Firstly, the technical solution to achieve the above objectives is: a manufacturing knowledge intelligent evaluation and generation system and method based on RAG enhanced retrieval, including a cloud-edge-device collaborative architecture, comprising: A cloud server cluster, wherein the cloud server cluster includes at least a computing server, a storage server, and a retrieval server; Edge devices include smart terminals deployed on the production site and IoT gateways for connecting workshop equipment. The edge devices are configured to collect and upload manufacturing site data. The data layer, deployed on the cloud server cluster, is used to store and manage the manufacturing static knowledge base, dynamic data, and benchmark test sets. The core evaluation layer, deployed on the cloud server cluster and communicating with the data layer, includes a retrieval accuracy evaluation module, a generation quality evaluation module, a knowledge tracing evaluation module, and a timeliness evaluation module. The intelligent model layer, deployed on the cloud server cluster, provides algorithmic support for the evaluation core layer. The intelligent model layer includes at least one language model pre-trained using corpus from the manufacturing sector. The application layer communicates with the core evaluation layer and is configured to provide an evaluation interaction interface and display evaluation results.

[0010] Secondly, this invention provides a method for intelligent evaluation and generation of manufacturing knowledge based on RAG enhanced retrieval, which uses the intelligent evaluation and generation system for manufacturing knowledge as described in claim 1 to generate intelligent evaluation and generation of manufacturing knowledge, and includes the following steps: Receive an evaluation request for the target retrieval enhancement generation system, wherein the evaluation request contains at least the query content to be evaluated; Based on the query content, the search results and related reference knowledge of the target retrieval enhancement generation system are obtained from the data layer; The evaluation core layer is invoked to evaluate the performance of the target retrieval enhancement generation system under the query in multiple dimensions based on the retrieval results and reference knowledge. The multi-dimensional evaluation includes at least retrieval accuracy evaluation and generation quality evaluation. The application layer outputs evaluation results containing assessment metrics for each dimension.

[0011] The aforementioned intelligent evaluation and generation method for manufacturing knowledge based on RAG enhanced retrieval includes the following process for evaluating retrieval accuracy: Obtain the initial set of search result documents returned by the target retrieval enhancement generation system for the query; The domain pre-trained language model in the intelligent model layer is invoked to calculate the semantic similarity between the query and each document in the initial retrieval result document set; Based on the semantic similarity, the initial search result document set is reordered or filtered to obtain optimized search results; Based on the optimized search results, a search accuracy index is calculated, which includes at least one of recall, precision, or F1 score.

[0012] The aforementioned intelligent evaluation and generation method for manufacturing knowledge based on RAG enhanced retrieval, wherein the domain pre-trained language model in the intelligent model layer is obtained through the following method: Collect domain corpus containing manufacturing standards, process documents, and failure cases; The base language model is pre-trained in a domain-adaptive manner using a masked language model and a next-sentence prediction task to optimize the model’s ability to understand manufacturing terminology and logical relationships.

[0013] The aforementioned intelligent evaluation and generation method for manufacturing knowledge based on RAG enhanced retrieval includes the following process for quality assessment: Obtain the response text generated by the target retrieval enhancement generation system based on the retrieval results; Calculate the objective text matching index between the response text and the reference document, wherein the objective text matching index includes BLEU value or ROUGE value; The generative quality assessment model in the intelligent model layer is invoked to subjectively score the response text, and the subjective score includes at least an accuracy score and / or a professional score. The objective text matching index and the subjective score are weighted and fused to obtain a comprehensive quality score.

[0014] The aforementioned intelligent evaluation and generation method for manufacturing knowledge based on RAG enhanced retrieval includes a multi-dimensional evaluation that further incorporates knowledge source tracing evaluation, specifically comprising the following steps: Extract key entities from the response text generated by the target retrieval enhancement generation system; In the pre-built manufacturing knowledge graph, query the source document information and knowledge paths associated with the key entities; The similarity between the knowledge path of an entity in the response text and the knowledge path of the corresponding entity in the source document is calculated based on the graph neural network model, and the consistency of the source is determined based on the similarity.

[0015] The above-mentioned intelligent evaluation and generation method for manufacturing knowledge based on RAG enhanced retrieval includes a manufacturing knowledge graph constructed based on dynamically updated manufacturing documents, where documents, terms, and standards are abstracted as nodes, and the relationships between nodes are abstracted as edges; the graph neural network model uses an aggregation function to aggregate the neighbor information of nodes to generate the embedded representation of nodes for calculating path similarity.

[0016] The aforementioned intelligent evaluation and generation method for manufacturing knowledge based on RAG enhanced retrieval includes a timeliness assessment as part of the multi-dimensional evaluation, which specifically comprises the following steps: Obtain the publication timestamp of the source document on which the target retrieval enhancement generation system uses to generate a response; Obtain the latest valid version timestamp of the technical standard or specification corresponding to the source document; Compare the publication timestamp with the latest valid version timestamp to determine whether the knowledge referenced in the generated result is the latest version; Calculate the response latency from standard update to synchronous update of generated results.

[0017] The aforementioned intelligent evaluation and generation method for manufacturing knowledge based on RAG enhanced retrieval, wherein the timeliness assessment further includes an incremental update detection sub-step, specifically: Periodically calculate hash values ​​for source documents in the data layer; Compare the currently calculated hash value with the historically stored hash values; If the hash value changes, it is determined that the corresponding source document has been updated, and the target retrieval enhancement generation system is triggered to re-retrieve the document to update the generated results.

[0018] The aforementioned intelligent evaluation and generation method for manufacturing knowledge based on RAG enhanced retrieval further includes a closed-loop optimization step, specifically comprising the following process: After outputting the evaluation results containing evaluation indicators of each dimension through the application layer, the specific problems of the target retrieval enhancement generation system in each evaluation dimension are analyzed based on the evaluation results. Generate targeted optimization suggestions, which include at least one of the following: adjusting the semantic similarity threshold of the retrieval model, optimizing the prompt template of the generated model, updating the knowledge graph data, or increasing the update detection frequency of the data source; The optimization suggestions are applied to the optimization process of the target retrieval enhancement generation system, and the evaluation is re-initiated based on the optimized system, forming an evaluation-optimization iterative closed loop.

[0019] Thirdly, the present invention provides a computer device, the computer including a processor and a memory, the memory storing computer-readable instructions, and the processor executing the computer-readable instructions to implement the steps of the above-described intelligent evaluation and generation method for manufacturing knowledge based on RAG enhanced retrieval.

[0020] Fourthly, the present invention provides a computer-readable storage medium storing at least one computer-readable instruction, which is loaded and executed by a processor to implement the steps of the above-described intelligent evaluation and generation method for manufacturing knowledge based on RAG enhanced retrieval.

[0021] The present invention provides a systematic solution to the four major pain points of existing RAG systems in the background art by providing an intelligent evaluation and generation system and method for manufacturing knowledge based on RAG enhanced retrieval: effectively solving the problem of insufficient retrieval accuracy through manufacturing-specific benchmark test sets and intelligent model optimization; improving the professionalism and reliability of generated content through multi-dimensional generation quality assessment; achieving accurate traceability of generated results through knowledge graphs and GraphSAGE models; and ensuring timely response of dynamic knowledge through incremental update detection mechanisms. Attached Figure Description

[0022] Figure 1 This is a hardware structure diagram of the intelligent evaluation and generation system for manufacturing knowledge based on RAG enhanced retrieval, as presented in this invention. Figure 2 This is a software architecture diagram of the intelligent evaluation and generation system for manufacturing knowledge based on RAG enhanced retrieval, as described in this invention. Figure 3 This is a flowchart of the intelligent evaluation and generation method for manufacturing knowledge based on RAG enhanced retrieval, as described in this invention. Figure 4 This is a schematic diagram of the evaluation process of the intelligent process guidance system for manufacturing provided as an example. Detailed Implementation

[0023] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0024] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0025] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0026] In this embodiment of the invention, sometimes a subscript such as W1 may be mistakenly written as a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0027] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0028] This invention provides a basis for optimizing knowledge-enhanced intelligent applications.

[0029] This invention can meet the needs of knowledge-enhanced applications in the manufacturing industry by providing comprehensive evaluation capabilities for retrieval-enhanced generation (RAG) systems.

[0030] This invention supports multi-dimensional evaluation of RAG applications in typical manufacturing scenarios such as technical document Q&A, process specification query, and fault solution generation, including: The accuracy of the retrieval was evaluated to verify the system's accuracy in retrieving relevant technical information from professional data sources such as process libraries and standard libraries. Generate quality assessments to test the accuracy and professionalism of the technical solutions, operation guides, and other content generated by the system based on the retrieved content; Knowledge traceability capability assessment verifies the traceability and consistency between the generated results and the source documents; Timeliness assessment verifies the system's ability to respond promptly to dynamic knowledge such as updated technical standards and process changes.

[0031] The application of this invention will promote the popularization of knowledge-enhanced intelligent applications in the manufacturing industry and provide reliable technical support for enterprise digital transformation.

[0032] The application principles and embodiments of this invention will be described in detail below with reference to the system implementing this method.

[0033] This invention proposes a manufacturing knowledge intelligent evaluation and generation system and method based on RAG enhanced retrieval. By constructing a manufacturing-specific benchmark test set and combining multi-dimensional evaluation modules and intelligent model optimization, it achieves a comprehensive and accurate evaluation of the RAG system, providing a basis for its optimization.

[0034] Please see Figure 1 , Figure 2 and Figure 3 The intelligent evaluation and generation system for manufacturing knowledge based on RAG enhanced retrieval, as described in this invention, adopts a "cloud-edge-device" collaborative architecture, covering the entire process of data collection, evaluation calculation, and result presentation, and supports the high concurrency and low latency requirements in manufacturing scenarios. The system's hardware and software composition principles are as follows: I. Hardware Components (e.g.) Figure 1 (as shown) 1. Cloud server: Compute Server: Used to run intelligent models (such as BERT, GPT-4, graph neural networks), possessing powerful parallel computing capabilities. Recommended configuration: NVIDIA A100 GPU server (8 A100 80GB GPUs, 2TB memory, 40TB SSD storage), supporting accelerated model training and inference.

[0035] Storage servers: Used to store massive amounts of manufacturing data (such as process documents, standards and specifications, test suites), featuring high reliability and scalability. Recommended configuration: Distributed storage system (such as Ceph), employing tiered storage of "hot data + cold data" (hot data stored on SSDs, cold data stored on HDDs), with a total capacity of no less than 100TB.

[0036] Search Server: Supports low-latency retrieval and provides efficient full-text search capabilities. Recommended configuration: Elasticsearch cluster (3 nodes, each node equipped with 32GB of memory and 4TB of SSD, supporting sharding and replication mechanisms) to achieve millisecond-level search response.

[0037] 2. Edge devices: Workshop Smart Terminal: Deployed on the production site (e.g., next to machine tools, assembly lines) to collect real-time data (e.g., IoT sensor data, worker operation records) and initiate evaluation requests. Recommended configuration: Industrial-grade tablet (equipped with Snapdragon 8 Gen2 processor, 16GB RAM, 512GB storage, supports 5G / Wi-Fi 6), with dustproof and waterproof (IP65) characteristics.

[0038] Data acquisition, processing and reporting process: (1) Acquisition: Real-time data is collected through built-in sensors (such as temperature and vibration sensors) or external IoT devices (worker operation records are entered via touch input and voice input; sensor data is received via Bluetooth / Wi-Fi); (2) Preprocessing: Noise filtering (such as removing abnormal sensor values) and format standardization (converting unstructured operation records into JSON format) are performed on the collected data; (3) Reporting: Data is uploaded to the IoT gateway (or directly to the cloud) via 5G / Wi-Fi 6 first, and supports breakpoint resume (data is cached to local storage when the network is interrupted).

[0039] IoT Gateway: Used to connect workshop equipment (such as machine tools, robots, and sensors), collect real-time data (such as spindle temperature and current values), and upload it to the cloud. Recommended configuration: Huawei AGV IoT Gateway (supports Modbus and OPC UA protocols, and can connect up to 1000 devices).

[0040] Data acquisition, processing and reporting process: (1) Protocol conversion: Convert the Modbus / OPC UA protocol data of the workshop equipment into the cloud-compatible MQTT / HTTP protocol; (2) Data cleaning: Remove duplicate data and correct format errors; (3) Caching and forwarding: Use Redis to cache temporary data (caching time 1 hour), and upload it to the cloud storage server in batches when the network is stable; (4) Anomaly reporting: When the equipment data exceeds the threshold (such as spindle temperature > 90℃), trigger a real-time alarm and upload it to the cloud first.

[0041] 3. Network equipment: Core switch: Used for network connectivity between cloud and edge devices, featuring high bandwidth and low latency. Recommended configuration: Huawei S12700E series switch (supports 100Gbps ports, forwarding rate 1760Mpps).

[0042] 5G base station: Used for wireless communication of edge devices in the workshop, supporting massive device access and low-latency transmission (such as real-time uploading of fault alarm data). Recommended configuration: ZTE 5G AAU (supports 3.5GHz band, maximum downlink speed of 2.5Gbps).

[0043] The hardware communication connection method is as follows: 1. Cloud-Edge Connectivity: The cloud server connects to the 5G base station / wired network through the core switch, and the edge devices (smart terminals, IoT gateways) access the network through 5G / Wi-Fi 6 to achieve high bandwidth (100Gbps) and low latency (<10ms) data transmission; retrieval requests are initiated first through the 5G network, and the evaluation results are pushed to the smart terminal via WebSocket.

[0044] 2. Edge connectivity: The workshop smart terminal and the IoT gateway communicate via Wi-Fi 6 or Bluetooth 5.3, supporting data exchange between devices (such as synchronizing operation records to the gateway); the IoT gateway connects to workshop equipment such as machine tools and sensors via Modbus / OPC UA protocol to collect real-time data.

[0045] 3. Protocol compatibility between devices: The cloud storage server and the retrieval server share data via the NFS protocol, and the computing server and the intelligent model layer call services via the gRPC protocol, ensuring efficient and compatible data transmission.

[0046] II. Software System Architecture Components: like Figure 2 As shown, the software architecture of this system is divided into a data layer, an evaluation core layer, an intelligent model layer, and an application layer. Each layer interacts with data through API interfaces.

[0047] Software architecture data interaction and control process: (1) Data layer → Evaluation core layer: Provide structured data (such as retrieved documents, knowledge graph entities) through RESTful API; (2) Evaluation core layer → Intelligent model layer: Call model services (such as semantic similarity calculation of ManufactureBERT) through gRPC; (3) Intelligent model layer → Evaluation core layer: Return model calculation results (such as similarity scores, entity link results); (4) Evaluation core layer → Application layer: Push real-time evaluation results through WebSocket; (5) Application layer → Data layer: Feed back user operation data (such as the adoption status of optimization suggestions) for data updates.

[0048] 1. Data Layer: The data layer is the foundation of the evaluation system, responsible for storing and preprocessing manufacturing knowledge data, including: Data sources: ① Static knowledge (process library, standard library, fault case library, such as the company's internal "Welding Process Specification V3.0" and GB / T 19001-2016 standard); ② Dynamic data (IoT sensor data, workshop operation records, such as "2024-05-01 14:30 Machine tool spindle temperature 85℃"); ③ Evaluation benchmark test set (manually annotated test cases, such as "Problem: Solution to excessive welding current; Expected search result: Section 5.2 of "Welding Process Specification V3.0"; Expected generated result: Adjust the current to 120A±5A").

[0049] The data preprocessing module is used to convert raw data into a structured format suitable for evaluation. Its main functions include: ① Data cleaning (removing duplicate documents and correcting typos, such as changing "welding" to "welding"); ② Entity extraction (using SpaCy to extract entities from documents, such as "7205 bearing" and "ISO 15614 standard"); ③ Knowledge labeling (using LabelStudio to label the relevance of search results and the accuracy of generated results, such as labeling "welding current adjustment method" as "relevant"); ④ Structured storage (storing the preprocessed data in MySQL (relational database), Elasticsearch (data retrieval), and Neo4j (knowledge graph)).

[0050] 2. Evaluation Core Layer: The evaluation core layer is the core logic module of the system, responsible for implementing multi-dimensional evaluation functions, including: Core layer data processing and control process: (1) Receive evaluation requests from the application layer (including query content and scenario type); (2) Obtain target data from the data layer (such as search results and reference documents); (3) Call the algorithm of the intelligent model layer (such as Manufacture BERT to calculate semantic similarity); (4) Integrate model results with the rule engine (such as manufacturing standard verification) to generate evaluation indicators; (5) Encapsulate the results in JSON format and return them to the application layer, and synchronize them to the data layer to store historical records.

[0051] The retrieval accuracy evaluation module calculates the relevance of retrieval results to the query, with metrics including recall, precision, and F1 score, and optimizes the ranking by combining semantic similarity.

[0052] The quality assessment module evaluates the accuracy, professionalism, and logic of the generated content, with indicators including BLEU (Bilingual Evaluation Study), ROUGE (Recall-Oriented Understudy for Gisting Evaluation), and GPT-4 subjective rating.

[0053] Knowledge traceability assessment module: Verifies the consistency between the generated results and the source documents, and traces the source of the generated content through knowledge graph (e.g., "Maintenance steps" comes from Section 3.4 of "Machine Tool Maintenance Manual V2.1").

[0054] Timeliness assessment module: Checks whether the generated results meet the latest technical standards. Indicators include "standard validity" (such as whether the source document is the current valid version) and "response latency" (such as the time from standard update to result update).

[0055] 3. Intelligent Model Layer: The intelligent model layer provides algorithmic support for the core evaluation layer, and the following models have been optimized for manufacturing scenarios: Manufacture BERT: A pre-trained model for the manufacturing industry based on BERT, used for semantic similarity calculation of retrieval results. The pre-training corpus includes 100GB of manufacturing text (ISO standards, GB standards, and enterprise process documents). The pre-training tasks are Masked Language Model (MLM) and Next Sentence Prediction (NSP), and terminology understanding capabilities (such as semantic representation of "melting depth" and "bevel angle") have been optimized.

[0056] GPT-4 Fine-tuned for Manufacturing: A fine-tuned model for quality assessment in the manufacturing industry. It takes "generated results + reference documents" as input and outputs "accuracy score (0-10 points)" and "professionalism score (0-10 points)" to solve the problem that objective indicators (such as BLEU) cannot capture professional logic.

[0057] Graph SAGE-based Knowledge Graph Embedding: A knowledge graph embedding model based on graph neural networks (GNNs) for entity linking and path finding in knowledge tracing. By aggregating neighbor node information (such as the relationship between "bearing" and "maintenance steps"), it improves the representational power of entity embedding and optimizes tracing accuracy.

[0058] 4. Application Layer: The application layer provides user interaction interfaces, including: Evaluation Dashboard: Visualizes evaluation results across various dimensions (such as retrieving F1 value trend charts and generating quality score distributions), and supports custom indicator filtering (such as filtering evaluation results for welding scenarios by "process type").

[0059] API Interface: Provides a RESTful API to support third-party systems (such as enterprise ERP and PLM) to call the evaluation function (such as "evaluating the generation quality of the intelligent process guidance system").

[0060] Report generation module: Automatically generates evaluation reports, including score overview, problem analysis (such as "Reasons for low retrieval accuracy: semantic similarity threshold set too low"), and optimization suggestions (such as "Adjust the fine-tune parameters of ManufactureBERT").

[0061] III. System Data Interaction Process: Please see Figure 3 The data interaction in this system follows a process of "acquisition-preprocessing-storage-evaluation-feedback", with the specific steps as follows: (a) Data Acquisition and Preprocessing: Data Acquisition: Static knowledge (process library, standard library) and dynamic data (workshop operation records, sensor data) are collected through edge devices (such as IoT gateways and smart terminals) and uploaded to the cloud via 5G / wired networks.

[0062] Hardware division of labor: The IoT gateway is responsible for the automatic collection of equipment data (such as spindle temperature and current); the smart terminal is responsible for collecting worker operation records and manually entered problem data; static knowledge is uploaded to the storage server by the administrator through the cloud backend.

[0063] Data preprocessing: Cleaning: Use Pandas to remove duplicate documents (such as multiple versions of the same process specification) and correct formatting errors (such as converting tables in PDFs to CSV).

[0064] Labeling: Use LabelStudio to label the relevance of search results (e.g., for the query "excessive welding current", label section 5.2 of "Welding Procedure Specification V3.0" as "relevant") and the accuracy of generated results (e.g., label "adjust current to 120A" as "accurate").

[0065] Structured approach: Use SpaCy to extract entities from the document (such as “7205 bearing” and “ISO 15614 standard”), and use Neo4j to build a knowledge graph (entity-relationship-attribute, such as “bearing” → “belongs to” → “machine tool spindle”, with the attribute “model: 7205”).

[0066] Data storage: The preprocessed data is stored in the corresponding databases: ① Relational data (such as evaluation benchmark test sets) is stored in MySQL; ② Retrieval data (such as process documents) is stored in Elasticsearch; ③ Knowledge graph data is stored in Neo4j.

[0067] (II) Evaluation Request Processing: Request Initiation: Users initiate evaluation requests through the evaluation Dashboard or API interface, specifying the evaluation object (e.g., "Intelligent Process Guidance System"), evaluation dimensions (e.g., "Retrieval Accuracy + Generation Quality"), and parameters (e.g., "Top 5 Retrieval Results" and "Generated Result Length 100-200 Characters").

[0068] Request distribution: The application layer forwards the request to the evaluation core layer, and the core layer calls the corresponding evaluation module (such as the retrieval accuracy evaluation module) according to the evaluation dimensions.

[0069] Module execution: Software interaction: Each evaluation module calls the services of the intelligent model layer (such as semantic similarity calculation of ManufactureBERT and path similarity calculation of GraphSAGE) through gRPC. After the model layer returns the calculation results, the module combines the manufacturing rule engine (such as standard verification) to generate the final indicators.

[0070] Retrieval accuracy assessment: Elasticsearch is called to retrieve the top 5 documents, and the semantic similarity (cosine similarity) between the query and each document is calculated using ManufactureBERT. After sorting, the top 3 are selected, and the recall, precision, and F1 score are calculated.

[0071] Generate quality assessment: Obtain the generated results and corresponding search documents, calculate the n-gram overlap rate using BLEU-4 (e.g., the overlap between "adjust current to 120A" and "current setting range 110-130A" in the reference document), calculate the longest common subsequence (LCS) using ROUGE-L, call the GPT-4 Fine-tuned model to output subjective scores, and combine objective and subjective scores (e.g., objective accounts for 40%, subjective accounts for 60%).

[0072] Knowledge source assessment: Extract entities from the generated results (e.g., “7205 bearing”), find the corresponding source document (e.g., Section 3.4 of “Machine Tool Maintenance Manual V2.1”) through Neo4j knowledge graph, calculate the path similarity between the generated entity and the source document entity using GraphSAGE (e.g., whether the path “7205 bearing” → “spindle assembly” → “maintenance steps” is consistent), and check for consistency.

[0073] Timeliness assessment: Obtain the timestamp of the source document (e.g., the release date of "Welding Process Specification V3.0" on 2024-03-01) and the timestamp of the generated result (e.g., 2024-04-01), and compare whether it is within the validity period (e.g., the standard validity period is 1 year); use an incremental update detection algorithm (e.g., hash-based document change detection) to check whether the source document has been updated, and calculate the response delay (e.g., the time from the standard update on 2024-03-15 to the update of the generated result on 2024-03-20).

[0074] Results Return: The core evaluation layer returns the results of each module (such as the F1 score of 0.85 and the quality score of 8.2) to the application layer.

[0075] Data synchronization: While the application layer displays the evaluation results to the user, it synchronizes the results to the MySQL database in the data layer (to store historical evaluation records) through an asynchronous message queue (such as Kafka) for subsequent analysis and optimization.

[0076] (III) Results Presentation and Feedback: Visualization: The application layer visualizes the results (e.g., using line charts to show the monthly trend of search accuracy, and using heatmaps to show the scene distribution of generated quality), and users can view detailed metrics through the Dashboard.

[0077] Report generation: The report generation module automatically generates an evaluation report, including: (1) a score overview (such as the average score of each dimension); (2) a problem analysis (such as "the reason for low retrieval accuracy: the semantic similarity threshold of ManufacturerBERT is set to 0.7, which causes some relevant documents to be filtered"); (3) optimization suggestions (such as "adjust the threshold to 0.65, or increase the case data in the process library").

[0078] Feedback and iteration: Users optimize the RAG system based on the report (such as adjusting retrieval model parameters and updating data sources) and re-initiate the evaluation, forming a closed loop of "evaluation-optimization-re-evaluation".

[0079] Software control and data interaction complement: User optimization operations are synchronized to the intelligent model layer (such as updating the ManufactureBERT threshold) and the data layer (such as uploading new process documents) through the application layer API; the data layer automatically triggers the preprocessing process (cleaning, labeling, structuring) and notifies the evaluation core layer to update the evaluation benchmark; when the user re-initiates the evaluation, the application layer directly calls the updated evaluation core layer service to realize the automated data flow of closed-loop iteration.

[0080] IV. Detailed introduction of key technology principles and algorithms: (a) Evaluation of retrieval accuracy: semantic similarity optimization based on ManufactureBERT.

[0081] 1. Technical Principles: Retrieval accuracy is the foundation of the RAG system, and its core is measuring the relevance of retrieval results to the query. Traditional retrieval metrics (such as Recall and Precision) rely on manually labeled relevance tags and cannot capture semantic relationships; while the general BERT model lacks understanding of manufacturing terminology (such as the relationship between "melt depth" and "bevel angle"). Therefore, this method uses a combination of ManufactureBERT and F1 scores to optimize the calculation of semantic relevance.

[0082] 2. Algorithm Principle: Indicator definition: Recall: The proportion of relevant documents in the search results out of the total number of relevant documents. The formula is: , Among them, TP (True Positive) is the number of relevant documents retrieved, which refers to the number of manufacturing documents in the search results that are related to the query semantics (such as the number of relevant chapters of the "Welding Process Specification" and the number of failure cases correctly returned when searching for "excessive welding current"); FN (False Negative) is the number of relevant documents not retrieved, which refers to the number of relevant documents that actually exist but were not retrieved (such as the number of cases about excessive current in the "Failure Case Database" that were omitted, the number of the latest process standard chapters that were not included in the search, etc.).

[0083] Recall is a core metric for measuring whether a retrieval system has missed any manufacturing-related professional documents. It is calculated as the proportion of relevant documents in the search results to all truly relevant documents. First, through manual annotation combined with ManufacturerBERT-assisted annotation, the set of truly relevant documents corresponding to the query is determined. Then, the number of documents belonging to this set (TP) and the number of documents not retrieved (FN) in the search results are counted and used in the formula for calculation. This method can evaluate whether the RAG system can comprehensively cover manufacturing professional knowledge and avoid incomplete results due to the omission of key process documents or failure cases.

[0084] Precision: The proportion of relevant documents in the search results out of the total number of documents retrieved. The formula is: , FP (False Positive) represents the number of irrelevant documents retrieved, referring to the number of manufacturing documents in the search results that are not semantically related to the query (e.g., when searching for "angular contact ball bearing installation", the number of documents incorrectly returned is "deep groove ball bearing maintenance"); TP has the same meaning as the recall formula.

[0085] Precision is a key metric used to measure the redundancy of search system results. It is calculated as the proportion of relevant documents in the search results to the total number of documents retrieved. The number of documents (TP) relevant to the query and the number of irrelevant documents (FP) in the search results are statistically analyzed and then used in the formula for calculation. This method can reduce redundant information in the results generated by manufacturing RAG systems and avoid the generated content deviating from the topic due to the introduction of irrelevant documents (e.g., when querying "bearing installation," irrelevant "gear maintenance" documents are excluded).

[0086] F1 score: the harmonic mean of recall and precision, calculated using the following formula: , The F1 score is the harmonic mean of recall and precision, comprehensively measuring the overall performance of the retrieval system. By using a harmonic mean (rather than an arithmetic mean), the weights of recall and precision are balanced, avoiding performance imbalances caused by favoring one metric (e.g., high recall but low precision will introduce a large number of redundant documents). This approach provides a single, comprehensive retrieval performance evaluation metric for manufacturing RAG systems, facilitating rapid identification of optimization directions for the system's retrieval module.

[0087] Semantic similarity calculation: Extract the [CLS] vectors (global semantic representations) of the query (Q) and document (D) using ManufactureBERT, and calculate the cosine similarity (). The formula is: , in, For the [CLS] vector being queried, The [CLS] vector of the document. For dot product, It is an L2 norm.

[0088] By extracting the [CLS] vectors (global semantic representations output by the model, incorporating the professional semantic features of manufacturing terminology) from the query (Q) and document (D) using ManufactureBERT, cosine similarity (Sim(Q,D)) is calculated to measure the semantic relevance between the two. ManufactureBERT is based on the BERT architecture and pre-trained using a 100GB manufacturing corpus (ISO standards, GB standards, enterprise process documents, etc.), optimizing its terminology understanding capabilities. For the input query and document, the model outputs a [CLS] vector of length 768, and then calculates the cosine similarity using the dot product and modulus. This approach addresses the pain point of traditional keyword retrieval failing to capture the semantic relevance of manufacturing (such as the causal relationship between "excessive welding current" and "excessive molten pool temperature," and the process logic relationship between "bevel angle" and "melt depth"), thus improving the relevance of search results.

[0089] Optimization logic: Traditional retrieval methods (such as Elasticsearch's BM25 algorithm) return the Top N documents. This method uses ManufacturerBERT to calculate the semantic similarity between these documents and the query, and then sorts them to select the Top K. Then calculate the F1 score. For example, when , If the semantic similarity of the first three documents is ≥0.8, then retain and calculate the F1 score to filter out documents with low similarity and improve retrieval accuracy.

[0090] 3. Application Example: Taking the search query "solutions for excessive welding current" as an example: Search results: Elasticsearch returned the top 5 documents, including Section 5.2 of "Welding Procedure Specification V3.0" (relevant), Chapter 2 of "Principles of Fusion Welding" (partially relevant), Section 10 of "Workshop Operation Manual" (irrelevant), "ISO 15614 Standard" (relevant), and "Catalogue of Welding Materials" (irrelevant).

[0091] Semantic similarity calculation: The cosine similarity between each document and the query was calculated using ManufactureBERT. The results are: 0.92 (Section 5.2), 0.75 (Chapter 2), 0.50 (Section 10), 0.88 (ISO 15614), and 0.45 (material catalog).

[0092] Sort and filter: Take two documents with a similarity of ≥0.8 (Section 5.2, ISO 15614).

[0093] Metric Calculation: Assuming a total of 3 relevant documents (Section 5.2, ISO 15614, "Welding Process Parameters Handbook"), then recall = 2 / 3 ≈ 0.67, precision = 2 / 2 = 1.0, F1 score = 2 × 1.0 × 0.67 / ≈0.80.

[0094] (ii) Generation quality assessment: based on the subjective and objective fusion score of GPT-4 Fine-tuned.

[0095] 1. Technical Principles: Generate quality is the core of the RAG system, and its key lies in assessing the accuracy (consistency with retrieved knowledge), professionalism (compliance with manufacturing standards), and logicality (correct steps) of the generated content. Traditional objective indicators (such as BLEU and ROUGE) measure accuracy through statistical n-gram overlap rates, but they cannot capture professional logic (such as the correctness of the sequence "turn off the power before replacing the bearing"); while manual assessment is inefficient and costly. Therefore, this method adopts a fusion strategy of objective indicators and GPT-4 subjective scoring to optimize the evaluation of generated quality.

[0096] 2. Algorithm Principle: Indicator definition: BLEU-4: Measures the 4-gram overlap between the generated result and the reference document, using the following formula: , Where N is 4 (i.e., BLEU-4), and when n=4, it is the 4-gram precision, which calculates the proportion of 4-grams appearing in the reference document in the generated text; For the accuracy of n-grams, The function penalizes cases where the generated result is too short. The accuracy of the generated content is reflected by measuring the overlap rate of 4-gram words between the generated result and manufacturing reference documents (such as process specifications and maintenance manuals). First, the effective frequency of all 4-grams (combinations of four consecutive words) in the generated text in the reference document is counted (to avoid duplicate counting), and the precision of each 4-gram is calculated. Then, the accuracy is determined based on the length of the generated text. ) and reference document length ) Proportion calculation: , Take the minimum of the two values, and penalize excessively short generated results; Finally, the geometric mean was used. Combination: , Achieve a BLEU-4 score.

[0097] This method can quickly assess the vocabulary matching between generated content and professional reference documents, and identify "illusory" content (such as incorrect terms or process parameters that do not appear in the reference).

[0098] ROUGE-L measures the proportion of the longest common subsequence (LCS) between the generated output and the reference document, reflecting the logical coherence of the generated content. The core formula of ROUGE-L is: FLCS = (1 + β) 2 ) × (PLCS × RLCS) / (β 2 × PLCS+ RLCS), The definitions of each item are as follows: RLCS (Result Recall): The length of the longest common subsequence (LCS) between the generated text and the reference document, “LCS_len”, is divided by the effective vocabulary length of the reference document, “len_ref”. It reflects the degree to which the generated content covers the key information of the reference document. The formula is RLCS = LCS_len / len_ref. PLCS (Result Precision): The length of the longest common subsequence (LCS) between the generated text and the reference document is divided by the effective vocabulary length of the generated text, “len_gen”. It reflects the information density and accuracy of the generated content. The formula is PLCS = LCS_len / len_gen. β: A weighting parameter that balances recall and precision. In manufacturing scenarios, β is usually set to 1 (equal importance is given to both). If more emphasis is needed on coverage, the value of β can be increased. FLCS: ROUGE-L's final score, ranging from 0 to 1. A higher value indicates a stronger logical coherence between the generated content and the reference document.

[0099] GPT-4 Subjective Scoring: The fine-tuned GPT-4 model outputs "Accuracy Score (0-10 points)" and "Professionalism Score (0-10 points)". The scoring criteria are based on manufacturing standards (such as GB / T 19001-2016).

[0100] Fusion strategy: A weighted average quality score (S) is calculated by combining objective indicators and subjective scores to balance vocabulary overlap and professional logic assessment. First, each indicator is normalized to a score range of 0-10, then a weighted sum is calculated according to preset weights (40% objective, 60% subjective). This method provides a comprehensive assessment of generated quality, avoiding the limitations of a single indicator (e.g., BLEU cannot assess logical coherence, and subjective scores cannot quantify vocabulary matching). The formula is as follows (the weights of each indicator are set to 0.4, 0.3, and 0.3 based on experience): S = 0.4 × BLEU⁻⁴ + 0.3 × ROUGE⁻¹ + 0.3 × (Accuracy Score + Professionalism Score) / 2 Objective indicators account for 40% (BLEU-4 accounts for 20%, ROUGE-L accounts for 20%), while subjective scores account for 60% (accuracy and professionalism each account for 30%), highlighting the importance of professional logic.

[0101] 3. Application Example: Taking the generated results of "Solutions to abnormal noise from machine tool spindles" as an example: The generated result is: "1. Turn off the machine tool power; 2. Remove the spindle end cover; 3. Check the bearing wear (model: 7205 angular contact ball bearing); 4. Replace the worn bearing; 5. Reassemble and test." Reference document: Section 3.4 of "Machine Tool Maintenance Manual V2.1" (Content: "The common cause of abnormal noise from the spindle is bearing wear. The solution steps are: turn off the power → remove the end cover → check the bearing (model 7205) → replace it → reassemble and test").

[0102] Objective indicator calculation: BLEU-4: The 4-gram overlap rate between the generated result and the reference document is 0.85 (e.g., "Turn off the machine tool power", "Remove the spindle end cover", "Check the bearing wear", "Replace the worn bearing"). The length penalty factor is 1 (the length of the generated result is consistent with the reference document), so BLEU-4=0.85.

[0103] ROUGE-L: The longest common subsequence is "turn off the machine power; remove the spindle end cover; check the bearing wear; replace the worn bearing; reassemble and test", with a length of 5. The reference document length is 5, and the generated result length is 5. Therefore, ROUGE-L = 2 × 5 / (5 + 5) = 1.0.

[0104] Subjective scoring: Call the GPT-4 Fine-tuned model, input "Generated results: [the above content]; Reference documents: [the above content]", output "Accuracy score: 10 points (fully conforms to the reference documents); Professionalism score: 9 points (the bearing model is correct, but the installation direction of "angular contact ball bearing" is not mentioned)".

[0105] Fusion score: (S=0.4×0.85 + 0.3×1.0 + 0.3×(10+9) / 2 = 0.34 + 0.30 + 0.3×9.5 = 0.34+0.30+2.85=3.49) (The full score is 4.0, which is a relatively high score).

[0106] (III) Knowledge traceability assessment: GraphSAGE-based knowledge graph path verification.

[0107] 1. Technical Principles: Knowledge tracing is the reliability guarantee of a RAG system, and its core is to verify the consistency between the generated results and the source documents. Traditional tracing methods (such as keyword matching) cannot handle complex knowledge relationships (such as the association between "maintenance steps" and "process specifications"); while knowledge graphs (KG) can clearly represent knowledge structures through entity-relationship models, but general GNN models (such as GCN) cannot efficiently handle dynamic knowledge (such as process specification updates). Therefore, this method adopts a strategy of GraphSAGE + path similarity to optimize the accuracy of knowledge tracing.

[0108] 2. Algorithm Principle: Knowledge Graph Construction: A manufacturing knowledge graph is constructed using Neo4j. Entities include "Document", "Terminology", "Steps", and "Standards", and relationships include "Contains" (Document → Terminology), "Follows" (Steps → Standards), and "Refers to" (Generated Result → Document). For example, "Welding Procedure Specification V3.0" → "Contains" → "Welding Current", and "Welding Current Adjustment Steps" → "Follows" → "ISO15614 Standard".

[0109] GraphSAGE Model: GraphSAGE is an inductive GNN model that generates node embeddings by aggregating neighbor node information. It supports dynamic updates to the manufacturing knowledge graph (such as adding new process documents or standard entities) without requiring model retraining. The mean aggregation function calculates the mean of the node's neighbor embeddings, integrating neighbor information into the node's own embedding to more accurately capture the relationships between manufacturing entities (such as the inclusion relationship between "7205 bearing" and "machine tool maintenance manual," or the compliance relationship between "welding steps" and "ISO standards"). Its core is the aggregation function; this method uses mean aggregation, with the formula: , in, The feature embedding (dimension dk) for node (v) at layer (k). Let be the trainable weight matrix of the (k)th layer (with dimension dk+1×dk). The activation function is ReLU (this scheme uses ReLU to avoid gradient vanishing). Let v be the set of neighbors of node (v) (e.g., the neighbors of “7205 bearing” include “machine tool maintenance manual V2.1”, “installation steps”, etc.). mean() is the mean aggregation function that calculates the average value of the embedding of neighboring nodes.

[0110] Path similarity calculation: The paths of entities in the generated result (e.g., "7205 bearing") and entities in the source document (e.g., "7205 bearing" in "Machine Tool Maintenance Manual V2.1") in the knowledge graph should be consistent (e.g., entities in the generated result need to be linked to the source document through "reference" relationships, and entities in the source document need to be linked to terms through "inclusion" relationships). Embedding vectors of entity paths are generated using GraphSAGE, and the cosine similarity between the embeddings of the two paths is calculated to determine the reliability of the knowledge source of the generated result. This formula aims to address the pain point of difficult knowledge traceability in the manufacturing industry, verify the consistency between the generated content and the source document, and avoid "illusion" content. The formula is:

[0111] in, Indicates the similarity between the generated entity path and the source document entity path; The path embedding of the resulting entity (generated by GraphSAGE, dimension 128). Embed the path of the source document entity (dimension 128). The cosine similarity value ranges from -1 to 1. The closer the value is to 1, the higher the path consistency.

[0112] 3. Application Example: Taking "bearing 7205 in the generated results" as an example: Knowledge graph path: "7205 bearing" in the generated result → "References" → "Machine Tool Maintenance Manual V2.1" (path 1); "7205 bearing" in the source document → "Includes" → "Machine Tool Maintenance Manual V2.1" (path 2).

[0113] GraphSAGE Embedding: Generating the embedding of path 1 using the GraphSAGE model (Dimension 128) and embedding of path 2 (Dimensions are 128).

[0114] Path similarity: Calculation If the value is ≥0.9, the path is considered consistent (the “7205 bearing” in the generated result comes from the source document); if the value is <0.9, the path is considered inconsistent (the generated result has an “illusion”).

[0115] Consistency check: If the path similarity is ≥0.9 and the attributes (such as "model" and "purpose") of "7205 bearing" in the generated result are consistent with those in the source document, the source tracing is successful; otherwise, the source tracing fails (e.g., "7205 bearing" in the generated result comes from an old version of the manual).

[0116] (iv) Timeliness assessment: Validation of the standard based on incremental update detection.

[0117] 1. Technical Principles: Timeliness is crucial for the adaptability of the RAG system, and its core is ensuring that the generated results conform to the latest technical standards. Manufacturing standards (such as ISO and GB) are frequently updated (e.g., ISO 15614-1:2023 replaces ISO 15614-1:2017), and existing systems cannot quickly detect standard updates and synchronize the generated results. Therefore, this method employs a strategy of timestamp comparison plus incremental update detection to optimize timeliness assessment.

[0118] 2. Algorithm Principle: Indicator definition: Standard Validity: This assesses whether the standards referenced in the generated results are currently valid versions. It's a core indicator for evaluating whether the manufacturing standards referenced in the generated results are currently valid versions. This involves comparing the publication date of the referenced documents with the publication date of the latest version of the standard to determine if it meets timeliness requirements. By ensuring that the generated content complies with the latest industry standards (such as ISO and GB standards), it avoids production risks caused by using outdated process parameters or maintenance procedures. The formula is: , Wherein, Validity is the standard validity index (a value of 1 indicates validity, and 0 indicates invalidity); the source document timestamp is the publication date of the document referenced in the generated result (e.g., the publication date of "Welding Procedure Specification V3.0" is 2023-05-01); the latest standard timestamp is the publication date of the currently valid version of the standard (e.g., the publication date of ISO 15614-1:2023 is 2023-06-01). When the source document timestamp is greater than or equal to the latest standard timestamp, Validity = 1 (referring to a valid version); otherwise, Validity = 0 (referring to an outdated version).

[0119] Response Latency: The time from standard update to result update. It measures the system's responsiveness to manufacturing standard updates, calculating the time difference between the official standard update and the synchronized result update. By evaluating system timeliness, it ensures rapid adaptation to new standards (such as updates to process specifications and safety standards) and avoids generating outdated content. The formula is: Latency = Generation result update time - Standard update time The generation result update time is the time it takes for the system to regenerate content based on the latest standard; the standard update time is the time when the official version of the standard is released. Latency is measured in days, with a target value of ≤3 days (meeting the rapid response requirements of the manufacturing industry).

[0120] Incremental update detection: A hash-based document change detection algorithm is used to periodically calculate the hash value (e.g., MD5) of the source document. If the hash value changes (e.g., due to a standard update), an update to the generated results is triggered. The algorithm steps are as follows: Initialization: Calculate an initial hash value for each source document and store it in a hash table (e.g., ...). ).

[0121] Periodic checks: Recalculate the document hash value every (T) time interval (e.g., 1 day) and compare it with the value in the hash table.

[0122] Trigger update: If the hash value changes (e.g.) If the document is marked as "updated", the RAG system will be triggered to retrieve the document again and generate new results.

[0123] 3. Application Example: Taking "ISO 15614-1:2023 Standard Update" as an example: Standard updated on: June 1, 2023.

[0124] Source document timestamp: The document referenced in the generated result is "Welding Procedure Specification V3.0" (released on 2023-05-01), which references the standard ISO 15614-1:2017 (old version).

[0125] Standard validity calculation: Therefore (invalid).

[0126] Incremental update detection: On June 2, 2023, the system detected a change in the hash value of "Welding Procedure Specification V3.0" (due to an update of the referenced standard), and marked it as "updated".

[0127] Response latency calculation: The generation result was updated on 2023-06-03, therefore... Days (meets requirements, target delay ≤ 3 days).

[0128] V. Intelligent Model Optimization: Adapting ManufactureBERT to GraphSAGE for the Manufacturing Industry.

[0129] (a) ManufactureBERT: Pre-trained optimization in the manufacturing sector: 1. Construction of pre-training corpus: To optimize the understanding of manufacturing terminology by ManufacturerBERT, the pre-training corpus covers the following types: Standards and specifications: ISO (such as ISO 15614-1:2023), GB (such as GB / T 19001-2016), industry standards (such as JB / T 10005-2010), totaling 50 GB.

[0130] Enterprise knowledge: internal process documents (such as "Welding Process Specification V3.0"), fault case library (such as "Machine Tool Fault Record 2023"), operation manuals (such as "Workshop Equipment User Guide"), totaling 30GB.

[0131] Publicly available data: manufacturing industry papers (such as the Welding Journal) and news (such as the China Machinery Industry News), totaling 20GB.

[0132] 2. Principles of pre-training tasks: Masked Language Model (MLM): Predicts masked content by masking 15% of the tokens in a randomized corpus. For manufacturing terms, the masking ratio is increased (e.g., 20%) to optimize term prediction capabilities (e.g., masking "melting depth" to predict "melting depth" instead of "depth").

[0133] Next Sentence Prediction (NSP): Input sentence pair ( ),predict Is it The next sentence. Based on the logic of the manufacturing industry, construct causal relationship sentences (such as...). "Welding current is too high" ("The molten pool temperature is too high"), optimize logical reasoning ability.

[0134] 3. Pre-training parameter settings: Model structure: 12 Transformer layers, 768-dimensional hidden layers, 12-head self-attention, total parameters approximately 110M.

[0135] Training hardware: 8 NVIDIA A100 GPUs (80GB VRAM), using data parallelism.

[0136] Training parameters: batch size=2048, learning rate=5e-5, training steps=1M, dropout=0.1.

[0137] 4. Optimization Results: Compared to the general BERT model, ManufactureBERT improves accuracy by 15% (from 75% to 90%) on semantic similarity tasks in the manufacturing industry, specifically as follows: Correctly identify the causal relationship between "excessive welding current" and "excessive molten pool temperature" (similarity 0.85 vs. 0.60 for general BERT).

[0138] Correctly distinguish the application differences between "angular contact ball bearings" and "deep groove ball bearings" (similarity 0.70 vs. 0.45 for general-purpose BERT).

[0139] (ii) GraphSAGE: Dynamic Knowledge Graph Embedding 1. Model adaptation: GraphSAGE is an inductive GNN model that can handle dynamic knowledge (such as adding new document entities), making it suitable for updating knowledge graphs in the manufacturing industry. This method's optimizations to GraphSAGE include: Node feature extension: Add "timestamp" and "version number" features to the "document" node (e.g., the timestamp of "Welding Procedure Specification V3.0" = 2023-05-01, version number = 3.0), and add "standard number" feature to the "terminology" node (e.g., the standard number of "7205 bearing" = GB / T 276-2013), enriching the node representation.

[0140] Aggregation function adjustment: A combined mean and max aggregation function is adopted. Addressing the dynamic nature of manufacturing knowledge graphs (such as document version updates and new terminology additions), this function integrates the average and extreme information of neighboring nodes, enhancing the model's ability to represent dynamic knowledge. First, the mean of the neighboring node embeddings is calculated (capturing average features such as common process parameters and document publication time). Then, the maximum value of the neighboring node embeddings is calculated (capturing extreme features such as the latest version number and the highest process standard). The two are concatenated, and a new node embedding is generated through linear transformation and activation functions. This approach solves the problem that general GraphSAGE cannot efficiently handle dynamic knowledge in the manufacturing industry, supporting rapid adaptation to scenarios such as process specification updates and new document additions, without requiring retraining the entire model. The formula is: , in, This represents the feature embedding of node v at the (k-1)th layer; σ is the trainable weight matrix of the k-th layer; σ is the activation function (using ReLU). is the mean of the embeddings of neighbor node u at layer k-1; mean aggregation captures the average information of neighbor nodes (such as the average publication time of documents, the general range of process parameters), and max aggregation captures the extreme information of neighbor nodes (such as the latest version number of documents, the highest requirement of process standards). This combined aggregation function optimizes dynamic knowledge representation, enabling the model to quickly identify updates in the manufacturing knowledge graph (such as when "Welding Process Specification V3.0" replaces V2.0, node embeddings will change significantly), supporting real-time evaluation of knowledge traceability.

[0141] 2. Optimization results: Compared to the general GraphSAGE model, the optimized GraphSAGE improves accuracy by 12% (from 82% to 94%) on the knowledge tracing path similarity task, specifically as follows: Accurately capture the impact of "process specification updates" on knowledge paths (e.g., the path embedding of "Welding Process Specification V3.0" → "Includes" → "Welding Current" is significantly different from the path embedding of the old version V2.0).

[0142] It efficiently handles dynamic knowledge (such as when adding a new document entity, there is no need to retrain the entire model, only to update the embeddings of neighboring nodes), reducing training time by 40% (from 24 hours to 14 hours).

[0143] Example: Please see Figure 4 Taking the evaluation of intelligent process guidance systems as an example, such as Figure 4 The process shown is as follows: Step 1, Benchmark Set Construction: Data collection: Extract 1,000 process guidance questions (such as "solutions for excessive welding current" and "repair steps for abnormal noise from machine tool spindle") from the enterprise PLM system (product lifecycle management), the corresponding search documents (such as "Welding Process Specification V3.0"), and generate results (such as "adjust the current to 120A±5A").

[0144] Data annotation: Use LabelStudio to annotate the following content: Search result relevance: Label the documents retrieved for each query as "relevant", "partially relevant", or "irrelevant".

[0145] Accuracy of generated results: Label each generated result as "accurate", "partially accurate", or "inaccurate".

[0146] Knowledge traceability accuracy: Label each generated result as "traceable" or "non-traceable".

[0147] Timeliness: Label each generated result as "valid" or "invalid".

[0148] Test set partitioning: The 1000 data points are divided into a training set (700 data points), a validation set (200 data points), and a test set (100 data points), with the test set used for the final evaluation.

[0149] Step 2, Evaluation Preparation: Configure data sources: Connect the enterprise process library, standard library, and fault case library to the evaluation system to ensure that the data is up-to-date (e.g., the "Welding Process Specification V3.0" updated in 2024 has been stored in Elasticsearch).

[0150] Select evaluation modules: Based on the requirements of the intelligent process guidance system, select four modules: "Retrieval Accuracy", "Generation Quality", "Knowledge Traceability", and "Timeliness".

[0151] Setting parameters: Search accuracy: Top 5 search results, Manufacture BERT semantic similarity threshold 0.7.

[0152] Generation quality: BLEU-4 weight 0.4, ROUGE-L weight 0.3, GPT-4 subjective rating weight 0.3.

[0153] Knowledge source: GraphSAGE path similarity threshold 0.9.

[0154] Timeliness: Standard validity threshold 1 (must cite the latest standard), response delay threshold 3 days.

[0155] Step 3, Search accuracy assessment: Input query: Select 100 process guidance questions from the test set (such as "solution to excessive welding current").

[0156] Search results: Elasticsearch is used to retrieve the top 5 documents for each query.

[0157] Semantic similarity calculation: The cosine similarity between each document and the query is calculated using ManufactureBERT, and the top 3 documents (with similarity ≥ 0.7) are selected after sorting.

[0158] Metrics calculation: Calculate the Recall, Precision, and F1 values ​​for each query and take the average (e.g., Recall=0.80, Precision=0.90, F1=0.85).

[0159] Step 4, generate quality assessment: Input generated results: Get the generated results of each query from the intelligent process guidance system (e.g., "Adjust the current to 120A±5A").

[0160] Objective index calculation: Calculate the BLEU-4 and ROUGE-L values ​​for each generated result (e.g., BLEU-4=0.85, ROUGE-L=0.90).

[0161] Subjective scoring: Use the GPT-4 Fine-tuned model, input "generated results + reference documents", and obtain accuracy and professionalism scores (e.g., accuracy = 10 points, professionalism = 9 points).

[0162] Fusion score: Calculate the fusion score for each generated result (e.g., 3.49 / 4.0) and take the average (e.g., 3.2 / 4.0).

[0163] Step 5, Knowledge Source Assessment: Entity Extraction: Use SpaCy to extract entities from the generated results (such as "120A±5A" "Welding Current").

[0164] Knowledge graph link: Find the source document corresponding to the entity through the Neo4j knowledge graph (e.g., “120A±5A” comes from Section 5.2 of “Welding Process Specification V3.0”).

[0165] Path verification: Use GraphSAGE to calculate the path similarity between the generated entity and the source document entity (e.g., 0.95 ≥ 0.9).

[0166] Consistency check: Check whether the entity attributes in the generated result (such as the current range of "120A±5A") are consistent with the source document (if consistent, the traceability is successful).

[0167] Step 6, Timeliness Assessment: Timestamp Acquisition: Obtain the timestamp of the document referenced in the generated result (e.g., the timestamp of "Welding Procedure Specification V3.0" = 2023-05-01).

[0168] Standard validity check: Compare with the latest standard timestamp (e.g., the timestamp of ISO 15614-1:2023 = 2023-06-01) to determine if it is valid (e.g., 2023-05-01 < 2023-06-01, invalid).

[0169] Response latency calculation: Calculate the time from standard update to result update (e.g., from 2023-06-01 to 2023-06-03, a delay of 2 days).

[0170] Indicator statistics: statistical effectiveness (e.g., 80% of generated results reference the latest standards) and average latency (e.g., 2.5 days).

[0171] Step 7, Results Analysis and Optimization: Results Summary: An evaluation report is generated, including scores for each module (e.g., retrieval accuracy F1=0.85, generation quality score=3.2 / 4.0, knowledge tracing pass rate=90%, timeliness effectiveness rate=80%).

[0172] Problem Analysis: Retrieval accuracy: F1=0.85, because some relevant documents had semantic similarity below the threshold (e.g., 0.68) and were not retained.

[0173] Generation quality: Score = 3.2 / 4.0, because some generated results did not mention the installation orientation of terms (such as the orientation of "angular contact ball bearing").

[0174] Knowledge tracing: pass rate = 90%, because some entities in the generated results were not linked to the knowledge graph (e.g., “120A±5A” did not have its source marked).

[0175] Timeliness: Effectiveness = 80%, because some documents were not updated in a timely manner (e.g., "Welding Procedure Specification V3.0" did not reference ISO 15614-1:2023).

[0176] Optimization suggestions: Search accuracy: The semantic similarity threshold for ManufactureBERT was adjusted from 0.7 to 0.65 to retain more relevant documents.

[0177] Build quality: Add "Please mention the installation direction of the term" (e.g., "The installation direction of the angular contact ball bearing should be towards the spindle") to the prompt of the build module.

[0178] Knowledge Origin: Use LabelStudio to annotate the source of entities in the generated results (e.g., “120A±5A” comes from Section 5.2 of “Welding Procedure Specification V3.0”), and update the knowledge graph.

[0179] Timeliness: Increase the frequency of incremental update detection (from once a day to once an hour) to update the data source in a timely manner.

[0180] Therefore, the core technical advantages of the intelligent evaluation and generation system and method for manufacturing knowledge based on RAG enhanced retrieval of the present invention are reflected in the following three aspects: (1) Deep adaptation to manufacturing scenarios: In response to the lack of industry-specificity in general evaluation in background technology, we have optimized four major modules: retrieval (semantic similarity calculation), generation (professional term verification), tracing (entity-document link) and timeliness (standard update detection), covering manufacturing-specific dimensions such as "process professionalism" and "standard compliance", and solving the pain points of traditional keyword retrieval being unable to capture process logic association and generating content that does not conform to industry standards. (2) Intelligent model-driven accurate evaluation: In response to the problems of low retrieval accuracy, generation illusion and difficulty in tracing the source in the background technology, models such as ManufactureBERT (improving semantic retrieval relevance and solving the problem of capturing the causal relationship between "excessive welding current" and "excessive molten pool temperature") and GraphSAGE (realizing entity-source document path verification and solving the pain point of untraceable generated results) are proposed, which greatly improve the accuracy and reliability of evaluation results.

[0181] (3) Continuous optimization capability of closed-loop iteration: In response to the problem that the system cannot be continuously improved in the background technology, a closed-loop mechanism of "evaluation-problem analysis-optimization suggestion-re-evaluation" is formed. By adjusting the model threshold, updating the knowledge graph, increasing the incremental detection frequency, etc., the pain points such as time delay and uneven generation quality are solved, and the dynamic iterative upgrade of the RAG system is supported.

[0182] In the future, the intelligent evaluation and generation method for manufacturing knowledge based on RAG enhanced retrieval of the present invention can be further extended to the following directions: Edge evaluation: Deploy evaluation modules on edge devices (such as smart terminals) in the workshop to achieve low-latency evaluation (such as real-time detection of the accuracy of fault solutions).

[0183] Multimodal evaluation: Supports the evaluation of multimodal data such as images and videos (e.g., evaluating the correlation between "welding defect images" and "defect causes").

[0184] Self-supervised evaluation: Utilizing self-supervised learning (such as comparative learning) reduces the cost of manual annotation and improves evaluation efficiency.

[0185] This embodiment also provides a computer device, which includes a processor and a memory. The memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, it implements the steps of the above-described intelligent evaluation and generation method for manufacturing knowledge based on RAG enhanced retrieval.

[0186] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0187] This embodiment also provides a computer-readable storage medium storing at least one computer-readable instruction. This instruction is loaded and executed by a processor to implement the steps of the aforementioned intelligent evaluation and generation method for manufacturing knowledge based on RAG-enhanced retrieval. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0188] In summary, the intelligent evaluation and generation system and method for manufacturing knowledge based on RAG enhanced retrieval of the present invention effectively solves the problem of insufficient retrieval accuracy through manufacturing-specific benchmark test sets and intelligent model optimization; improves the professionalism and reliability of generated content through multi-dimensional generation quality assessment; achieves accurate traceability of generated results through knowledge graphs and GraphSAGE models; and ensures timely response of dynamic knowledge through incremental update detection mechanisms.

[0189] Those skilled in the art should recognize that the above embodiments are merely illustrative of the present invention and are not intended to limit the present invention. Any variations or modifications to the above embodiments that are within the spirit and essence of the present invention will fall within the scope of the claims of the present invention.

Claims

1. A manufacturing knowledge intelligent evaluation and generation system based on RAG enhanced retrieval, characterized in that, Adopting a cloud-edge-device collaborative architecture, including: A cloud server cluster, wherein the cloud server cluster includes at least a computing server, a storage server, and a retrieval server; Edge devices include smart terminals deployed on the production site and IoT gateways for connecting workshop equipment. The edge devices are configured to collect and upload manufacturing site data. The data layer, deployed on the cloud server cluster, is used to store and manage the manufacturing static knowledge base, dynamic data, and benchmark test sets. The core evaluation layer, deployed on the cloud server cluster and communicating with the data layer, includes a retrieval accuracy evaluation module, a generation quality evaluation module, a knowledge tracing evaluation module, and a timeliness evaluation module. The intelligent model layer, deployed on the cloud server cluster, provides algorithmic support for the evaluation core layer. The intelligent model layer includes at least one language model pre-trained using corpus from the manufacturing sector. The application layer communicates with the core evaluation layer and is configured to provide an evaluation interaction interface and display evaluation results.

2. A method for intelligent evaluation and generation of manufacturing knowledge based on RAG enhanced retrieval, comprising using the intelligent evaluation and generation system for manufacturing knowledge as described in claim 1, characterized in that, Includes the following steps: Receive an evaluation request for the target retrieval enhancement generation system, wherein the evaluation request contains at least the query content to be evaluated; Based on the query content, the search results and related reference knowledge of the target retrieval enhancement generation system are obtained from the data layer; The evaluation core layer is invoked to evaluate the performance of the target retrieval enhancement generation system under the query in multiple dimensions based on the retrieval results and reference knowledge. The multi-dimensional evaluation includes at least retrieval accuracy evaluation and generation quality evaluation. The application layer outputs evaluation results containing assessment metrics for each dimension.

3. The intelligent evaluation and generation method for manufacturing knowledge based on RAG enhanced retrieval as described in claim 2, characterized in that, The retrieval accuracy assessment specifically includes the following process: Obtain the initial set of search result documents returned by the target retrieval enhancement generation system for the query; The domain pre-trained language model in the intelligent model layer is invoked to calculate the semantic similarity between the query and each document in the initial retrieval result document set; Based on the semantic similarity, the initial search result document set is reordered or filtered to obtain optimized search results; Based on the optimized search results, a search accuracy index is calculated, which includes at least one of recall, precision, or F1 score.

4. The intelligent evaluation and generation method for manufacturing knowledge based on RAG enhanced retrieval as described in claim 3, characterized in that, The domain-pretrained language model in the intelligent model layer is obtained through the following methods: Collect domain corpus containing manufacturing standards, process documents, and failure cases; The base language model is pre-trained in a domain-adaptive manner using a masked language model and a next-sentence prediction task to optimize the model’s ability to understand manufacturing terminology and logical relationships.

5. The intelligent evaluation and generation method for manufacturing knowledge based on RAG enhanced retrieval as described in claim 2, characterized in that, The quality assessment process specifically includes the following steps: Obtain the response text generated by the target retrieval enhancement generation system based on the retrieval results; Calculate the objective text matching index between the response text and the reference document, wherein the objective text matching index includes BLEU value or ROUGE value; The generative quality assessment model in the intelligent model layer is invoked to subjectively score the response text, and the subjective score includes at least an accuracy score and / or a professional score. The objective text matching index and the subjective score are weighted and fused to obtain a comprehensive quality score.

6. The intelligent evaluation and generation method for manufacturing knowledge based on RAG enhanced retrieval as described in claim 2, characterized in that, The multi-dimensional assessment also includes knowledge traceability assessment, which specifically includes the following process: Extract key entities from the response text generated by the target retrieval enhancement generation system; In the pre-built manufacturing knowledge graph, query the source document information and knowledge paths associated with the key entities; The similarity between the knowledge path of an entity in the response text and the knowledge path of the corresponding entity in the source document is calculated based on the graph neural network model, and the consistency of the source is determined based on the similarity.

7. The intelligent evaluation and generation method for manufacturing knowledge based on RAG enhanced retrieval as described in claim 6, characterized in that, The manufacturing knowledge graph is constructed based on dynamically updated manufacturing documents, where documents, terms, and standards are abstracted as nodes, and the relationships between nodes are abstracted as edges. The graph neural network model uses an aggregation function to aggregate the neighbor information of nodes to generate the embedded representation of nodes, which is used to calculate path similarity.

8. The intelligent evaluation and generation method for manufacturing knowledge based on RAG enhanced retrieval as described in claim 2, characterized in that, The multi-dimensional assessment also includes a timeliness assessment, which specifically includes the following process: Obtain the publication timestamp of the source document on which the target retrieval enhancement generation system uses to generate a response; Obtain the latest valid version timestamp of the technical standard or specification corresponding to the source document; Compare the publication timestamp with the latest valid version timestamp to determine whether the knowledge referenced in the generated result is the latest version; Calculate the response latency from standard update to synchronous update of generated results.

9. The intelligent evaluation and generation method for manufacturing knowledge based on RAG enhanced retrieval as described in claim 8, characterized in that, The timeliness assessment also includes an incremental update detection sub-step, specifically: Periodically calculate hash values ​​for source documents in the data layer; Compare the currently calculated hash value with the historically stored hash values; If the hash value changes, it is determined that the corresponding source document has been updated, and the target retrieval enhancement generation system is triggered to re-retrieve the document to update the generated results.

10. The intelligent evaluation and generation method for manufacturing knowledge based on RAG enhanced retrieval as described in claim 2, characterized in that, The intelligent evaluation and generation method for manufacturing knowledge also includes a closed-loop optimization step, specifically comprising the following process: After outputting the evaluation results containing evaluation indicators of each dimension through the application layer, the specific problems of the target retrieval enhancement generation system in each evaluation dimension are analyzed based on the evaluation results. Generate targeted optimization suggestions, which include at least one of the following: adjusting the semantic similarity threshold of the retrieval model, optimizing the prompt template of the generated model, updating the knowledge graph data, or increasing the update detection frequency of the data source; The optimization suggestions are applied to the optimization process of the target retrieval enhancement generation system, and the evaluation is re-initiated based on the optimized system, forming an evaluation-optimization iterative closed loop.

11. A computer device comprising a processor and a memory, the memory storing computer-readable instructions, characterized in that, When the processor executes the computer-readable instructions, it implements the steps of the intelligent evaluation and generation method for manufacturing knowledge based on RAG enhanced retrieval as described in any one of claims 2 to 10.

12. A computer-readable storage medium storing at least one computer-readable instruction, characterized in that, The computer-readable instructions are loaded and executed by a processor to implement the steps of the manufacturing knowledge intelligent evaluation generation method based on RAG enhanced retrieval as described in any one of claims 2 to 10.

Images