Industrial equipment maintenance intelligent question and answer system based on multi-agent cooperation

Abstract

The application relates to an industrial equipment maintenance intelligent question and answer system based on multi-agent cooperation. The input unit is used for receiving multi-modal user input information such as text, voice, image or equipment scanning and the like and analyzing the same into structured question information; the scheduling unit performs semantic understanding and question classification on the structured question information based on a fine-tuned cross-language pre-training language model and an incremental training mechanism; the processing unit calls corresponding domain agents according to the classification result to generate intelligent question and answer processing results including predictive maintenance suggestions, structured reply contents and semantic annotation information; and the fusion unit fuses local knowledge bases, graph databases and network search information to perform multi-hop semantic reasoning on the intelligent question and answer processing results to generate multi-modal reply information containing text instructions, image screenshots, prediction curves and recommended links. The system can support multi-language and multi-modal intelligent question and answer and predictive maintenance in a complex industrial maintenance scene.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an intelligent question-and-answer system for industrial equipment maintenance based on multi-agent collaboration. Background Technology

[0002] In existing technologies, the maintenance and management of industrial equipment mainly rely on traditional fault reporting systems or question-and-answer platforms based on static knowledge bases. These systems typically respond to user questions using rule matching or keyword retrieval, providing some maintenance guidance. However, their processing capabilities and response accuracy are significantly limited when facing real-world application scenarios with diverse languages, complex questions, and varied document structures. In recent years, with the development of artificial intelligence technology, some systems have begun to incorporate natural language processing and knowledge graph technologies to improve question-and-answer performance. However, overall, they still primarily rely on single models, static knowledge structures, and fixed interaction formats, and have not yet achieved effective synergy between complex semantic understanding and reasoning capabilities.

[0003] Existing technologies generally suffer from the following problems: First, they lack support for multimodal inputs and cannot effectively process non-textual information such as images and speech; second, classification and semantic recognition models cannot adapt to multilingual inputs and are difficult to continuously optimize performance through online learning; third, the question-answering results are of a single form and cannot provide composite information such as structured, visualized, and predictive maintenance; and fourth, their knowledge source fusion capabilities are weak, making it difficult to achieve joint reasoning and dynamic completion of local knowledge, knowledge graphs, and external information.

[0004] Therefore, there is an urgent need for a technical solution that can adapt to the complex interactive needs of industrial sites, support multilingual and multimodal input, and possess semantic reasoning, knowledge fusion, and intelligent question answering collaborative processing capabilities. Summary of the Invention

[0005] This application provides an intelligent question-and-answer system for industrial equipment maintenance based on multi-agent collaboration, which improves the accuracy, real-time performance, and multimodal response capability of intelligent question-and-answer in industrial equipment maintenance scenarios.

[0006] This application provides an intelligent question-and-answer system for industrial equipment maintenance based on multi-agent collaboration, including:

[0007] The input unit is used to receive multimodal user input information and parse the multimodal user input information into structured question information, wherein the multimodal user input information includes text input, voice input, image input or device scanning input;

[0008] The scheduling unit is used to perform semantic understanding and question classification on the structured question information based on a multilingual intent recognition model and an incremental training mechanism to obtain question classification results; wherein, the multilingual intent recognition model is a fine-tuned cross-lingual pre-trained language model;

[0009] The processing unit is used to invoke the domain agent corresponding to the question classification result to generate intelligent question-answering processing results. The domain agent includes an equipment maintenance agent, a management consulting agent, a digital operation and maintenance agent, or a technical service agent. The intelligent question-answering processing results include predictive maintenance suggestions, structured response content, and semantic annotation information.

[0010] The fusion unit is used to perform multi-hop semantic reasoning based on the intelligent question-answering processing results and by fusing local knowledge base information, graph database information and online retrieval information to obtain multimodal response information, which includes text descriptions, image screenshots, prediction curves and external recommendation links.

[0011] The beneficial effects of this application mainly include: (1) Supporting multimodal input and multilingual understanding, it can process various forms of user input such as text, voice, image and device scanning, and achieve accurate classification of mixed language problems through cross-language pre-trained models, significantly improving the system's adaptability in multilingual and multi-scenario environments. (2) Realizing the collaborative scheduling of problem classification and domain intelligence agents, dynamically calling dedicated intelligence agents in the corresponding domain according to the classification results, thereby achieving more targeted knowledge processing and fault response, and improving the accuracy and professionalism of industrial question answering. (3) Integrating local knowledge, graph databases and network retrieval information, constructing multi-hop semantic reasoning paths, and outputting multimodal response results including text descriptions, image screenshots and predictive curves, effectively improving the visualization and predictive decision support capabilities of the responses. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of an intelligent question-and-answer system for industrial equipment maintenance based on multi-agent collaboration, provided in the first embodiment of this application. Detailed Implementation

[0013] Many specific details are set forth in the following description to provide a full understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this application; therefore, this application is not limited to the specific embodiments disclosed below.

[0014] The first embodiment of this application provides an intelligent question-and-answer system for industrial equipment maintenance based on multi-agent collaboration. Please refer to... Figure 1 This figure is a schematic diagram of the first embodiment of this application. The following is in conjunction with... Figure 1 The first embodiment of this application provides a detailed description of an intelligent question-and-answer system for industrial equipment maintenance based on multi-agent collaboration.

[0015] The intelligent question-and-answer system for industrial equipment maintenance based on multi-agent collaboration includes an input unit 101, a scheduling unit 102, a processing unit 103, and a fusion unit 104.

[0016] The input unit 101 is used to receive multimodal user input information and parse the multimodal user input information into structured question information. The multimodal user input information includes text input, voice input, image input, or device scanning input.

[0017] The main function of input unit 101 is to receive multimodal input information submitted by users through different interaction methods, and accurately convert this information into structured question information, providing a standardized data foundation for subsequent semantic understanding and question classification. Input unit 101 includes an information acquisition component, a modality recognition component, a unified encoding component, and a structure conversion component, which work together to ensure the consistency of multimodal input at the semantic level and the standardization of technical processing.

[0018] The information acquisition component is responsible for receiving raw input data transmitted from the user terminal. This data can originate from human-machine interfaces (such as touch panels, mobile terminals, industrial terminals, voice assistants, or image acquisition devices). Text input is typically obtained through keyboard input or OCR extraction. Voice input is captured through a microphone and transcribed into text in real time by a speech recognition engine. Image input includes photos of equipment malfunctions, screenshots of interfaces, and images of abnormal indicator lights, which are acquired through cameras or uploaded files. Equipment scanning input uses devices such as AR devices, QR code scanners, and barcode recognition modules to acquire information such as machine labels, part numbers, or maintenance record page numbers.

[0019] Upon receiving multi-source input, the modality recognition component first performs modality recognition and signal separation to ensure that various modal data can be routed to their corresponding preprocessing paths. For example, speech data is fed into the Automatic Speech Recognition (ASR) model, while image data is processed by an image preprocessor for edge enhancement, denoising, and image segmentation to extract key regions. Text data undergoes syntax checking and encoding standard conversion to ensure Unicode consistency across different language input scenarios.

[0020] The unified encoding component converts the above modal inputs into intermediate representations in a unified format. Specifically, for text and speech transcription results, a structured token sequence is generated using tokenization, along with semantic tags such as timestamps, syntactic features, and keyword density. Image data, after recognition, has image recognition tags extracted (such as component type, anomaly descriptions, etc.) and described in JSON format. Equipment scan data is converted into indexable metadata information through an equipment encoding parsing algorithm, including fields such as equipment ID, component serial number, production line, and maintenance timestamp.

[0021] The structured conversion component, based on the aforementioned unified format, merges multimodal inputs into a structured problem information object. This object includes at least a problem category preliminary judgment field, a language type field, a key equipment field, a visualization reference field (such as an image URL or screenshot coordinates), and a raw input content field. This object is transmitted to the scheduling unit 102 through a predefined interface, and the structured problem data is invoked using a common protocol (such as gRPC or a RESTful API) to achieve efficient and low-latency data interaction.

[0022] The input unit 101 also includes an anomaly detection and fault-tolerance mechanism to identify inputs with incorrect formatting, unsupported modalities, or low confidence. When an input anomaly is detected, the unit can guide the user to re-enter the input through a user feedback interface or provide modality conversion suggestions (such as suggesting the user take a clear image or use voice input). In addition, the input unit 101 supports a user identification information binding mechanism to associate input information with user identity, permission level, and historical maintenance records, ensuring contextual consistency in subsequent processing.

[0023] Through the above mechanisms, the input unit 101 can not only accurately receive and parse complex multimodal user input in the industrial environment, but also output question data in a structured and standardized manner, ensuring that the entire question-and-answer system has good readability, interpretability and technical stability in the semantic processing stage, thus laying a solid input foundation for achieving high-quality intelligent question-and-answer results.

[0024] The scheduling unit 102 is used to perform semantic understanding and question classification on the structured question information based on a multilingual intent recognition model and an incremental training mechanism to obtain question classification results; wherein, the multilingual intent recognition model is a fine-tuned cross-language pre-trained language model.

[0025] Furthermore, the scheduling unit is specifically used for:

[0026] The system receives structured question information output by the input unit and extracts the text input field, speech recognition transcription field, and image recognition label field to form a multimodal semantic label sequence.

[0027] The multimodal semantic label sequence is concatenated into a unified input vector according to a preset encoding rule and input into the multilingual intent recognition model. The multilingual intent recognition model is a fine-tuned cross-lingual pre-trained language model used to generate an embedding vector containing language-independent semantic features.

[0028] Based on the semantic content represented by the embedded vector, a semantic consistency discriminant network is used to evaluate the stability of the language mixing features of the input information and output a language consistency score.

[0029] Based on the language consistency score and user feedback, the classification head parameters in the multilingual intent recognition model are updated only through a streaming incremental fine-tuning mechanism, while keeping the underlying encoder parameters frozen. The problem classification result is then output based on the fine-tuned model.

[0030] The scheduling unit first receives structured question information output by the input unit. This structured question information includes at least three types of fields: text input field, speech recognition transcription field, and image recognition label field. The text input field consists of industrial maintenance-related descriptions directly typed by the user through the terminal interface or imported from text. The speech recognition transcription field is standardized text obtained by the speech recognition module after transcribing the speech input. The image recognition label field consists of semantic labels extracted after recognizing the scanned image of the equipment through an image classification and object detection model, such as "abnormal indicator light," "pump casing crack," and "corrosion marks."

[0031] To construct a semantically unified input representation, the scheduling unit designed a multimodal label encoding method based on token expansion and sequence interpolation. Specifically, after segmenting the text input field, a basic semantic token sequence is constructed. Keyword token fragments extracted from the speech recognition transcription field are then inserted into this sequence according to their contextual semantic positions, and boundary annotations are performed using specific delimiters (e.g., [#AUDIO]). The tags extracted from the image recognition label field are not directly embedded as natural language, but are mapped to standardized descriptive sentence fragments through a lookup table, such as "image recognition as: control valve surface anomaly," and added to the end of the sequence in an embedded form. Furthermore, to avoid the input vector length exceeding the model input limit, the system uses a key fragment retention mechanism based on TF-IDF weight calculation to dynamically prune semantically redundant content, retaining only semantic units with high weight scores.

[0032] The concatenated multimodal semantic label sequence is encoded into a unified input vector and fed into a multilingual intent recognition model. This model is a cross-lingual model built on XLM-R or mBERT and fine-tuned with local industrial corpora. The model employs a Transformer architecture, where the parameters of the underlying encoder are frozen at runtime, allowing only the upper-level classification head to undergo task-specific training and updates. The model performs contextual modeling on the input vector and outputs a language-independent high-dimensional semantic embedding vector.

[0033] To address issues such as syntactic structure mismatch and semantic reconstruction ambiguity in multilingual input scenarios, the scheduling unit embeds a semantic consistency discriminant network. This network consists of two components: first, a dual-branch encoder for the input embedding vectors, where each branch corresponds to a language channel (e.g., Simplified Chinese and English), sharing weight parameters but introducing language tag vectors into the structure to achieve semantic domain differentiation modeling; and second, a cross-channel contrastive discriminant module employing a contrastive learning mechanism. During training, it uses the Triplet Loss function to construct training samples by sampling language-consistent and language-inconsistent pairs, enabling it to perform cross-language semantic judgment. During the inference phase, the semantic consistency discriminant network projects the current input embedding vector across the semantic space and outputs a language consistency score, which is a real number between 0 and 1, with scores closer to 1 indicating greater semantic consistency between different modalities and languages.

[0034] After obtaining the language consistency score, the system combines this score with user feedback to update the model's classification capabilities. User feedback includes user satisfaction tags for responses, error correction annotations, or selection behavior regarding recommended content. The system utilizes a streaming incremental fine-tuning mechanism based on the River framework to progressively update the parameters of the classification head of the multilingual intent recognition model. In its implementation, the system first organizes the embedding vectors and feedback results into sample pairs and caches them in a local memory pool. When the memory pool capacity exceeds a set threshold, the fine-tuning mechanism is triggered. The fine-tuning operation is limited to the fully connected weights and bias parameters of the classification head layer, without affecting the self-attention parameters of the underlying Transformer, ensuring the model's stability and generalization ability during long-term operation. Immediately after the model fine-tuning is complete, the updated question classification result is generated, serving as the index for subsequent processing unit agents.

[0035] Through the above design, the scheduling unit not only realizes the fusion and unified encoding of multimodal inputs, but also forms a closed-loop linkage in cross-language semantic understanding and system adaptive evolution capabilities. The entire path has clear dependencies and calling relationships between input, encoding, discrimination, feedback and fine-tuning updates. Moreover, the construction and calling process of key modules are far away from the existing conventional classifier calling methods or static model execution paths, providing a highly reliable solution for the classification of mixed language and multimodal problems in industrial scenarios.

[0036] The processing unit 103 is used to call the domain agent corresponding to the question classification result to generate intelligent question answering processing results. The domain agent includes an equipment maintenance agent, a management consulting agent, a digital operation and maintenance agent, or a technical service agent. The intelligent question answering processing results include predictive maintenance suggestions, structured response content, and semantic annotation information.

[0037] The function of processing unit 103 is to accurately select and invoke the corresponding domain agent based on the question classification results output by scheduling unit 102, and generate structured, predictable, and interpretable intelligent question-answering processing results. These results include not only answers to the question content but also predictive maintenance suggestions, structured response content, and semantic annotation information related to the source information, thereby providing complete and usable knowledge output for subsequent fusion units.

[0038] In the specific implementation process, after receiving the problem classification results output by the scheduling unit, the processing unit 103 selects the most suitable problem-handling entity from four domain agents based on the preset category labels and agent call mapping table. These four types of agents are designed for different industrial maintenance application scenarios. The equipment maintenance agent is typically responsible for hardware fault diagnosis, maintenance plan recommendation, and fault mode identification. It calls equipment operation logs, sensor time-series data, and standard operating procedure documents for joint analysis. The management consulting agent mainly handles management issues such as equipment usage systems, maintenance cycle planning, and supply chain policies, and extracts rule logic from the company's internal control manual, ISO documents, and historical case library to provide answers. The digital operation and maintenance agent focuses on SCADA data analysis, alarm record interpretation, and automatic control logic backtracking in the Industrial Internet of Things (IIoT) scenario, often using time series modeling and state recognition models to assist in judgment. The technical service agent focuses on spare parts traceability, retrieval of maintenance service provider information and related compliance records, and can link blockchain audit nodes and spare parts supply graph databases to improve service transparency.

[0039] The implementation of intelligent agents in various domains is based on an integrated framework of "domain knowledge graph + question answering generation model + context adaptation mechanism". Knowledge acquisition logic, model structure, and calling interface are constructed separately according to task characteristics. The equipment maintenance intelligent agent is implemented by integrating a PHM (Prognostics and Health Management) platform and a rule-driven inference engine. Its underlying components include an industrial sensor data processing module, a fault mode recognition model (such as LSTM or variational autoencoder for time-series anomaly detection), an equipment structure graph embedding module, and a maintenance solution recommendation library. This intelligent agent can extract features from fields such as equipment parameters and fault phenomena in structured problem information, and generate targeted diagnostic suggestions and preventative maintenance time window predictions by combining the equipment-fault-operation ternary relationship in the prior knowledge graph.

[0040] The management consulting agent is implemented based on a document reasoning engine and a policy and rule matching mechanism. Internally, it integrates a semantic index structure of corporate rules and regulations, process management documents, and historical approval cases. It supports generating answers using rule trees or a BERT-based question-and-answer matching model, and can handle questions such as "Does the equipment maintenance plan meet the standards?" or "Is the maintenance frequency aligned with corporate indicators?". This agent also supports historical decision-making path reconstruction to explain the logical processes of corporate management when handling similar problems.

[0041] The digital operation and maintenance intelligent agent is built using multimodal sensor data fusion and SCADA interface parsing modules. Its core is a semantically driven industrial time-series data analysis model, including components such as periodic feature extraction, alarm event clustering, and state transition modeling. It integrates structured data (such as data tables and time-series logs) with unstructured documents (such as technical manuals) to achieve consistent responses from multiple information sources. This intelligent agent can handle questions such as "Is the current abnormal fluctuation related to historical equipment downtime?" and can return fault trend prediction graphs and suggested measures.

[0042] The technical service agent relies on a supply chain graph, blockchain evidence storage system, and contract knowledge extraction model to achieve traceable management of the spare parts supply chain. It possesses entity parsing, association path matching, and service provider qualification verification capabilities. Combined with multi-hop reasoning, it generates service recommendations and procurement suggestions, and outputs a structured response with review records. Each agent registers with processing unit 103 through a unified interface protocol and supports asynchronous response mechanisms and confidence score feedback, ensuring the system has good scalability and stability.

[0043] After the agent is invoked, the processing unit sends structured question information along with relevant semantic context information to the agent's interface. The agent executes a reasoning process based on rule bases, machine learning models, or graph neural networks, and outputs multiple structured fields as its response, supported by knowledge graphs and document semantic indexes. These fields typically include: question response text, predictive maintenance suggestions (e.g., the probability of failure of a component within the next X hours or the optimal maintenance window), semantically labeled locations (e.g., location coordinates in a document or path nodes in a knowledge graph), along with confidence scores and response timestamps.

[0044] The processing unit also includes a response standardization module to unify the format of data returned by different agents, ensuring it conforms to the input specifications of the fusion unit. This module annotates text content by paragraph and hierarchical labels, attaches OCR results and links to the original files for image-related content, and formats predictive data into plottable sequences or standard JSON structures. To ensure information integrity, the processing unit also performs a verification process, checking for missing key fields or abnormal model output in the response, and triggering a backup agent or returning a prompt message according to system policy.

[0045] In addition, to support the self-learning improvement of agent response quality, the processing unit generates a call log after each response, recording the input information, the called agent identifier, the output result summary and confidence score. This log will be passed to the system's feedback learning module to provide a source of supervised data for agent behavior optimization.

[0046] In summary, the processing unit 103 is not only responsible for achieving accurate scheduling and agent invocation based on the question classification results, but also ensures that the content of various responses is complete, structured and predictable, thereby providing a high-quality knowledge output interface for the entire intelligent question answering system and supporting the smooth progress of subsequent multimodal fusion and semantic reasoning processes.

[0047] Furthermore, the device maintenance agent is specifically used for:

[0048] Receive the problem classification result and the device identification field, fault description field and image recognition label field in the structured problem information, and construct a structured input sample for predictive modeling;

[0049] The structured input samples are input into a preset fault mode and health management model. The fault mode and health management model generates a prediction curve based on a time series prediction model and maps the current device status to labels based on the fault label system defined in the knowledge graph.

[0050] Based on the generated prediction curves and fault labels, the maintenance priority score of each equipment component is jointly calculated. The maintenance priority score is a priority value obtained by fusing the confidence function output and the remaining life estimation result according to a preset weighting coefficient.

[0051] Based on the maintenance priority score and the hierarchical relationship between device structure nodes in the knowledge graph, an intelligent question-answering processing result is generated. The intelligent question-answering processing result includes predictive maintenance suggestions, structured response content, and semantic annotation information.

[0052] The processing flow of the equipment maintenance agent begins with receiving the problem classification result output by the scheduling unit. This classification result is typically labeled "Equipment Fault," indicating that the problem to be addressed falls under the category of equipment malfunction, performance degradation, or potential failure risk. Simultaneously, the agent acquires three key fields from the structured problem information output by the input unit: the equipment identification field, the fault description field, and the image recognition label field. The equipment identification field includes basic information such as equipment ID, component number, and installation location code; the fault description field is the core sentence units extracted from the user-input text or speech transcription, typically involving manifest symptoms or alarm signals; the image recognition label field is automatically extracted from the equipment image after processing by a deep learning model (such as YOLOv5 or EfficientDet), and includes standardized descriptions such as "pump shaft misalignment," "filter carbon buildup," and "surface corrosion" mapped by trained labels.

[0053] The above three fields are aggregated into a structured input sample construction module. This module employs field normalization and timestamp alignment mechanisms to uniformly align and organize data fields from multiple modalities into data objects for time series prediction modeling. The construction process involves not only field concatenation but also the application of semantic mapping tables, fault dictionary calibration, and image label confidence filtering. Internally, it generates a structured JSON input package in the form of {Device ID, Component Location, Time Index, Fault Type Code, Image Anomaly Intensity Score}. This input package is fed into a pre-defined fault mode and health management model. This model has a composite structure, containing a device state time series prediction sub-model based on a Transformer structure and a label mapping sub-module for structured event labels.

[0054] The time series prediction sub-model takes component operating parameters (such as vibration frequency, current fluctuation, and temperature gradient) as sequence input, calculates the evolution trend of historical patterns within the current time window through a multi-head attention mechanism, and outputs a prediction curve. This prediction curve is a time-state pair sequence, formatted as {T+1: Normal, T+2: Abnormal probability 0.45, T+3: Abnormal probability 0.72}. In parallel, the label mapping sub-module embeds and encodes image recognition labels and fault description keywords from the input samples, performs semantic alignment and similarity matching with a predefined fault label system in a predefined knowledge graph, and outputs one or more of the most relevant fault labels, such as "bearing wear," "valve hysteresis," and "seal leakage."

[0055] After receiving the two outputs mentioned above, the equipment maintenance agent performs priority score calculation. The scoring logic is a key aspect of this invention, using a confidence function and remaining lifetime estimate as primary inputs. The confidence function is driven by the maximum anomaly probability in the prediction curve, while simultaneously normalizing the anomaly score of the current state using an image recognition model. For example, if the anomaly probability of the prediction curve at time T+2 is 0.72, and the anomaly score of the same component in image recognition is 0.65, then the confidence function takes their weighted average (e.g., 0.7). The remaining lifetime estimate is obtained through another deep regression network prediction, outputting the estimated remaining working hours of the component in its current state. The two are fused using preset weighting coefficients (e.g., confidence weight 0.6, RUL weight 0.4), ultimately outputting a maintenance priority score between 0 and 1, with a higher score indicating greater maintenance urgency.

[0056] This score serves as the foundational input for subsequent knowledge reasoning modules, combining structural relationships within the knowledge graph for hierarchical weighted judgment. The device structure graph is organized as a graph database, where nodes represent components and edges represent physical connections and functional dependencies. For example, if a "functional dependency" edge exists between the "bearing" node and the "main shaft" node, and the score of the "bearing" node exceeds a threshold, the "main shaft" node will also be assigned a secondary reasoning warning label. This hierarchical logic can be executed based on Graph Convolutional Networks (GCNs) or path activation algorithms.

[0057] Finally, based on the scoring results and the graph propagation relationship, the equipment maintenance agent generates intelligent question-and-answer processing results, including: a set of predictive maintenance suggestions, specifically including suggested replacement timing and recommended operation schemes; a set of structured response content, including a scoring detail table, failure tag groups, and graph reasoning chains; and a set of semantic annotation information, indicating the corresponding location of the suggested content in the knowledge document or screenshot coordinates. All outputs are structured and returned to the fusion unit for subsequent response generation.

[0058] The aforementioned approach not only possesses execution details not previously described in existing technologies regarding data structure organization, parallel model design, scoring generation logic, and structure graph-driven approaches, but also achieves robust maintenance suggestion generation in high-noise, multimodal input environments through explicit weighting of control parameters and coupling of confidence scoring. This provides an implementable, interpretable, and updatable technical solution for equipment maintenance scenarios in industrial intelligent question-and-answer systems.

[0059] The following is a reference implementation code for an equipment maintenance intelligent agent:

[0060] import json

[0061] import numpy as np

[0062] #Example of structured input fields

[0063] input_data = {

[0064] "question_class": "device failure class",

[0065] "device_id": "PUMP_23A",

[0066] "component_id": "BEARING",

[0067] "location": "Line_2_Section_5",

[0068] "description": "Periodic abnormal noise is occurring during operation; an alarm has been triggered."

[0069] "image_tags": [{"label": "bearing crack", "confidence": 0.78}, {"label": "excessive oil contamination", "confidence": 0.62}],

[0070] "time_series_data": {

[0071] "vibration": [0.3, 0.35, 0.42, 0.6, 0.7],

[0072] "current": [2.1, 2.4, 2.5, 2.9, 3.2],

[0073] "temperature": [65, 66, 68, 71, 74]

[0074] }

[0075] }

[0076] # Structured Sample Building Module

[0077] def build_structured_sample(input_data):

[0078] #Construct a standard input structure for the model, including component identification, fault semantic labeling, and image anomalies. #Confidence

[0079] structured_sample = {

[0080] "device_id": input_data["device_id"],

[0081] "component_id": input_data["component_id"],

[0082] "location": input_data["location"],

[0083] "fault_description": input_data["description"],

[0084] "fault_keywords": extract_fault_keywords(input_data["description"]),

[0085] "image_tags": filter_top_image_tags(input_data["image_tags"]),

[0086] "time_series": input_data["time_series_data"]

[0087] }

[0088] return structured_sample

[0089] def extract_fault_keywords(text):

[0090] # Extract keywords using a keyword dictionary or language model

[0091] keywords = []

[0092] If "abnormal noise" appears in the text:

[0093] keywords.append("bearing wear")

[0094] If "alarm" is in the text:

[0095] keywords.append("Running offset")

[0096] return keywords

[0097] def filter_top_image_tags(tags, threshold=0.7):

[0098] # Filter image labels and retain those with higher confidence levels

[0099] return [tag for tag in tags if tag["confidence"] >= threshold]

[0100] # Fault Prediction Model and Health Management System (PHM) Main Interface

[0101] def predict_with_phm_model(sample):

[0102] #Process input samples and generate prediction curves and fault label sets

[0103] # Implementation details: Generate a prediction curve by calling a preset time series model.

[0104] prediction_curve = {

[0105] "T+1": {"status": "normal", "prob": 0.1},

[0106] "T+2": {"status": "warning", "prob": 0.45},

[0107] "T+3": {"status": "fail_likely", "prob": 0.72}

[0108] }

[0109] # Simulated Tag Matching: Semantic Alignment + Tag Mapping

[0110] fault_labels = list(set(sample["fault_keywords"] + [tag["label"]for tag in sample["image_tags"]]))

[0111] return prediction_curve, fault_labels

[0112] # Remaining lifetime estimation model (simplified)

[0113] def estimate_rul(sample):

[0114] #Estimate the remaining lifespan (in hours) of a component using a regression network or rule base.

[0115] #Implementation: Predicting based on vibration trends

[0116] vibration = np.array(sample["time_series"]["vibration"])

[0117] if vibration[-1] - vibration[0] > 0.4:

[0118] return 40 # High-risk equipment with a short estimated lifespan

[0119] return 120 # Estimated lifespan is relatively long

[0120] # Confidence function + remaining lifetime fusion -> maintenance priority scoring

[0121] def calculate_priority_score(pred_curve, image_tags, rul, alpha=0.6,beta=0.4):

[0122] #Calculate maintenance priority score, fuse prediction curve with image anomaly confidence and remaining lifetime

[0123] pred_probs = [v["prob"] for v in pred_curve.values()]

[0124] pred_conf = max(pred_probs)

[0125] # Average confidence level of image anomalies (filtered)

[0126] img_conf = np.mean([t["confidence"] for t in image_tags]) ifimage_tags else 0.5

[0127] combined_conf = (pred_conf + img_conf) / 2 # Overall confidence level

[0128] normalized_rul_score = 1 - min(rul / 200.0, 1.0) # Remaining lifespan is converted into a risk score; the lower the score, the higher the risk score.

[0129] priority_score = alpha * combined_conf + beta * normalized_rul_score

[0130] return round(priority_score, 3)

[0131] # Knowledge Graph Structure Propagation

[0132] def propagate_score_through_graph(component_score, component_id):

[0133] # Generate inference chains based on the propagation priority scores of the device structure diagram.

[0134] graph = {

[0135] "BEARING": ["SHAFT"],

[0136] "SHAFT": ["GEAR"],

[0137] "GEAR": []

[0138] }

[0139] propagation = {}

[0140] propagation[component_id] = component_score

[0141] queue = [(component_id, component_score)]

[0142] while queue:

[0143] current, score = queue.pop(0)

[0144] for neighbor in graph.get(current, []):

[0145] propagated_score = round(score * 0.6, 3) # Decay propagation

[0146] propagation[neighbor] = propagated_score

[0147] queue.append((neighbor, propagated_score))

[0148] return propagation

[0149] # Output intelligent question-answering processing results

[0150] def generate_response(prediction_curve, fault_labels, priority_score,propagation_chain):

[0151] # Construct the response structure, including predicted suggestions, structural scores, and semantically labeled locations.

[0152] result = {

[0153] "maintenance_advice": f"It is recommended to check / replace the {fault_labels[0]} component within 72 hours, with a priority score of {priority_score}",

[0154] "structured_output": {

[0155] "score_detail": {

[0156] "component_scores": propagation_chain,

[0157] "top_component": max(propagation_chain, key=propagation_chain.get)

[0158] },

[0159] "fault_tags": fault_labels

[0160] },

[0161] "semantic_annotations": {

[0162] "source_text_mapping": "Location of Figure 3 on page 12 of the document 'Maintenance Manual 2023 Edition'",

[0163] "image_highlight_coords": [120, 300, 180, 360]

[0164] }

[0165] }

[0166] return result

[0167] # Main execution flow simulation

[0168] sample = build_structured_sample(input_data)

[0169] prediction_curve, fault_labels = predict_with_phm_model(sample)

[0170] rul = estimate_rul(sample)

[0171] priority_score = calculate_priority_score(prediction_curve, sample["image_tags"], rul)

[0172] propagation_chain = propagate_score_through_graph(priority_score,sample["component_id"])

[0173] response = generate_response(prediction_curve, fault_labels,priority_score, propagation_chain)

[0174] # Print the final generated intelligent question-and-answer results (structured output)

[0175] print(json.dumps(response, indent=2, ensure_ascii=False))

[0176] The fusion unit 104 is used to perform multi-hop semantic reasoning based on the intelligent question-answering processing results and by fusing local knowledge base information, graph database information and network retrieval information to obtain multimodal response information, which includes text descriptions, image screenshots, prediction curves and external recommendation links.

[0177] The main function of the fusion unit 104 is to further integrate multi-source heterogeneous knowledge resources based on the intelligent question-answering results generated by the processing unit 103, and generate more valuable multimodal response information through semantic reasoning. Its function is not limited to information integration, but also constructs cross-domain knowledge linkage through graph relationship modeling, context alignment and multi-hop semantic path search, so as to realize the structured, visualized and predictive enhancement of the response.

[0178] In actual deployment, the fusion unit first receives the intelligent question-and-answer processing results from the processing unit, which includes predictive maintenance suggestions, structured response content, and semantic annotation information. The fusion unit identifies key entities, fault tags, component numbers, or semantic anchors in the response content and uses these as the starting point for semantic fusion retrieval. The system's built-in local knowledge base typically includes documents such as maintenance manuals, equipment structure diagrams, and enterprise operation and maintenance records, using full-text indexing (such as Elasticsearch) combined with semantic embedding models (such as Sentence-BERT) for knowledge matching. The graph database is built on platforms such as Neo4j or TigerGraph, storing structured triple relationships such as multi-level topological relationships between devices and faults, component dependency paths, and mappings to maintenance operations. Online retrieval information includes calling external data sources such as Google APIs, industrial forum crawlers, and standard document library interfaces to supplement local knowledge when it is lacking or has low confidence.

[0179] The fusion unit maps knowledge from these different sources into a unified semantic space and performs multi-hop path searches centered on the semantics of the current problem. For example, when the agent returns a predictive maintenance suggestion of "abnormal pump shaft vibration," the fusion unit identifies the system path of the pump shaft through the knowledge graph and further searches for related historical failure cases, maintenance frequency of adjacent components, and inventory status of alternative components. With the support of the knowledge graph, the system can perform 2-hop or 3-hop path expansion, such as "pump shaft → pump body → abnormal flow → sensor drift," thereby discovering potential hidden causes or cross-referenced data evidence.

[0180] The fusion results are not limited to structured text but also include images and charts. The system can automatically call document screenshot tools or SCADA interfaces to embed key reference paragraphs or historical fluctuation curves into the response results as images. Combined with OCR text positioning and highlighting technology, users can intuitively see the annotations of key information on the images. In addition, predictive results such as time series analysis curves will be inserted into the response in standard SVG or Base64 format, accompanied by text explanations; if the external link is a recommended service platform, standard specification page, or repair service provider directory, clickable external recommendation links will be automatically generated, and link call logs will be recorded in the background for subsequent optimization.

[0181] To ensure the reliability and relevance of the responses, the fusion unit also incorporates a confidence-based fusion mechanism. Each type of information source (local knowledge base, graph database, online search) has corresponding weight parameters and is dynamically scored based on factors such as the timeliness, citation frequency, and structural integrity of the source data. The final response content is fused from all sources according to their weights, retaining the source paths and scores to help users understand the basis provided by the system. After the response is generated, the fusion unit combines structured text, screenshots, predicted charts, and external links into a unified multimodal response package and outputs it in a standardized format to the interface module or subsequent feedback learning unit.

[0182] The complete process of the fusion unit 104 ensures that the system can not only generate targeted preliminary answers, but also construct composite response results with knowledge depth and visual expressiveness based on multi-source information, enabling users to more comprehensively understand the background of the problem and the coping strategy.

[0183] Furthermore, the fusion unit is specifically used for:

[0184] Based on the fault label information contained in the intelligent question answering processing results, a semantic path expansion operation with a limited number of hops is performed in the graph database, starting from the knowledge graph node corresponding to the fault label. The semantic path expansion operation includes filtering according to the edge type, path weight and semantic consistency of the intersection node defined in the graph, and constructing multiple inference paths with fault semantic association.

[0185] The knowledge fragments corresponding to the nodes involved in each reasoning path are used as candidate answer sources. The original content, citation frequency, update timeliness and semantic relevance of the corresponding knowledge fragments in the graph database, local knowledge base and online search information are obtained respectively, and the source confidence score of each source is calculated.

[0186] Based on the source confidence scores, the candidate knowledge fragments from multiple sources are reorganized and deredundant according to a preset dynamic fusion strategy, and a multimodal response information draft with a unified structure is output. The multimodal response information draft includes text descriptions, image fragments and prediction curves that are weighted by confidence scores.

[0187] Based on the semantic anchor points described in the text, the optical character recognition module is invoked to perform paragraph-level text localization and structure recognition on the original document image, and a highlighted border is generated in the image segment to mark the position of the semantic anchor points in the original text, thereby generating multimodal response information with visual cues and semantic correspondence.

[0188] To realize the graph database inference, multi-source confidence fusion, and image-text linkage annotation functions described in the fusion unit, the entire implementation process should have a highly integrated and modular data flow design. This section describes in detail the specific execution method of each key step in natural and fluent language to ensure that those skilled in the art can accurately implement it and avoid common problems in the prior art such as abstract generalization and unclear operation paths.

[0189] First, after receiving the intelligent question-and-answer processing results output by the equipment maintenance agent, the fusion unit extracts the fault label information contained therein. These fault labels are standard codes after semantic normalization, such as "bearing_wear", "shaft_misalignment", or "seal_leakage", corresponding to node identifiers in the knowledge graph of the pre-defined graph database. These node identifiers not only serve as unique key-value indexes of the graph but also include associated edge attributes, node type (such as phenomenon, cause, and suggested handling), and a description of their semantic relationships with other nodes.

[0190] Starting from the labeled node, the system performs semantic path expansion operations in the graph database. Path expansion follows a hop limit, generally limited to three hops or less, to avoid semantic generalization and lengthy inference chains. In each hop, the system filters edges based on predefined edge types in the graph, for example, only allowing edges of the "causal relationship," "functional dependency," or "maintenance reference" categories to be activated. Simultaneously, semantic consistency is calculated for intermediate nodes in the path. Specifically, language models such as BERT or RoBERTa are used to encode the semantic representations of the current path node and the initial fault label, respectively, and the vector cosine similarity is calculated to ensure the overall path has directional consistency in the semantic space. When a path contains multiple intersection nodes (i.e., the same node belongs to multiple paths), the system retains the path branch with the highest semantic consistency score and calculates the path weight value. The weight is defined as the product of the cumulative edge weights of each hop in the path and the average semantic similarity.

[0191] For each retained reasoning path, the system extracts the knowledge fragments linked to all nodes in the path as the core source for constructing candidate answers. Each node in the knowledge graph contains resource indexes such as documents, manuals, and operation logs pointing to local knowledge bases or external networked knowledge platforms. The system performs a source confidence assessment on each candidate fragment in four dimensions: first, citation frequency, which counts the frequency of the fragment's calls in historical work orders, user queries, and recommended answers; second, update timeliness, which records the timestamp of the fragment's last content update and reflects its timeliness in a weighted manner; third, the contextual completeness and paragraph position stability of the original content, used to measure its sufficiency as an independent knowledge unit; and fourth, semantic relevance, which is determined by similarity encoding matching between the full text of the candidate fragment and the fault tag to obtain the content fit. After the four indicators are normalized, a confidence score for the source is generated through a preset function, for example:

[0192] Source confidence score = 0.4 × Citation frequency score + 0.3 × Update timeliness score + 0.2 × Paragraph completeness score + 0.1 × Semantic relevance score.

[0193] Subsequently, the system executes a dynamic fusion strategy based on the confidence scores of multiple candidate sources. This strategy first sorts candidate knowledge fragments in descending order of confidence and then performs a content reorganization process, including disambiguation and merging of repeated paragraphs in the text, redundancy detection of image content (such as multiple occurrences from the same device's perspective), and alignment of trend segments in the prediction curve. A confidence-weighted mechanism is introduced during the fusion process: when multiple descriptions of a fault cause or suggestion exist, the description from the source with the highest confidence is retained first, followed by the description with the second highest confidence as supplementary annotations. The final output constitutes a structurally unified draft of multimodal response information, with a format including sorted text fields, matched image fields, and predicted time series fields. Each field carries metadata tags to identify its source and confidence weight.

[0194] For each paragraph of text description in the draft response, the system further invokes the embedded optical character recognition (OCR) module to analyze the matching original document image and locate the semantic anchor point within the document paragraph. The OCR module uses a pre-trained document structure recognition model (such as LayoutLM or Donut) to reconstruct the block-level structure of the image, mapping the identified paragraphs to text anchor points and generating location marker coordinates. The system then uses these coordinates as the center area to draw a highlighted border in the image fragment, employing a border color-confidence level mapping rule (e.g., green for high confidence, yellow for medium confidence) for visual cues. For annotation areas located in complex backgrounds or overlapping layers, the system automatically adjusts the border size and annotation arrow direction to ensure users can quickly identify the annotation content in the visual interface.

[0195] Finally, the fusion unit outputs the aforementioned multimodal response information with text and image linkage to the response generation module, where subsequent processes will further perform operations such as language polishing, tone style generation, or speech transcription.

[0196] Furthermore, the fusion unit is also used for:

[0197] Based on the question type field in the question classification results, a preset modal configuration rule table is invoked to dynamically determine the combination method of each modal element in the multimodal response information draft. The combination method includes text description, prediction curve, image fragment, and document jump link. When the question type field is characterized as "equipment failure", the combination of prediction curve and image fragment is given priority; when it is characterized as "procedure consultation", the combination of text description and document jump link is given priority.

[0198] After generating the multimodal response information draft, a unique traceable tag is attached to each modal element. The traceable tag points to the original knowledge fragment source node and records the node number sequence, edge weight cumulative value and path semantic similarity score of the node participating in the reasoning path in the graph database to support subsequent path backtracking and result verification.

[0199] When the multimodal response information draft is displayed and sorted in a structured manner, a reinforcement learning sorting model based on user interaction feedback is introduced. The reinforcement learning sorting model receives the user's click behavior, dwell time and feedback rating for different modal combinations in historical interactions as reward signals, and dynamically updates the display order weight parameters to form a user-personalized multimodal display configuration.

[0200] In the final multimodal response information, the traceable tags and original path annotation mechanism are retained, and expandable links are provided to jump to the original knowledge fragments, graph paths and source documents, so as to make the answers traceable, verifiable and interactive, and improve the transparency and evolution capability of the question answering system for complex tasks in industrial scenarios.

[0201] In this invention, the fusion unit not only undertakes the tasks of content aggregation and multimodal response information generation, but also further assumes the deep processing functions of multimodal content structure configuration, source path tracing, and user feedback-driven personalized ranking. To ensure that different types of questions can obtain structurally adapted, accurately expressed, interpretable, and evolving response results, this invention designs a dynamic modality configuration mechanism, a traceable path recording mechanism, and a reinforcement learning-driven display ranking optimization mechanism within the fusion unit, with each part operating collaboratively.

[0202] First, after the fusion unit receives the intelligent question-and-answer processing results output by the equipment maintenance agent, it immediately reads the question classification results contained therein and parses the question type field from these results. This field indicates whether the current question belongs to different types such as "equipment failure," "procedure consultation," or "comprehensive diagnosis." The system has a built-in modal configuration rule table, which consists of several pre-defined modal combination strategies. Each strategy corresponds to a specific question type and defines the priority and display strategy of each modal element. The scope of modal elements includes text descriptions, prediction curves, image fragments, and document jump links, with each modality corresponding to different response expression dimensions and source paths. Taking "equipment failure" questions as an example, the system will prioritize selecting the future state curve generated by the prediction sub-model and the image fragments extracted from the equipment image as the main components; while for "procedure consultation" questions, it will prioritize structured text descriptions and document jump links, where the jump links point to the anchor page of the corresponding procedure clause in the local knowledge base or online query.

[0203] After selecting the modal combination structure, the system will perform traceability annotation on each modal element. This process involves attaching a unique tag to each response fragment. This tag not only indicates the generation path of the information but also includes the sequence of source node numbers in the graph database, the cumulative edge weights along that path, and the semantic similarity score of the path. This information will be embedded in the metadata of each modal element in a JSON structure for easy subsequent retrieval and visualization. The node number sequence records all intermediate graph nodes traversed from the initial fault tag node to the final information fragment, the cumulative edge weights reflect the total semantic credibility along that path, and the semantic similarity score is an important basis for selecting the optimal path during the path selection stage.

[0204] After all modal elements are constructed and traceable tags are attached, the system enters the response display and ranking stage. This stage employs a ranking optimization model based on reinforcement learning. Its core idea is to continuously update the display strategy by introducing user interaction feedback as a reward signal. This model extracts user click behavior, dwell time, swipe trajectory, and human rating data for different modal combinations from historical response interaction records, transforming them into state-behavior-reward triples in a reinforcement learning environment. For example, if a user shows a longer dwell time and gives positive feedback ratings in a set of responses containing prediction curves, the system will automatically increase the display priority of such modal combinations in similar questions. In terms of algorithm implementation, a deep reinforcement learning model based on policy gradients, such as REINFORCE or PPO, can be used to ensure the ranking strategy maintains stability and generalization ability when facing a large number of users and question types.

[0205] Ultimately, when presenting the response to the user, the system not only personalizes the structured multimodal response information but also highlights traceable tags and path annotations to ensure the traceability and interactivity of the response content. All modal elements are configured with expandable links, with links targeting the corresponding reasoning path graph in the graph database, detailed knowledge graph node information, screenshots of the original document, or jump links. Semantic anchors embedded in the text description are precisely located at the paragraph positions in the original document image by the OCR module, and visually highlighted borders are rendered in the corresponding image fragments, helping users quickly locate the context of the answer in the original material and creating a semantically consistent document annotation interface.

[0206] The following is a reference implementation code for the fusion unit:

[0207] import json

[0208] import uuid

[0209] import random

[0210] import torch

[0211] import torch.nn as nn

[0212] import torch.optim as optim

[0213] import torch.nn.functional as F

[0214] from typing import List, Dict

[0215] # Example Modal Configuration Rule Table

[0216] MODAL_CONFIG = {

[0217] "Equipment Failure Category": ["Prediction Curve", "Image Clip"],

[0218] "Procedure Consultation": ["Text Description", "Document Link"],

[0219] "Comprehensive Diagnostic Category": ["Text Description", "Prediction Curve", "Image Clip", "Document Link"]

[0220] }

[0221] # Modal content generator simulation interface (to connect to its respective sub-model)

[0222] def generate_modal_content(modal_type, question):

[0223] return f"Simulated {modal_type}:{question}"

[0224] # Simulated Path Tracing Information Generator

[0225] def generate_trace_metadata():

[0226] return {

[0227] "source_node_ids": ["N1", "N2", "N3"],

[0228] "edge_weight_sum": round(random.uniform(0.5, 1.0), 4),

[0229] "semantic_similarity": round(random.uniform(0.6, 0.95), 4)

[0230] }

[0231] # Fusion Unit Class

[0232] class FusionUnit:

[0233] def __init__(self):

[0234] self.modal_config = MODAL_CONFIG

[0235] self.sorter = ReinforceRanker(state_size=10, action_size=4) # Reinforcement learning sorter, the action space represents the order of modality combinations.

[0236] def process(self, question_result):

[0237] question_type = question_result["type"]

[0238] content = question_result["content"]

[0239] modal_sequence = self.modal_config.get(question_type, [])

[0240] multi_modal_draft = []

[0241] for modal in modal_sequence:

[0242] modal_content = generate_modal_content(modal, content)

[0243] trace = generate_trace_metadata()

[0244] element = {

[0245] "modal_type": modal,

[0246] "content": modal_content,

[0247] "trace_tag": str(uuid.uuid4()),

[0248] "trace_metadata": trace,

[0249] "link": f"http: / / example.com / path?modal={modal}&id={trace['source_node_ids'][-1]}"

[0250] }

[0251] multi_modal_draft.append(element)

[0252] # Construct a status representation (which can be populated based on actual user profiles or problem characteristics)

[0253] state = torch.rand(10)

[0254] action_idx, log_prob = self.sorter.select_action(state)

[0255] sorted_draft = self.apply_action_sort(multi_modal_draft,action_idx)

[0256] # Return the final multimodal response result and log_prob for subsequent training.

[0257] return sorted_draft, log_prob

[0258] def apply_action_sort(self, draft, action_idx):

[0259] #Action sorting strategy (for handling more granular display strategies)

[0260] return sorted(draft, key=lambda x: (hash(x["modal_type"]) +action_idx) % 10)

[0261] # Reinforcement Learning Ranking Machine (REINFORCE Implementation)

[0262] class ReinforceRanker(nn.Module):

[0263] def __init__(self, state_size, action_size):

[0264] super(ReinforceRanker, self).__init__()

[0265] self.fc1 = nn.Linear(state_size, 64)

[0266] self.fc2 = nn.Linear(64, action_size)

[0267] self.optimizer = optim.Adam(self.parameters(), lr=0.001)

[0268] def forward(self, x):

[0269] x = F.relu(self.fc1(x))

[0270] return F.softmax(self.fc2(x), dim=-1)

[0271] def select_action(self, state):

[0272] probs = self.forward(state)

[0273] m = torch.distributions.Categorical(probs)

[0274] action = m.sample()

[0275] return action.item(), m.log_prob(action)

[0276] def update_policy(self, log_probs, rewards, gamma=0.99):

[0277] G = 0

[0278] loss = 0

[0279] for r, log_prob in zip(reversed(rewards), reversed(log_probs)):

[0280] G = r + gamma * G

[0281] loss -= log_prob * G

[0282] self.optimizer.zero_grad()

[0283] loss.backward()

[0284] self.optimizer.step()

[0285] # Example call logic

[0286] if __name__ == '__main__':

[0287] fusion_unit = FusionUnit()

[0288] # Result returned by the simulated question-and-answer platform

[0289] result = {

[0290] "type": "Equipment Failure Category",

[0291] "content": "Abnormal noise from fan blades"

[0292] }

[0293] draft, log_prob = fusion_unit.process(result)

[0294] # Simulated User Feedback Rewards

[0295] reward = 1.0 if draft[0]["modal_type"] == "predicted curve" else 0.0

[0296] fusion_unit.sorter.update_policy([log_prob], [reward])

[0297] # Show final results

[0298] for m in draft:

[0299] print(json.dumps(m, ensure_ascii=False, indent=2))

[0300] Furthermore, the aforementioned intelligent question-and-answer system for industrial equipment maintenance based on multi-agent collaboration also includes:

[0301] The control unit is used to receive the multimodal response information and user feedback information, update the multilingual intent recognition model and knowledge weight configuration based on the user feedback information, and perform permission verification and tamper-proof storage processing on the multimodal response information using a role-based access control mechanism and a blockchain-based operation log mechanism.

[0302] This invention further introduces a control unit, aiming to construct a closed-loop question-answering system architecture with self-optimization capabilities, security and controllability, and reliable auditing functions. The control unit is responsible for receiving multimodal response information output by the fusion unit and feedback information returned by the user terminal. Based on this, it dynamically adjusts the semantic recognition model and knowledge fusion mechanism within the system, thereby enabling the system to learn and adaptively optimize user behavior during actual operation. Furthermore, the control unit also undertakes critical data access control and auditable operation recording tasks, ensuring that the system operation meets the information security and compliance requirements of industrial scenarios.

[0303] In implementation, the control unit first receives user feedback on the responses from the front-end or interface module. This feedback can include user satisfaction ratings, correction suggestions, error annotations, supplementary questions, or comparative queries. The system associates this feedback data with the original structured question information and the corresponding multimodal response information, constructing feedback sample data pairs as the data source for incremental training. After preprocessing, this feedback data is automatically labeled as positive or negative feedback and input into the multilingual intent recognition model in the system for fine-tuning the model's classification weights. Specifically, the control unit calls a model training component based on a streaming update framework (such as River or a similar lightweight neural network parameter update module) to quickly adjust only the weights of the classification head, keeping the core parameters of the language model stable, thus balancing stability and adaptability. During this process, the system maintains a feedback buffer, accumulating samples according to time windows and periodically triggering the training process, while recording model change logs for recovery or backtracking.

[0304] In terms of knowledge fusion, the control unit dynamically scores the frequency of use and contribution of responses to local knowledge bases, graph databases, or online search results based on feedback, and adjusts the weighting of various information sources during knowledge fusion. For example, if multiple user feedback indicates that a certain knowledge fragment is misleading, its weight in future searches will be reduced, and it may even be placed on a gray list prohibited from indexing. Conversely, frequently cited document paths or graph paths will receive higher confidence priority and be cited more frequently in multi-hop inference. This dynamic knowledge weighting mechanism is driven by a configurable parameter file and, in conjunction with a data version control mechanism, ensures information consistency and controllability during knowledge evolution.

[0305] Regarding system security and access control, the control unit incorporates an RBAC (Role-Based Access Control) mechanism. The system first identifies the current user's role category through user authentication (such as account, access token, or device signature), e.g., field operator, system engineer, maintenance manager, or external auditor. Then, based on the role-permission mapping table, it determines whether the user has permission to view a specific type of response information, especially content involving core technical documents, historical alarm curves, or supply chain compliance information. If the user does not have access rights, the system automatically masks sensitive fields and returns a prompt message to prevent information leakage risks.

[0306] To ensure the authenticity and immutability of all operation records, the control unit also integrates a blockchain operation log mechanism. For each user query, multimodal response generation, model update, or knowledge weight adjustment, the system generates a corresponding operation summary, constructs a hash, and records it in a private blockchain. Blockchain nodes can be deployed on enterprise local servers, edge computing nodes, or cloud control platforms to ensure consensus synchronization and long-term storage of log data across multiple terminals and locations. If future audits, accountability tracking, or system status reverts are required, the blockchain records can provide technical support as an authoritative chain of evidence.

[0307] In summary, the control unit not only realizes a closed-loop self-learning path from user feedback to model optimization and knowledge weight adjustment, but also ensures the compliance and controllability of system data access through a role-based access control mechanism, and builds an auditable and trustworthy operation record system with the help of a blockchain log mechanism.

[0308] Furthermore, the control unit is specifically used for:

[0309] The system receives semantic difference values ​​between the task expression in the user feedback information and the multimodal response information, interface interaction behavior data, and access user identification information. The semantic difference values ​​are obtained by calculating the vector cosine similarity between the user's natural language input and the system's semantic parsing results in a unified embedding space. The interface interaction behavior data includes the number of clicks, dwell time, and feedback rating of each modal element in the multimodal response information. The access user identification information includes an access timestamp, access permission level, and access path identification code.

[0310] Based on the semantic difference value and the interface interaction behavior data, modal elements with semantic deviations in the response content are extracted, and a set of correction parameters is generated, which includes language adjustment prompts, response completion suggestions and modal weight correction suggestions. The set of correction parameters records the key expression offset points between the original response content and the user's actual needs, forming feedback samples that can be used to improve the response generation process.

[0311] The modality composition strategy of the multimodal response information is reconstructed using the modified parameter set, specifically including: optimizing the text description expression according to the language adjustment prompts; adding missing image fragments, prediction curves or document jump links according to the response completion suggestions; and updating the modality priority sorting in the modality configuration rule table according to the modality weight correction suggestions, so as to dynamically adjust the generation structure of the multimodal response information in subsequent questions of the same type.

[0312] The access user identification information is bound to each feedback record in the correction parameter set, and a unique digest value is generated using a hash algorithm and timestamp encryption method. This digest value, together with the multimodal response information, the semantic difference value, and the interface interaction behavior data, is written into the on-chain evidence storage structure and broadcast to the blockchain node network to form an audit entry with tamper-proof characteristics, supporting the source tracing, responsibility determination, and evolution verification of any of the multimodal response information.

[0313] In the intelligent question-and-answer system for industrial equipment maintenance based on multi-agent collaboration described in this invention, the control unit not only possesses basic response receiving and distribution functions, but also further implements a complete logical chain including deep structural analysis of user feedback information, semantic deviation quantification processing, dynamic adjustment of modal strategies, and result storage. The following details the technical path executed by the control unit when implementing this invention.

[0314] First, after receiving user feedback, the control unit extracts three core data elements from it: the semantic difference value between the task expression and the response content, interface interaction behavior data, and the user's identification information. The "semantic difference value" quantifies the expression offset between the system-generated multimodal response and the user's original task intent. The control unit uses a unified semantic embedding space modeling technique to embed the user's natural language input and the system's response content into the same high-dimensional vector space, and then calculates the cosine similarity between them to obtain their semantic consistency. When the cosine similarity value is lower than a system-set threshold (e.g., 0.85), it indicates a significant semantic deviation, requiring further processing.

[0315] Meanwhile, collecting user interaction behavior data is also a core function of this control unit. This primarily includes metrics such as the number of clicks, dwell time, swipe trajectory, and user ratings for each modal element (e.g., text description, prediction curve, image fragment, document jump link) in the response interface. The control unit uses front-end tracking technology combined with a back-end event monitoring mechanism to record user actions on the response content in real time. For example, if a prediction curve is clicked twice, dwells for 6 seconds, and the user's final rating is "somewhat dissatisfied," this modality is recorded as a negative feedback sample. This behavioral data is encoded into structured entries and categorized into the feedback record set under the current session identifier. Simultaneously, the control unit also receives the access user's identity information, including the access timestamp (accurate to the second), the user's access permission level (e.g., maintenance worker, engineer, administrator, etc.), and a unique access path identifier (used to record which entry point the user used to access the current Q&A system). All of this data together constitutes the access user identification information.

[0316] After completing the basic data extraction, the control unit enters the semantic deviation identification and correction suggestion generation stage. The system first filters out question-and-answer interaction entries with deviations exceeding a threshold based on semantic difference values. Then, combining this with interaction behavior data, it evaluates each modal element in the response content, identifying modal elements with incomplete semantic coverage, missing content, or expressions that do not conform to the user's intent. The control unit then generates a set of correction parameters for these problematic modalities, containing three key subsets: language adjustment prompts, response completion suggestions, and modality weight correction suggestions. Language adjustment prompts point out incomplete expressions or ambiguous wording in the response text, such as unclear parameter units or operation sequences in operation instruction questions. Response completion suggestions predict the user's desired content format based on their behavior; for example, if a user fails to zoom in on a device image, the system suggests adding a high-resolution image fragment of that area. Modality weight correction suggestions provide new weighting factors to the modality configuration rule table based on the click-through rate and rating performance of different modalities in the interaction; for example, adjusting the "prediction curve" from the default third to the second.

[0317] The generated set of corrected parameters is used not only to improve the current question-and-answer content but also to reconstruct the response generation strategy for similar questions in subsequent question-and-answer sessions. The control unit dynamically adjusts the modality combination structure based on the corrected parameter set. First, it fine-tunes the language of the text description to ensure complete coverage of the task intent and maintain professional expression using domain terminology. Second, it automatically calls the image library and document indexing system based on completion suggestions, inserting missing image fragments or adding new jump anchors. Finally, it updates the priority array of each modality item in the modality configuration rule table based on modality weight correction suggestions. This array directly affects the modality selection and sorting of response information in similar questions in the future, thereby achieving personalized optimization.

[0318] To achieve full lifecycle evidence preservation and traceability of question-and-answer behavior and feedback trajectories, the control unit will construct immutable on-chain data records using encryption technology after completing the aforementioned strategy reconstruction. First, the access user identification information is bound to each correction parameter record, and this is combined with structured content such as multimodal response information, semantic difference values, and interface interaction behavior data. A digest value is calculated using a secure hash algorithm such as SHA-256. The control unit embeds a timestamp field in the digest value to ensure that even if the same user submits the same feedback content, their record remains globally unique. Subsequently, this digest value and its corresponding data entry are written to the blockchain ledger and broadcast to all nodes in the blockchain network using a consensus algorithm, forming an irreversible transaction record. This record supports future user requests for accountability, feedback callbacks, or improvement path verification for any response content. The system can reconstruct the multimodal composition strategy, feedback source, and update chain at the time based on the records in the blockchain, thereby ensuring the system's evolution process has complete interpretability and data compliance.

[0319] Through the above structural design, the control unit of the present invention achieves the technical effect of feedback-driven content evolution, adaptive strategy update and compliance traceability in one, effectively overcoming the problems of static response strategies, ignoring user differences and the inability to track historical updates in existing industrial question-and-answer systems.

[0320] Here is a specific example:

[0321] Taking a real-world scenario as an example, an equipment inspector with intermediate-level privileges submitted a natural language question, "What faults can frequent equipment start-ups and shutdowns cause?" during the annual maintenance of a compressor unit. After intent recognition and knowledge retrieval, the system generated a multimodal response containing a text description, a prediction curve, and a historical fault image fragment. However, the user clicked to view the prediction curve zero times, stayed for less than one second, then marked "current problem unresolved" in the feedback area and mentioned in the manually supplemented description that they were "more concerned about the allowable number of frequent start-ups and shutdowns in the specifications." The control unit immediately analyzed the interaction and the supplementary text, identifying that the modal combination in the original response did not fully cover the user's key needs. Through semantic embedding space calculation, the semantic difference between the original response and the user's supplementary expression was only 0.67, far below the system threshold, indicating a significant deviation.

[0322] The system then generates a language adjustment prompt, "Add a citation to the clause regarding frequent start-stop regulations," and a response completion suggestion, "Add a link to the original document related to this issue." Based on the user's low-rating behavior and the low interaction value of the predicted curve modality, it forms a modality weight correction suggestion, "Lower the priority of the predicted curve and increase the display weight of the document link." These correction parameters, along with the user's access identification information (such as timestamp, permission level, and access point), are bound and encoded into a structured feedback sample. A digest value is calculated using the SHA-256 algorithm and written to the blockchain's notarization node. The next time the system encounters a similar issue and identifies the user, it will automatically prioritize presenting structured clause content and highlighted document links according to the updated modality configuration strategy, instead of displaying the predicted curve, significantly improving the accuracy of the Q&A response and user satisfaction. This process not only achieves personalized optimization but also forms a complete and auditable response evolution chain.

[0323] Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of this application. Therefore, the scope of protection of this application should be determined by the scope defined in the claims of this application.

Claims

1. An intelligent question-and-answer system for industrial equipment maintenance based on multi-agent collaboration, characterized in that, include: An input unit is used to receive multimodal user input information and parse the multimodal user input information into structured question information, wherein the multimodal user input information includes text input, voice input, image input, or device scanning input; The scheduling unit is used to perform semantic understanding and question classification on the structured question information based on a multilingual intent recognition model and an incremental training mechanism to obtain question classification results; wherein, the multilingual intent recognition model is a fine-tuned cross-lingual pre-trained language model; The processing unit is used to invoke the domain agent corresponding to the question classification result to generate intelligent question-answering processing results. The domain agent includes an equipment maintenance agent, a management consulting agent, a digital operation and maintenance agent, or a technical service agent. The intelligent question-answering processing results include predictive maintenance suggestions, structured response content, and semantic annotation information. The fusion unit is used to perform multi-hop semantic reasoning based on the intelligent question-answering processing results and by fusing local knowledge base information, graph database information and online retrieval information to obtain multimodal response information, which includes text descriptions, image screenshots, prediction curves and external recommendation links. Specifically, the scheduling unit is used for: The system receives structured question information output by the input unit and extracts the text input field, speech recognition transcription field, and image recognition label field to form a multimodal semantic label sequence. The multimodal semantic label sequence is concatenated into a unified input vector according to a preset encoding rule and input into the multilingual intent recognition model. The multilingual intent recognition model is a fine-tuned cross-lingual pre-trained language model used to generate an embedding vector containing language-independent semantic features. Based on the semantic content represented by the embedded vector, a semantic consistency discriminant network is used to evaluate the stability of the language mixing features of the input information and output a language consistency score. Based on the language consistency score and user feedback, the classification head parameters in the multilingual intent recognition model are updated only through a streaming incremental fine-tuning mechanism, while keeping the underlying encoder parameters frozen. The problem classification result is then output based on the fine-tuned model. The fusion unit is specifically used for: Based on the fault label information contained in the intelligent question answering processing results, a semantic path expansion operation with a limited number of hops is performed in the graph database, starting from the knowledge graph node corresponding to the fault label. The semantic path expansion operation includes filtering according to the edge type, path weight and semantic consistency of the intersection node defined in the graph, and constructing multiple inference paths with fault semantic association. The knowledge fragments corresponding to the nodes involved in each reasoning path are used as candidate answer sources. The original content, citation frequency, update timeliness and semantic relevance of the corresponding knowledge fragments in the graph database, local knowledge base and online search information are obtained respectively, and the source confidence score of each source is calculated. Based on the source confidence scores, the candidate knowledge fragments from multiple sources are reorganized and deredundant according to a preset dynamic fusion strategy, and a multimodal response information draft with a unified structure is output. The multimodal response information draft includes text descriptions, image fragments and prediction curves that are weighted by confidence scores. Based on the semantic anchor points described in the text, the optical character recognition module is invoked to perform paragraph-level text localization and structure recognition on the original document image, and a highlighted border is generated in the image segment to mark the position of the semantic anchor points in the original text, thereby generating multimodal response information with visual cues and semantic correspondence.

2. The intelligent question-and-answer system for industrial equipment maintenance based on multi-agent collaboration as described in claim 1, characterized in that, Also includes: The control unit is used to receive the multimodal response information and user feedback information, update the multilingual intent recognition model and knowledge weight configuration based on the user feedback information, and perform permission verification and tamper-proof storage processing on the multimodal response information using a role-based access control mechanism and a blockchain-based operation log mechanism.

3. The intelligent question-and-answer system for industrial equipment maintenance based on multi-agent collaboration as described in claim 1, characterized in that, The equipment maintenance intelligent agent is specifically used for: Receive the problem classification result and the device identification field, fault description field and image recognition label field in the structured problem information, and construct a structured input sample for predictive modeling; The structured input samples are input into a preset fault mode and health management model. The fault mode and health management model generates a prediction curve based on a time series prediction model and maps the current device status to labels based on the fault label system defined in the knowledge graph. Based on the generated prediction curves and fault labels, the maintenance priority score of each equipment component is jointly calculated. The maintenance priority score is a priority value obtained by fusing the confidence function output and the remaining life estimation result according to a preset weighting coefficient. Based on the maintenance priority score and the hierarchical relationship between device structure nodes in the knowledge graph, an intelligent question-answering processing result is generated. The intelligent question-answering processing result includes predictive maintenance suggestions, structured response content, and semantic annotation information.

4. The intelligent question-and-answer system for industrial equipment maintenance based on multi-agent collaboration as described in claim 1, characterized in that, The fusion unit is also used for: Based on the question type field in the question classification results, a preset modal configuration rule table is invoked to dynamically determine the combination method of each modal element in the multimodal response information draft. The combination method includes text description, prediction curve, image fragment, and document jump link. When the question type field is characterized as "equipment failure", the combination of prediction curve and image fragment is given priority; when it is characterized as "procedure consultation", the combination of text description and document jump link is given priority. After generating the multimodal response information draft, a unique traceable tag is attached to each modal element. The traceable tag points to the original knowledge fragment source node and records the node number sequence, edge weight cumulative value and path semantic similarity score of the node participating in the reasoning path in the graph database to support subsequent path backtracking and result verification. When the multimodal response information draft is displayed and sorted in a structured manner, a reinforcement learning sorting model based on user interaction feedback is introduced. The reinforcement learning sorting model receives the user's click behavior, dwell time and feedback rating for different modal combinations in historical interactions as reward signals, and dynamically updates the display order weight parameters to form a user-personalized multimodal display configuration. In the final multimodal response information, the traceable tag and original path annotation mechanism are retained, and expandable links are provided to jump to the original knowledge fragments, graph paths and source documents.

5. The intelligent question-and-answer system for industrial equipment maintenance based on multi-agent collaboration as described in claim 2, characterized in that, The control unit is specifically used for: The system receives semantic difference values ​​between the task expression in the user feedback information and the multimodal response information, interface interaction behavior data, and access user identification information. The semantic difference values ​​are obtained by calculating the vector cosine similarity between the user's natural language input and the system's semantic parsing results in a unified embedding space. The interface interaction behavior data includes the number of clicks, dwell time, and feedback rating of each modal element in the multimodal response information. The access user identification information includes an access timestamp, access permission level, and access path identification code. Based on the semantic difference value and the interface interaction behavior data, modal elements with semantic deviations in the response content are extracted, and a set of correction parameters is generated, which includes language adjustment prompts, response completion suggestions and modal weight correction suggestions. The set of correction parameters records the key expression offset points between the original response content and the user's actual needs, forming feedback samples that can be used to improve the response generation process. The modality composition strategy of the multimodal response information is reconstructed using the modified parameter set, specifically including: optimizing the text description expression according to the language adjustment prompts; adding missing image fragments, prediction curves, or document jump links according to the response completion suggestions; updating the modality priority sorting in the modality configuration rule table according to the modality weight correction suggestions to dynamically adjust the generation structure of the multimodal response information in subsequent questions of the same type; establishing a binding relationship between the access user identification information and each feedback record in the modified parameter set, and generating a unique digest value using a hash algorithm and timestamp encryption method; the digest value, together with the multimodal response information, the semantic difference value, and the interface interaction behavior data, is written into the on-chain evidence storage structure and broadcast to the blockchain node network to form an audit entry with tamper-proof characteristics, supporting the source tracing, responsibility determination, and evolution verification of any of the multimodal response information.

Images