Industrial equipment maintenance intelligent question-answering system based on multi-agent cooperation

Abstract

The invention relates to an industrial equipment maintenance intelligent question-answering system based on multi-agent collaboration. Wherein the input unit is used for receiving text, voice, image or equipment scanning and other multi-mode user input information and analyzing the information into structured problem information; the scheduling unit performs semantic understanding and problem classification on the structured problem information based on the fine-tuned cross-language pre-training language model and an incremental training mechanism; the processing unit calls a corresponding domain agent according to the classification result, and generates an intelligent question and answer processing result including predictive maintenance suggestions, structured reply content and semantic annotation information; and the fusion unit fuses the local knowledge base, the graph database and the networking retrieval information, performs multi-hop semantic reasoning on the intelligent question and answer processing result, and generates multi-modal reply information including text description, image screenshots, prediction curves and recommendation links. The system can support multi-language and multi-mode intelligent question answering and predictive maintenance in a complex industrial maintenance scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to an industrial equipment maintenance intelligent question and answer system based on multi-agent collaboration. BACKGROUND

[0002] In the prior art, the maintenance and management of industrial equipment mainly rely on traditional fault repair systems or question and answer platforms based on static knowledge bases. Such systems usually use rule matching or keyword retrieval to respond to user questions, although they can provide some maintenance guidance, but in the face of language diversification, problem complexification and document structure diversification in actual application scenarios, the processing capacity and response accuracy are greatly limited. In recent years, with the development of artificial intelligence technology, some systems have begun to introduce natural language processing and knowledge graph technology to improve the question and answer effect, but overall, they still mainly rely on single models, static knowledge structures and fixed interaction forms, and have not yet realized the effective collaboration of complex semantic understanding and reasoning capabilities.

[0003] The prior art generally has the following problems: first, it lacks support for multi-modal input and cannot effectively process image, voice and other non-text information; second, the classification and semantic recognition model cannot adapt to multi-language input and is difficult to continuously optimize performance through online learning; third, the question and answer result form is single and cannot provide structured, visual and predictive maintenance information; fourth, the knowledge source fusion capability is weak and it is difficult to realize joint reasoning and dynamic completion of local knowledge, graph and external information.

[0004] Therefore, there is an urgent need for a technical solution that can adapt to the complex interaction requirements of industrial sites, support multi-language and multi-modal input, and have semantic reasoning, knowledge fusion and intelligent question and answer collaborative processing capabilities. SUMMARY

[0005] The present application provides an industrial equipment maintenance intelligent question and answer system based on multi-agent collaboration, which improves the accuracy, real-time performance and multi-modal response capability of intelligent question and answer in the industrial equipment maintenance scenario.

[0006] The present application provides an industrial equipment maintenance intelligent question and answer system based on multi-agent collaboration, comprising: An input unit for receiving multi-modal user input information and parsing the multi-modal user input information into structured question information, the multi-modal user input information including text input, voice input, image input or device scanning input; A scheduling unit for performing semantic understanding and question classification on the structured question information based on a multi-language intent recognition model and an incremental training mechanism to obtain a question classification result; wherein the multi-language intent recognition model is a fine-tuned cross-language pre-training language model; The processing unit is configured to invoke a domain intelligent agent corresponding to the question classification result to generate an intelligent question and answer processing result, the domain intelligent agent including a device maintenance intelligent agent, a management consultation intelligent agent, a digital operation and maintenance intelligent agent, or a technical service intelligent agent; and the intelligent question and answer processing result includes a predictive maintenance suggestion, structured reply content, and semantic labeling information. The fusion unit is configured to perform multi-hop semantic reasoning based on the intelligent question and answer processing result, and fuse local knowledge base information, graph database information, and network search information, to obtain multi-modal reply information including text description, image screenshot, prediction curve, and external recommendation link.

[0007] The beneficial effects of the present application mainly include: (1) supporting multi-modal input and multi-language understanding, capable of processing various forms of user input such as text, voice, image, and device scanning, and achieving accurate classification of mixed language questions through a cross-language pre-training model, significantly improving the adaptability of the system in multi-lingual and multi-scene environments. (2) Realizing the cooperative scheduling of question classification and domain intelligent agents, dynamically invoking special intelligent agents in corresponding domains according to the classification result, thereby realizing more targeted knowledge processing and fault response, and improving the accuracy and professionalism of industrial question and answer. (3) Fusing local knowledge, graph database, and network search information, constructing a multi-hop semantic reasoning path, and outputting multi-modal reply results including text description, image screenshot, and predictive curve, effectively improving the visual expression ability and predictive decision support ability of the reply. BRIEF DESCRIPTION OF DRAWINGS

[0008] Figure 1 is a schematic diagram of an industrial device maintenance intelligent question and answer system based on multi-agent cooperation provided by the first embodiment of the present application. DETAILED DESCRIPTION

[0009] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application. However, the present application can be practiced in a variety of ways beyond those described herein without departing from the scope of the present application, and those skilled in the art can make similar modifications without departing from the spirit of the present application, so the present application is not limited to the specific implementations disclosed below.

[0010] The first embodiment of the present application provides an industrial device maintenance intelligent question and answer system based on multi-agent cooperation. Please refer to Figure 1 , which is a schematic diagram of the first embodiment of the present application. The following will be described in detail Figure 1 The first embodiment of the present application provides an industrial device maintenance intelligent question and answer system based on multi-agent cooperation.

[0011] The industrial device maintenance intelligent question and answer system based on multi-agent cooperation includes an input unit 101, a scheduling unit 102, a processing unit 103, and a fusion unit 104.

[0012] The input unit 101 is configured to receive multi-modal user input information including text input, voice input, image input or device scanning input, and parse the multi-modal user input information into structured problem information.

[0013] The main function of the input unit 101 is to receive multi-modal input information submitted by users through different interaction methods, and accurately convert the information into structured problem information, providing a standardized data basis for subsequent semantic understanding and problem classification. The input unit 101 includes an information collection component, a modal recognition component, a unified coding component and a structured conversion component, which work together to ensure the consistency of multi-modal input at the semantic level and the standardization of technical processing.

[0014] The information collection component is responsible for receiving raw input data transmitted by the user end, which can come from human-machine interfaces such as touch panels, mobile terminals, industrial terminals, voice assistants or image acquisition devices. Text input is usually input by keyboard or extracted by OCR, voice input is collected by microphone and transcribed into text by speech recognition engine in real time, image input includes device failure photos, interface screenshots, abnormal indicator light patterns, etc., which are obtained through camera or file upload, and device scanning input obtains machine labels, part numbers or maintenance record page codes, etc. through AR devices, two-dimensional code scanning guns, barcode recognition modules and other devices.

[0015] After receiving multi-source input, the modal recognition component first performs modal recognition and signal separation on the data to ensure that various modal data can be routed to the corresponding preprocessing path. For example, voice data will be sent to the ASR model (Automatic Speech Recognition), and image data will be handed over to the image preprocessor for edge enhancement, noise reduction, image segmentation, etc. to extract key areas. Text data is checked for grammar and converted to a standard encoding to ensure Unicode consistency under different language input conditions.

[0016] The unified coding component converts the above-mentioned modal inputs into a unified format of intermediate representation. Specifically, for text and voice transcription results, Tokenization is used to generate structured Token sequences with timestamps, syntax features, keyword density and other semantic tags. Image data is extracted after recognition to extract image recognition tags (such as component type, state abnormality description, etc.) and described in JSON format. Device scanning data is converted into indexable metadata information including device ID, component serial number, production line and maintenance timestamp fields through device coding parsing algorithm.

[0017] The structured conversion component fuses the multi-modal input into a structured question information object according to the unified format, which at least contains a question category preliminary judgment field, a language type field, a key equipment field, a visualization reference field (such as an image URL or screenshot coordinates), and an original input content field. The object is transmitted to the dispatch unit 102 through a pre-defined interface and uses a general protocol (such as gRPC or RESTful API) to call structured question data to achieve efficient and low-latency data interaction.

[0018] The input unit 101 also includes an exception detection and fault tolerance processing mechanism for identifying format errors, unsupported modalities, or low-confidence input situations. When detecting input exceptions, the unit can guide re-input through a user feedback interface or provide modality conversion suggestions (such as suggesting users to take clear images or use voice input). In addition, the input unit 101 supports a user identification information binding mechanism for associating input information with user identity, permission level, and historical maintenance records to ensure context consistency in subsequent processing.

[0019] Through the above mechanisms, the input unit 101 not only accurately receives and parses complex multi-modal user input in industrial field environments, but also outputs question data in a structured and standardized manner, ensuring that the entire question and answer system has good readability, explainability, and technical stability in the semantic processing stage, thereby laying a solid foundation for achieving high-quality intelligent question and answer results.

[0020] The dispatch unit 102 is configured to perform semantic understanding and question classification on the structured question information based on a multi-language intent recognition model and an incremental training mechanism to obtain a question classification result. The multi-language intent recognition model is a fine-tuned cross-language pre-trained language model.

[0021] Further, the dispatch unit is specifically configured to: receive the structured question information output by the input unit and extract the text input field, speech recognition transcription field, and image recognition label field to form a multi-modal semantic label sequence; concatenate the multi-modal semantic label sequence into a unified input vector according to a pre-set encoding rule and input it into the multi-language intent recognition model, which is a fine-tuned cross-language pre-trained language model for generating an embedding vector containing language-independent semantic features; based on the semantic content represented by the embedding vector, using a semantic consistency discrimination network to evaluate the stability of the language mixing features of the input information and output a language consistency score; According to the language consistency score and user feedback information, only the classification head parameters in the multi-language intent recognition model are updated through a streaming incremental fine-tuning mechanism, the underlying encoder parameters are kept in a frozen state, and the problem classification result is output according to the fine-tuned model.

[0022] The scheduling unit first receives structured question information output by the input unit, which includes at least three types of fields: a text input field, a speech recognition transcription field, and an image recognition label field. The text input field is an industrial maintenance related description directly entered by the user through the terminal interface or text imported, the speech recognition transcription field is a standardized text obtained by transcribing speech input by a speech recognition module, and the image recognition label field extracts semantic labels such as "abnormal indicator light", "pump shell crack", and "corrosion trace" by identifying scanned images of equipment through image classification and object detection models.

[0023] To construct an input representation with semantic consistency, the scheduling unit designs a multi-modal label encoding method based on Token expansion and sequence interpolation. Specifically, the text input field is tokenized to construct a basic semantic Token sequence, and key word Token fragments extracted from the speech recognition transcription field are inserted into the sequence according to the context semantic position, with a specific delimiter (e.g. [#AUDIO]) used for boundary marking; the labels extracted from the image recognition label field are not directly used as natural language embeddings, but are mapped to standardized description sentence fragments through a lookup table, such as "image recognition: control valve surface anomaly", and are added to the end of the sequence in embedded form. In addition, to avoid input vector length exceeding the model input limit, the system uses a key fragment retention mechanism based on TF-IDF weight calculation to dynamically trim semantic redundant content, retaining only semantic units with high weight scores.

[0024] The multi-modal semantic label sequence after splicing is encoded into a unified input vector and input into the multi-language intent recognition model, which is a cross-language model based on XLM-R or mBERT and fine-tuned on local industrial corpus. The model uses a Transformer structure, with the underlying encoder parameters kept in a frozen state during runtime, allowing only the classification head above it to be trained and updated for task-specific training. The model performs context modeling on the input vector and outputs a language-independent high-dimensional semantic embedding vector.

[0025] To solve the problems of syntactic structure mismatch and semantic reconstruction ambiguity in the mixed language input scene, a semantic consistency discrimination network is embedded in the scheduling unit. The network consists of two parts: first, a double-branch encoder of input embedding vectors, each branch corresponds to a language channel (such as Simplified Chinese and English), shares weight parameters but introduces language marker vectors in the structure to realize semantic domain differentiation modeling; then a cross-channel comparison discrimination module is adopted, using the contrast learning mechanism, using the Triplet Loss function during training, constructing training samples by sampling language consistent pairs and language inconsistent pairs, so that it has cross-language semantic judgment ability. In the reasoning stage, the semantic consistency discrimination network performs cross-semantic space projection on the current input embedding vector and outputs a language consistency score. The score is a real number between 0 and 1, and the closer to 1 indicates that the semantic expression between different modalities and languages is more consistent.

[0026] After obtaining the language consistency score, the system uses the score and user feedback information to update the model classification ability. User feedback information includes user satisfaction labels for reply content, error correction annotations, or selection behavior for recommended content. The system uses a streaming incremental fine-tuning mechanism based on the River framework to gradually update the classification head of the multi-language intent recognition model. In specific implementation, the system first organizes the embedding vector and feedback result into sample pairs and caches them in a local memory pool. When the memory pool capacity exceeds the set threshold, the fine-tuning mechanism is triggered. The fine-tuning operation is limited to the fully connected weight and bias parameters of the classification head layer, without affecting the self-attention parameters of the underlying Transformer, ensuring the stability and generalization ability of the system during long-term operation. After the model fine-tuning is completed, the updated question classification results are generated immediately and used as the index basis for subsequent processing unit agent calls.

[0027] Through the above design, the scheduling unit not only realizes the fusion and unified encoding of multi-modal input, but also forms a closed-loop linkage in cross-language semantic understanding and system adaptive evolution ability. The entire path has a clear dependency and call relationship between input, encoding, discrimination, feedback, and fine-tuning update. The construction and call process of key modules are far away from the existing regular classifier call method or static model execution path, providing a highly reliable solution for mixed language and multi-modal question classification in industrial scenarios.

[0028] The processing unit 103 is configured to call a domain agent corresponding to the question classification result to generate an intelligent question and answer processing result. The domain agent includes a device maintenance agent, a management consultation agent, a digital operation and maintenance agent, or a technical service agent. The intelligent question and answer processing result includes a predictive maintenance suggestion, structured reply content, and semantic annotation information.

[0029] The function of the processing unit 103 is to accurately select and call the corresponding domain agent according to the question classification result output by the scheduling unit 102, and generate structured, predictable and explainable intelligent question and answer processing results. The processing result not only includes the answer to the question content, but also covers predictive maintenance recommendations, structured reply content and semantic annotation information related to the source information, thereby providing complete and usable knowledge output for the subsequent fusion unit.

[0030] In the specific implementation process, after receiving the question classification result output by the scheduling unit, the processing unit 103 will select the most suitable problem processing entity from the four domain agents according to the preset category label and agent call mapping table. The four types of agents are respectively for different industrial maintenance application scenarios. The equipment maintenance agent is usually responsible for hardware fault diagnosis, repair scheme recommendation and fault mode identification, which will call device operation logs, sensor time series data and standard operation procedure documents for joint analysis. The management consulting agent mainly handles management type problems such as equipment use system, maintenance cycle planning, supply chain policy, and extracts rule logic from enterprise internal control manual, ISO document and historical case library for answering. The digital operation and maintenance agent focuses on SCADA data analysis, alarm record interpretation and automatic control logic backtracking processing in the industrial Internet of Things (IIoT) scene, and often uses time series modeling and state recognition model to assist in judgment. The technical service agent focuses on spare parts traceability, maintenance service provider information and related compliance record retrieval, and can link blockchain audit nodes and spare parts supply graph database to improve service transparency.

[0031] The implementation of each domain agent is based on an integrated framework of "domain knowledge graph + question and answer generation model + context adaptation mechanism", which combines task characteristics to build knowledge acquisition logic, model structure and call interface. The equipment maintenance agent is realized by integrating the PHM (Prognostics and Health Management) platform and the rule-driven reasoning engine, which includes an industrial sensor data processing module, a fault mode identification model (such as LSTM or variational autoencoder for time series anomaly detection), a device structure graph embedding module and a repair scheme recommendation library. The agent can extract features from device parameters, fault phenomena and other fields in structured question information, and generate targeted diagnostic recommendations and preventive maintenance time window predictions in combination with the device-fault-operation ternary relationship in the prior knowledge graph.

[0032] The management consulting agent is implemented based on a document reasoning engine and a policy rule matching mechanism. The agent integrates an enterprise regulation and system graph, process management files, and a semantic index structure of historical approval cases. The agent supports the use of a rule tree or a BERT-based question and answer matching model to generate answers, and can handle problems such as "whether the equipment maintenance plan meets the standard" or "the alignment of maintenance frequency with enterprise indicators". The agent also supports historical decision path reconstruction, which is used to explain the logical process of the enterprise management layer when handling similar problems.

[0033] The digital operation and maintenance agent is constructed using a multi-modal sensor data fusion and SCADA interface analysis module. The core of the agent is a set of semantic-driven industrial time series data analysis models, including periodic feature extraction, alarm event clustering, and state transition modeling components. The agent also integrates structured data (such as data tables and time series logs) and unstructured documents (such as technical manuals) to achieve multi-source information consistency response. The agent can handle problems such as "whether the current fluctuation anomaly is related to historical equipment downtime", and can return fault trend prediction graphs and recommended measures.

[0034] The technical service agent is constructed based on a supply chain graph, a blockchain storage system, and a contract knowledge extraction model. The agent can trace the supply chain of spare parts and manage the supply chain. The agent has entity analysis, correlation path matching, and service provider qualification verification capabilities. The agent generates service recommendations and procurement suggestions based on multi-hop reasoning, and outputs structured responses with audit records. Each agent is registered to the processing unit 103 through a unified interface protocol, and supports asynchronous response mechanism and confidence score feedback, ensuring good scalability and stability of the system.

[0035] After the agent is called, the processing unit sends structured problem information and related semantic context information to the agent interface. The agent performs reasoning based on a rule base, a machine learning model, or a graph neural network. Based on the knowledge graph and the document semantic index, the agent outputs multiple structured fields as the answer content. These fields generally include: question response text, predictive maintenance recommendations (such as the failure probability of a component in X hours or the optimal maintenance window), semantic annotation positions (such as coordinates in the document or path nodes in the knowledge graph), and confidence scores and response timestamps.

[0036] The processing unit also includes a response standardization module that unifies the formats of data returned by different agents to meet the input specifications of the fusion unit. The module labels the text content by paragraph and hierarchical tag, and attaches OCR results and original file links to image-related content. Predictive data is formatted as a graphable series or a standard JSON structure. To ensure information integrity, the processing unit also performs a verification process to check if the answer is missing key fields or the model output is abnormal, and triggers a backup agent or returns a prompt message according to system policies.

[0037] In addition, in order to support the self-learning improvement of the response quality of the agent, the processing unit generates a call log after completing each response, which records the input information, the called agent identifier, the output result summary and the confidence score. The log will be passed to the feedback learning module of the system to provide a source of supervision data for the behavior optimization of the agent.

[0038] In summary, the processing unit 103 is not only responsible for implementing accurate scheduling and agent calling based on the problem classification results, but also ensures that various reply contents have completeness, structured and predictability, thereby providing a high-quality knowledge output interface for the entire intelligent question and answer system, and supporting the smooth development of subsequent multi-modal fusion and semantic reasoning process.

[0039] Further, the device maintenance agent is specifically used for: receiving the device identifier field, the fault description field and the image recognition tag field in the problem classification result and the structured problem information, and constructing a structured input sample for prediction modeling; inputting the structured input sample into a preset fault mode and health management model, the fault mode and health management model generating a prediction curve based on a time series prediction model, and mapping the current device state based on a fault label system defined in a knowledge graph; based on the generated prediction curve and fault label, jointly calculating the maintenance priority score of each device component, the maintenance priority score being a priority value obtained by fusing the confidence function output and the remaining life estimation result according to a preset weighting coefficient; According to the maintenance priority score, combined with the hierarchical relationship between the device structure nodes in the knowledge graph, an intelligent question and answer processing result is generated, which includes predictive maintenance suggestions, structured reply content and semantic annotation information.

[0040] The processing flow of the device maintenance agent starts with the classification result output by the receiving and scheduling unit, which is usually a "device failure class" label indicating that the current problem belongs to the category of device operation abnormality, performance decline or potential failure risk. The agent simultaneously obtains three key fields from the structured problem information output by the input unit, namely the device identification field, the fault description field and the image recognition label field. The device identification field includes device ID, component number, installation location code and other basic information; the fault description field is the core sentence element extracted from the user input text or voice transcription, usually involving performance symptoms or alarm signals; the image recognition label field is automatically extracted from the device image after processing by a deep learning model such as YOLOv5 or EfficientDet, such as "pump shaft skew", "filter carbon deposition", "surface corrosion" and other standardized descriptions mapped by trained labels.

[0041] The above three fields are gathered into a structured input sample construction module, which uses field standardization and timestamp alignment mechanisms to unify and align data fields from multiple modalities and organize them into data objects for time series prediction modeling. The construction process not only involves field splicing, but also includes semantic mapping table application, fault dictionary calibration, image label confidence screening, and internally generates structured JSON input packages such as {device ID, component location, time index, fault type code, image anomaly intensity score}. The input package is sent to the pre-set fault mode and health management model, which is a combined structure containing a device state time series prediction sub-model based on the Transformer structure and a label mapping sub-module for structured event labels.

[0042] The time series prediction sub-model takes component operating parameters such as vibration frequency, current fluctuation and temperature gradient as sequence input, calculates the evolution trend of historical patterns within the current time window through multi-head attention mechanism, and outputs the prediction curve. The prediction curve is a time-state pair sequence, formatted as {T+1: normal, T+2: abnormal probability 0.45, T+3: abnormal probability 0.72}. In parallel, the label mapping sub-module encodes the image recognition label and fault description keywords in the input sample, and performs semantic alignment and similarity matching with the pre-defined fault label system in the knowledge graph, outputting one or more most relevant fault labels, such as "bearing wear", "valve hysteresis", "seal leakage", etc.

[0043] The device maintenance agent obtains the above two outputs, and performs priority score calculation. The scoring logic is an important part of the present application, and uses a confidence function and a remaining life estimate as the main input source. The confidence function is driven by the maximum abnormal probability in the prediction curve, and is combined with the abnormal score of the current state of the image recognition model to perform normalization correction. For example, if the prediction curve has an abnormal probability of 0.72 at time T+2 in the future, and the abnormal score of the same component in image recognition is 0.65, then the confidence function takes the weighted average (such as 0.7). The remaining life estimate is obtained by another deep regression network prediction, which outputs the estimated remaining working hours of the current state of the component. The two are fused with a preset weighting coefficient (such as a confidence weight of 0.6 and an RUL weight of 0.4), and finally output a maintenance priority score between 0 and 1. The higher the score, the higher the maintenance urgency.

[0044] This score is used as the basic input of the subsequent knowledge reasoning module, and the device structure relationship in the knowledge graph is used for structure level weighted judgment. The device structure graph is organized in the form of a graph database, with nodes representing components and edges representing physical connection relationships and functional dependencies. For example, there is a "functional dependency" edge between the "bearing" node and the "spindle" node. If the "bearing" node score is higher than the threshold, the "spindle" node will also be assigned a secondary reasoning warning label. This structure level logic can be executed based on graph convolution network (GCN) or path activation algorithm.

[0045] Finally, the device maintenance agent generates intelligent question and answer processing results based on the score results and graph propagation relationships, including: a set of predictive maintenance recommendations, including recommended replacement timing and recommended operation scheme; a set of structured reply content, including score details table, failure label group, and graph reasoning chain; and a set of semantic annotation information indicating the corresponding location or screenshot coordinate information of the suggestion content in the knowledge document. All outputs are structured and returned to the fusion unit for subsequent reply generation.

[0046] The above path not only has execution details that have not been described in the prior art in terms of data structure organization, model parallel design, score generation logic, and structure graph driving, but also realizes robust maintenance suggestion generation in a high-noise, multi-modal input environment through explicit weighting of control parameters and coupling of confidence scores, providing an implementable, interpretable, and updatable technical solution for device maintenance scenarios in industrial intelligent question and answer systems.

[0047] The following is a reference implementation code for the device maintenance agent: import json import numpy as np # Structured input field example input_data = { "question_class": "Equipment Failure Class", "device_id": "PUMP_23A", "component_id": "BEARING", "location": "Line_2_Section_5", "description": "Running with periodic abnormal noise, alarm raised", "image_tags": [{"label": "Bearing Crack", "confidence": 0.78}, {"label": "Excessive Oil", "confidence": 0.62}], "time_series_data": { "vibration": [0.3, 0.35, 0.42, 0.6, 0.7], "current": [2.1, 2.4, 2.5, 2.9, 3.2], "temperature": [65, 66, 68, 71, 74] } } # Structured Sample Building Module def build_structured_sample(input_data): # Build model-oriented standard input structure, including component identification, fault semantic label, and image anomaly confidence. structured_sample = { "device_id": input_data["device_id"], "component_id": input_data["component_id"], "location": input_data["location"], "fault_description": input_data["description"], "fault_keywords": extract_fault_keywords(input_data["description"]), "image_tags": filter_top_image_tags(input_data["image_tags"]), "time_series": input_data["time_series_data"] } return structured_sample def extract_fault_keywords(text): # Extract keywords using a keyword dictionary or language model keywords = [] If "abnormal noise" appears in the text: keywords.append("bearing wear") If "alarm" is in the text: keywords.append("Running offset") return keywords def filter_top_image_tags(tags, threshold=0.7): # Filter image labels and retain those with higher confidence levels return [tag for tag in tags if tag["confidence"] >= threshold] # Fault Prediction Model and Health Management System (PHM) Main Interface def predict_with_phm_model(sample): #Process input samples and generate prediction curves and fault label sets # Implementation details: Generate a prediction curve by calling a preset time series model. prediction_curve = { "T+1": {"status": "normal", "prob": 0.1}, "T+2": {"status": "warning", "prob": 0.45}, "T+3": {"status": "fail_likely", "prob": 0.72} } # Simulated Tag Matching: Semantic Alignment + Tag Mapping fault_labels = list(set(sample["fault_keywords"] + [tag["label"]for tag in sample["image_tags"]])) return prediction_curve, fault_labels # Remaining lifetime estimation model (simplified) def estimate_rul(sample): #Estimate the remaining lifespan (in hours) of a component using a regression network or rule base. #Implementation: Predicting based on vibration trends vibration = np.array(sample["time_series"]["vibration"]) if vibration[-1] - vibration[0] > 0.4: return 40 # High-risk equipment with a short estimated lifespan return 120 # Estimated lifespan is relatively long # Confidence function + remaining lifetime fusion -> maintenance priority scoring def calculate_priority_score(pred_curve, image_tags, rul, alpha=0.6,beta=0.4): #Calculate maintenance priority score, fuse prediction curve with image anomaly confidence and remaining lifetime pred_probs = [v["prob"] for v in pred_curve.values()] pred_conf = max(pred_probs) # Average confidence level of image anomalies (filtered) img_conf = np.mean([t["confidence"] for t in image_tags]) ifimage_tags else 0.5 combined_conf = (pred_conf + img_conf) / 2 # Overall confidence level normalized_rul_score = 1 - min(rul / 200.0, 1.0) # Remaining life converted to risk score, less is more priority_score = alpha * combined_conf + beta * normalized_rul_score return round(priority_score, 3) # Knowledge graph structure propagation def propagate_score_through_graph(component_score, component_id): # Propagate priority score through device structure graph, generate inference chain graph = { "BEARING": ["SHAFT"], "SHAFT": ["GEAR"], "GEAR": [] } propagation = {} propagation[component_id] = component_score queue = [(component_id, component_score)] while queue: current, score = queue.pop(0) for neighbor in graph.get(current, []): propagated_score = round(score * 0.6, 3) # Decaying propagation propagation[neighbor] = propagated_score queue.append((neighbor, propagated_score)) return propagation # Output intelligent question and answer processing result def generate_response(prediction_curve, fault_labels, priority_score,propagation_chain): # Construct the response structure, including predicted suggestions, structural scores, and semantically labeled locations. result = { "maintenance_advice": f"It is recommended to check / replace the {fault_labels[0]} component within 72 hours, with a priority score of {priority_score}", "structured_output": { "score_detail": { "component_scores": propagation_chain, "top_component": max(propagation_chain, key=propagation_chain.get) }, "fault_tags": fault_labels }, "semantic_annotations": { "source_text_mapping": "Location of Figure 3 on page 12 of the document 'Maintenance Manual 2023 Edition'", "image_highlight_coords": [120, 300, 180, 360] } } return result # Main execution process simulation sample = build_structured_sample(input_data) prediction_curve, fault_labels = predict_with_phm_model(sample) rul = estimate_rul(sample) priority_score = calculate_priority_score(prediction_curve, sample["image_tags"], rul) propagation_chain = propagate_score_through_graph(priority_score,sample["component_id"]) response = generate_response(prediction_curve, fault_labels,priority_score, propagation_chain) # Print the final generated intelligent question-and-answer results (structured output) print(json.dumps(response, indent=2, ensure_ascii=False)) The fusion unit 104 is used to perform multi-hop semantic reasoning based on the intelligent question-answering processing results and by fusing local knowledge base information, graph database information and network retrieval information to obtain multimodal response information, which includes text descriptions, image screenshots, prediction curves and external recommendation links.

[0048] The main function of the fusion unit 104 is to further integrate multi-source heterogeneous knowledge resources based on the intelligent question-answering results generated by the processing unit 103, and generate more valuable multimodal response information through semantic reasoning. Its function is not limited to information integration, but also constructs cross-domain knowledge linkage through graph relationship modeling, context alignment and multi-hop semantic path search, so as to realize the structured, visualized and predictive enhancement of the response.

[0049] In actual deployment, the fusion unit first receives the intelligent question and answer processing results from the processing unit, which contain predictive maintenance recommendations, structured reply content, and semantic labeling information. The fusion unit identifies key entities, fault labels, component numbers, or semantic anchors in the reply content and uses them as the starting point for semantic fusion. The system's built-in local knowledge base typically includes maintenance manuals, device structure diagrams, enterprise operation records, and other document information. Full-text indexing (such as Elasticsearch) combined with semantic embedding models (such as Sentence-BERT) is used for knowledge matching. The graph database part is based on Neo4j or TigerGraph, storing structured triple relationships such as multi-level topological relationships between devices and faults, component dependency paths, and maintenance operation mappings. Networked search information includes calling Google API, industrial forum crawlers, standard library interfaces, and other external data sources to supplement information when local knowledge is missing or confidence is low.

[0050] The fusion unit maps these different sources of knowledge into a semantic space and performs multi-hop path search centered on the current problem semantics. For example, when the intelligent agent returns a predictive maintenance recommendation of "pump shaft vibration anomaly," the fusion unit identifies the system path where the pump shaft is located through the graph, and further searches for associated historical fault cases, maintenance frequencies of adjacent components, and alternative component inventory status information. With the support of the knowledge graph, the system can perform 2-hop or 3-hop path expansion, such as "pump shaft → pump body → flow anomaly → sensor drift," to discover potential hidden causes or cross-evidence data.

[0051] The fusion results are not limited to structured text, but also include image and chart content. The system can automatically call document screenshot tools or SCADA interfaces to embed key reference paragraphs or historical fluctuation curves in image form into the reply results. Combined with OCR text positioning and highlighting technology, users can visually see the labeling of key information on the image. In addition, predictive results such as time series analysis curves will be inserted into the reply in standard SVG or Base64 format, accompanied by text explanations; if external links are recommended service platforms, standard specification pages, or maintenance service provider catalogs, clickable external recommendation links will be automatically generated, and link call logs will be recorded in the background for subsequent optimization.

[0052] To ensure the reliability and relevance of the answer, the fusion unit is also provided with a confidence fusion mechanism. Each type of information source (local knowledge base, graph database, network search) is provided with a corresponding weight parameter, and is dynamically scored according to factors such as timeliness, reference frequency, and structural integrity of the source data. The final answer content is fused by each type of source according to the weight, and the source path and score are retained to help the user understand the basis given by the system. After the answer is generated, the fusion unit combines the structured text, screenshot image, predicted chart, and external link into a unified multi-modal answer package, and outputs it to the interface module or the subsequent feedback learning unit in a standardized format.

[0053] The complete process of the fusion unit 104 ensures that the system not only generates a targeted preliminary answer, but also constructs a composite answer result with knowledge depth and visual expression based on multi-source information, enabling the user to more comprehensively understand the problem background and coping strategies.

[0054] Further, the fusion unit is specifically used for: Based on the fault label information contained in the intelligent question and answer processing result, a semantic path expansion operation with a limited number of hops is performed in the graph database, starting from the knowledge graph node corresponding to the fault label. The semantic path expansion operation includes filtering according to the edge type defined in the graph, path weight, and semantic consistency of intersecting nodes, and constructing multiple reasoning paths with fault semantic association; The knowledge fragments corresponding to the nodes involved in each reasoning path are used as candidate answer sources, and the original content, reference frequency, update timeliness, and semantic relevance of the corresponding knowledge fragments in the graph database, local knowledge base, and network search information are obtained respectively. The source confidence score of each source is calculated; According to the source confidence score, a content reorganization and deduplication process is performed on the candidate knowledge fragments of multiple sources according to a preset dynamic fusion strategy, and a structured multi-modal answer information draft is output, which includes text description, image fragments, and predicted curve sorted by confidence weighting; Based on the semantic anchor points of the text description, an optical character recognition module is called to perform paragraph-level text positioning and structure recognition on the original document image, and a highlight frame is generated in the image fragment to mark the position of the semantic anchor points in the original text, thereby generating a multi-modal answer information with visual cues and semantic correspondence.

[0055] To achieve the graph database reasoning, multi-source belief fusion and graph-text interactive labeling functions described in the fusion unit, the entire implementation process should have a highly integrated and modular data flow design. This section describes the specific implementation of each key step in natural and fluent language to ensure that the skilled person can accurately implement it and avoid the common problems of abstract generalization and unclear operation path in the prior art.

[0056] First, the fusion unit extracts the fault label information contained in the intelligent question and answer processing results output by the equipment maintenance agent after receiving them. The fault label is a standard code after semantic normalization, such as "bearing_wear", "shaft_misalignment" or "seal_leakage", which corresponds to the node identifier of the knowledge graph in the preset graph database. The node identifier not only serves as a unique key value index graph, but also contains associated edge attributes, node types (such as phenomena, causes, treatment suggestions, etc.) and semantic relationship descriptions with other nodes.

[0057] Starting from the label node, the system performs a semantic path expansion operation in the graph database. Path expansion follows a hop limit setting, generally limited within three hops, to avoid semantic generalization and long reasoning chains. In each jump, the system filters according to the pre-defined edge types in the graph, such as only allowing "cause-effect relationship", "functional dependency" or "maintenance reference" type edges to be activated. At the same time, the system performs semantic consistency calculation on the intermediate nodes in the path. The specific method is to use BERT or RoBERTa language models to encode the semantic representation of the current path node and the starting fault label, respectively, to calculate the vector cosine similarity, and to ensure that the overall path has directional consistency in the semantic space. When multiple cross nodes (i.e. the same node belongs to multiple paths) are included in the path, the system will retain the path branch with the highest semantic consistency score and calculate the path weight value, which is defined as the product of the edge weight of each hop in the path and the average semantic similarity.

[0058] For each final retained inference path, the system extracts all the knowledge fragments linked by the nodes in the path as the core source for building candidate answers. Each node contains an index of resources such as literature, manuals, and operation logs pointing to local knowledge bases or external networked knowledge platforms in the knowledge graph. The system performs a four-dimensional source confidence assessment on each candidate fragment: first, the reference frequency, which counts the frequency of the fragment being called in historical work orders, user queries, and recommended replies; second, the update timeliness, which records the timestamp of the last content update of the fragment, reflecting its real-time nature in the form of a weight; third, the context completeness and paragraph position stability of the original content, which measures the sufficiency of its expression as an independent knowledge unit; and fourth, the semantic relevance, which matches the similarity coding between the full text of the candidate fragment and the fault label to obtain the content adaptation degree. After normalization, the four indicators generate a confidence score for the source through a preset function, for example: Source confidence = 0.4 x reference frequency score + 0.3 x update timeliness score + 0.2 x paragraph completeness score + 0.1 x semantic relevance score.

[0059] After that, the system performs a dynamic fusion strategy based on the confidence scores of multiple candidate sources. This strategy first ranks the candidate knowledge fragments in descending order of confidence and performs a content reorganization process, including merging duplicate paragraphs, detecting redundant images (such as multiple appearances of the same device perspective), and aligning trend paragraphs in the prediction curve. A confidence weighting mechanism is introduced during the fusion process, which prioritizes high-confidence source descriptions when multiple segments express the same fault cause or suggestion, and then uses the second-highest confidence description as an annotation supplement. The final output is a structured multi-modal reply information draft, with a format that includes a sorted text field, a matched image field, and a predicted time series field, each field with metadata labels to identify its source and confidence weight.

[0060] For each text description in the reply information draft, the system further calls an embedded optical character recognition module to analyze the original document image matched with it and locate the semantic anchor point in the document paragraph. The OCR module uses a pre-trained document structure recognition model (such as LayoutLM or Donut) to restore the block-level structure of the image, and after mapping the recognized paragraph to the text anchor point, it generates a positioning marker coordinate. The system then draws a highlight box centered on this coordinate in the image segment, using a color-to-confidence level mapping rule (such as green for high confidence and yellow for medium confidence) for visual hint rendering. For annotation areas located in complex backgrounds or overlapping layers, the system automatically adjusts the size of the box and the direction of the annotation arrow to ensure that users can quickly identify the annotated content in the visual interaction interface.

[0061] Finally, the fusion unit outputs the above multi-modal reply information with the graphic-text linkage to the reply generation module, and the subsequent process further performs language polishing, tone style generation or voice transcription, etc.

[0062] Further, the fusion unit is also used for: Based on the question type field in the question classification result, a preset modal configuration rule table is called to dynamically determine the combination mode of each modal element in the multi-modal reply information draft, the combination mode including text description, predicted curve, image segment and document jump link, wherein when the question type field represents the "equipment failure type", the predicted curve and image segment combination is preferentially enabled; when it represents the "regulation consultation type", the text description and document jump link combination is preferentially enabled. After generating the multi-modal reply information draft, a unique traceable label is attached to each modal element therein, the traceable label pointing to the corresponding original knowledge segment source node, and the node number sequence, edge weight cumulative value and path semantic similarity score in the graph database participating in the reasoning path are recorded, for supporting subsequent path backtracking and result review; When the multi-modal reply information draft is structured and displayed, a reinforcement learning display sorting model based on user interaction feedback is introduced, the reinforcement learning display sorting model receiving the click behavior, stay time and feedback score of different modal combinations in historical interaction as reward signals, and dynamically updating the display order weight parameter to form a user personalized multi-modal display configuration; In the final multi-modal reply information, the traceable label and original path annotation mechanism are retained, the jump entry of the original knowledge segment, graph path and source document is provided through an expandable link, the answer is traceable, verifiable and interactive, and the transparency and evolution ability of the complex task question and answer system in the industrial scene are improved.

[0063] In the present application, the fusion unit not only undertakes the task of content aggregation and multi-modal reply information generation, but also further undertakes the deep processing functions of multi-modal content structure configuration, source path tracing and user feedback driven personalized sorting. In order to ensure that different types of questions can obtain reply results with structure adaptation, accurate expression and explainability and evolution ability, the present application designs a dynamic modal configuration mechanism, a traceable path recording mechanism and a reinforcement learning driven display sorting optimization mechanism in the fusion unit, and each part cooperates to run.

[0064] Firstly, after the fusion unit receives the intelligent question and answer processing results output by the equipment maintenance intelligent agent, it will immediately read the question classification results contained therein and parse the question type field from the results, which indicates that the current question belongs to different types such as "equipment failure type", "procedural consultation type" or "comprehensive diagnosis type". The system has a built-in modal configuration rule table, which is composed of a number of modal combination strategies calibrated in advance, each strategy corresponds to a specific question type, and defines the priority and display strategy of each modal element. The range of modal elements includes text description, prediction curve, image segment and document jump link, each of which corresponds to different dimensions of reply expression and source path. Taking the "equipment failure type" question as an example, the system will preferentially select the future state curve generated by the prediction sub-model and the image segment extracted from the device image as the main component; while for "procedural consultation type" questions, structured text description and document jump link are preferred, and the jump link points to the anchor page of the corresponding procedural article in the local knowledge base or network query.

[0065] After completing the selection of modal combination structure, the system will perform traceability labeling processing on each modal element. This process attaches a uniquely identified label to each reply segment, which not only indicates the generation path of the information, but also contains the source node number sequence in the graph database, the edge weight cumulative value on the path and the path semantic similarity score. These information will be embedded in the metadata of each modal element in JSON structure, which is convenient for subsequent calling and visual presentation. The node number sequence records all the intermediate graph nodes experienced from the starting fault label node to the final information segment, the edge weight cumulative value reflects the total amount of semantic credibility on the path, and the semantic similarity score is an important basis for selecting the optimal path in the path selection stage.

[0066] After all modal elements are built and labeled with traceability, the system will enter the reply display sorting phase. This phase uses a reinforcement learning-based sorting optimization model, whose core idea is to continuously update the display strategy by introducing user interaction feedback behavior as a reward signal. The model extracts user click behavior, dwell time, sliding trajectory and artificial scoring data on different modal combinations from historical reply interaction records, and converts them into state-action-reward triplets in the reinforcement learning environment. For example, if a user shows a long dwell time and gives a positive feedback score in a reply containing a prediction curve, the system will automatically increase the display priority of such modal combinations in similar questions. In algorithm implementation, a deep reinforcement learning model based on policy gradient (Policy Gradient) such as REINFORCE or PPO can be used to ensure the stability and generalization ability of the sorting strategy when facing a large number of users and question types.

[0067] Finally, when presenting the reply content to the user front-end, the system not only sorts the structured multi-modal reply information individually, but also highlights the traceable labels and path annotations to achieve the traceability and interactivity of the reply content. All modal elements are configured with expandable links, and the link targets include the corresponding reasoning path graph in the graph database, the knowledge graph node detailed information, the original document screenshot or the jump link page. The semantic anchor points embedded in the text description are accurately located in the paragraph position of the original document image by the OCR module, and the visual highlight frame is rendered in the corresponding image segment, helping users quickly locate the context of the answer in the original material, forming a semantically consistent document annotation interface.

[0068] The following is the reference implementation code of the fusion unit: import json import uuid import random import torch import torch.nn as nn import torch.optim as optim import torch.nn.functional as F from typing import List, Dict # Example modal configuration rule table MODAL_CONFIG = { "Equipment failure class": ["prediction curve", "image segment"], "Procedural consultation class": ["text description", "document jump link"], "Comprehensive diagnosis class": ["text description", "prediction curve", "image segment", "document jump link"] } # Modal content generator simulation interface (interface with respective sub-models) def generate_modal_content(modal_type, question): return f"Simulated generated {modal_type}:{question}" # Simulate path trace information generator def generate_trace_metadata(): return { "source_node_ids": ["N1", "N2", "N3"], "edge_weight_sum": round(random.uniform(0.5, 1.0), 4), "semantic_similarity": round(random.uniform(0.6, 0.95), 4) } # Fusion Unit Class class FusionUnit: def __init__(self): self.modal_config = MODAL_CONFIG self.sorter = ReinforceRanker(state_size=10, action_size=4) # Reinforcement learning sorter, the action space represents the order of modality combinations. def process(self, question_result): question_type = question_result["type"] content = question_result["content"] modal_sequence = self.modal_config.get(question_type, []) multi_modal_draft = [] for modal in modal_sequence: modal_content = generate_modal_content(modal, content) trace = generate_trace_metadata() element = { "modal_type": modal, "content": modal_content, "trace_tag": str(uuid.uuid4()), "trace_metadata": trace, "link": f"http: / / example.com / path?modal={modal}&id={trace['source_node_ids'][-1]}" } multi_modal_draft.append(element) # Construct a status representation (which can be populated based on actual user profiles or problem characteristics) state = torch.rand(10) action_idx, log_prob = self.sorter.select_action(state) sorted_draft = self.apply_action_sort(multi_modal_draft,action_idx) # Return the final multimodal response result and log_prob for subsequent training. return sorted_draft, log_prob def apply_action_sort(self, draft, action_idx): #Action sorting strategy (for handling more granular display strategies) return sorted(draft, key=lambda x: (hash(x["modal_type"]) +action_idx) % 10) # Reinforcement Learning Ranking Machine (REINFORCE Implementation) class ReinforceRanker(nn.Module): def __init__(self, state_size, action_size): super(ReinforceRanker, self).__init__() self.fc1 = nn.Linear(state_size, 64) self.fc2 = nn.Linear(64, action_size) self.optimizer = optim.Adam(self.parameters(), lr=0.001) def forward(self, x): x = F.relu(self.fc1(x)) return F.softmax(self.fc2(x), dim=-1) def select_action(self, state): probs = self.forward(state) m = torch.distributions.Categorical(probs) action = m.sample() return action.item(), m.log_prob(action) def update_policy(self, log_probs, rewards, gamma=0.99): G = 0 loss = 0 for r, log_prob in zip(reversed(rewards), reversed(log_probs)): G = r + gamma * G loss -= log_prob * G self.optimizer.zero_grad() loss.backward() self.optimizer.step() # Example call logic if __name__ == '__main__': fusion_unit = FusionUnit() # Result returned by the simulated question-and-answer platform result = { "type": "Equipment Failure Category", "content": "Abnormal noise from fan blades" } draft, log_prob = fusion_unit.process(result) # Simulated User Feedback Rewards reward = 1.0 if draft[0]["modal_type"] == "predicted curve" else 0.0 fusion_unit.sorter.update_policy([log_prob], [reward]) # Show final results for m in draft: print(json.dumps(m, ensure_ascii=False, indent=2)) Furthermore, the aforementioned intelligent question-and-answer system for industrial equipment maintenance based on multi-agent collaboration also includes: The control unit is used to receive the multimodal response information and user feedback information, update the multilingual intent recognition model and knowledge weight configuration based on the user feedback information, and perform permission verification and tamper-proof storage processing on the multimodal response information using a role-based access control mechanism and a blockchain-based operation log mechanism.

[0069] This invention further introduces a control unit, aiming to construct a closed-loop question-answering system architecture with self-optimization capabilities, security and controllability, and reliable auditing functions. The control unit is responsible for receiving multimodal response information output by the fusion unit and feedback information returned by the user terminal. Based on this, it dynamically adjusts the semantic recognition model and knowledge fusion mechanism within the system, thereby enabling the system to learn and adaptively optimize user behavior during actual operation. Furthermore, the control unit also undertakes critical data access control and auditable operation recording tasks, ensuring that the system operation complies with information security and compliance requirements in industrial scenarios.

[0070] In a specific implementation, the control unit first receives feedback information from the front end or interface module, which can include user satisfaction evaluation, correction suggestions, error labeling, supplementary questions, or comparative queries. The system associates this feedback data with the original structured question information and the corresponding multi-modal reply information, and constructs a feedback sample data pair as a data source for incremental training. After preprocessing, the feedback data is automatically labeled as positive or negative feedback and input into the multi-language intent recognition model in the system for fine-tuning the model classification weights. Specifically, the control unit calls a model training component based on a streaming update framework (such as River or similar lightweight neural network parameter update module), and only adjusts the weights of the classification head part quickly, keeping the language model backbone parameters stable, thus balancing stability and adaptability. During this process, the system maintains a feedback buffer, accumulates samples in time windows and triggers the training process regularly, while recording model change logs for recovery or backtracking.

[0071] In terms of knowledge fusion, the control unit also dynamically scores the use frequency and answer contribution of local knowledge base, graph database, or network search results based on feedback results, and adjusts the weight configuration of various information sources during knowledge fusion. For example, if multiple user feedback indicates that a certain knowledge fragment is misleading, the weight of this fragment in future retrieval will be reduced, or even enter the gray list to prohibit indexing; on the contrary, frequently cited document paths or graph paths will have higher confidence priority and be preferentially cited in multi-hop reasoning. This dynamic knowledge weight mechanism is driven by a configurable parameter file and cooperates with a data version control mechanism to ensure information consistency and controllability during knowledge evolution.

[0072] In terms of system security and permission management, the control unit embeds an access control mechanism based on RBAC (Role-Based Access Control). The system first identifies the role category of the current user through user identity authentication (such as account, access token, or device signature), such as field operator, system engineer, maintenance manager, or external auditor. Then, according to the role and permission mapping table, it determines whether the user has the right to view a certain type of reply information, especially content related to core technical documents, historical alarm curves, or supply chain compliance information. If the user does not have access rights, the system will automatically shield sensitive fields and return a prompt message to avoid information leakage risks.

[0073] To ensure the authenticity and non-tamperability of all operation records, the control unit also integrates a blockchain operation log mechanism. Each time a user query, multi-modal reply generation, model update, or knowledge weight adjustment operation is performed, the system generates a corresponding operation summary, constructs a hash, and records it in a private blockchain. Blockchain nodes can be deployed on enterprise local servers, edge computing nodes, or cloud control platforms to ensure consensus synchronization and long-term preservation of log data in multiple terminals and locations. If future audits, responsibility tracking, or system state backtracking are required, the blockchain records can serve as an authoritative evidence chain to provide technical support.

[0074] In summary, the control unit not only implements a closed-loop self-learning path from user feedback to model optimization and knowledge weight adjustment, but also ensures the compliance and controllability of system data access through a role-based permission management mechanism, and builds an auditable and trustworthy operation record system with the help of a blockchain log mechanism.

[0075] Furthermore, the control unit is specifically used for: receiving a semantic difference value between task expression in the user feedback information and the multi-modal reply information, interface interaction behavior data, and access user identification information, wherein the semantic difference value is obtained by calculating the vector cosine similarity of user natural language input and system semantic analysis results in a unified embedding space, the interface interaction behavior data includes the number of clicks, dwell time, and feedback scores for each modal element in the multi-modal reply information, and the access user identification information includes an access timestamp, an access permission level, and an access path identification code; based on the semantic difference value and the interface interaction behavior data, extracting modal elements with semantic deviations in the reply content, and generating a correction parameter set containing language adjustment prompts, reply completion suggestions, and modal weight correction suggestions, the correction parameter set records key expression deviation points between original reply content and user actual needs, forming feedback samples that can be used to improve the reply generation process; reconfiguring the multi-modal reply information's modal composition strategy using the correction parameter set, specifically including: optimizing the text description expression method according to the language adjustment prompt; adding missing image segments, predicted curves, or document jump links according to the reply completion suggestion; and updating the modal priority order in the modal configuration rule table according to the modal weight correction suggestion to dynamically adjust the generation structure of the multi-modal reply information in subsequent problems of the same type; The access user identification information is bound to each feedback record in the correction parameter set, and a unique digest value is generated using a hash algorithm and timestamp encryption. The digest value is written into the on-chain storage structure together with the multi-modal reply information, the semantic difference value and the interface interaction behavior data, broadcast to the blockchain node network, forming an audit entry with tamper-proof characteristics, supporting the source tracing, responsibility identification and evolution verification of any multi-modal reply information.

[0076] In the industrial equipment maintenance intelligent question and answer system based on multi-agent cooperation according to the present application, the control unit not only has the basic reply receiving and distribution function, but also further realizes the deep structure analysis of user feedback information, semantic deviation quantization processing, modal strategy dynamic adjustment and result storage, etc. The following will explain in detail the technical path of the control unit when implementing the present application.

[0077] Firstly, after receiving the user feedback information, the control unit needs to extract three core data elements from the feedback information, namely the semantic difference value between the task expression and the reply content, the interface interaction behavior data and the access user identification information. The "semantic difference value" is used to quantify the expression deviation between the multi-modal reply information generated by the system and the original task intention of the user. The control unit embeds the user natural language input and the system reply content into the same high-dimensional vector space through unified semantic embedding space modeling technology, and then calculates the cosine similarity between them to obtain the semantic consistency degree. When the cosine similarity value is lower than the system set threshold (for example, 0.85), it indicates that there is significant semantic deviation, which needs to be further processed.

[0078] At the same time, the collection of interface interaction behavior data is also one of the core functions of the control unit, mainly including the number of clicks, dwell time, sliding track and artificial score of each modal element (such as text description, prediction curve, image segment, document jump link) in the reply interface. The control unit records the user's operation behavior on the reply content in real time through front-end burying technology combined with back-end event listening mechanism. For example, when a prediction curve is clicked twice, the dwell time is 6 seconds, and the user's final score is "less satisfied", this modal is recorded as a negative feedback sample. The behavior data is encoded as a structured entry and is included in the feedback record set under the current session identifier. At the same time, the control unit also receives the identity information of the access user, including the access timestamp (accurate to seconds), the user access permission level (such as maintenance worker, engineer, manager, etc.) and the unique access path identifier code (used to record which entrance the user enters the current question and answer system), which together constitute the access user identification information.

[0079] After the above basic data extraction is completed, the control unit enters the semantic deviation identification and correction suggestion generation stage. The system first screens out the question and answer interaction items whose deviations exceed the threshold value according to the semantic difference value, and further combines the interaction behavior data to evaluate each modal element in the reply content one by one, and identifies the modal elements whose semantic coverage is not complete, the content is missing, or the expression does not conform to the user's intention. The control unit then generates a correction parameter set for these problem modes, which includes three key subsets: language adjustment prompt, reply completion suggestion, and modal weight correction suggestion. The language adjustment prompt points out the incomplete expression or word ambiguity in the reply text, such as the lack of parameter unit or operation sequence in operation instruction type questions; the reply completion suggestion predicts the user's expected content form according to the user's behavior, such as suggesting to supplement the high-resolution image segment of the part when the user fails to click and enlarge the operation on the device picture; the modal weight correction suggestion configures new weight factors for the modal configuration rule table according to the click rate and score performance of different modalities in the interaction, such as adjusting the "prediction curve" from the default third to the second.

[0080] The generated correction parameter set is not only used for the improvement of the current question and answer content, but also used for the reconstruction of the reply generation strategy for similar problems in subsequent question and answer. The control unit dynamically adjusts the modal combination structure according to the correction parameter set, first fine-tunes the language level of the text description part to ensure complete task intention coverage and use of domain terminology to maintain professional expression; second, automatically calls the image library and document index system according to the completion suggestion to insert missing image segments or add jump anchor points; finally, updates the priority array of each modal item in the modal configuration rule table according to the modal weight correction suggestion, which will directly affect the modal selection and ordering of reply information in future similar problems, and thus realize personalized optimization.

[0081] In order to realize the whole life cycle of question and answer behavior and feedback track, the control unit will use encryption technology to build an immutable on-chain data record after completing the above strategy reconstruction. First, bind the access user identification information with each correction parameter record, and combine it with the structured content such as multi-modal reply information, semantic difference value, interface interaction behavior data, etc. to calculate its digest value through SHA-256 and other secure hash algorithms. The control unit embeds a timestamp field in the digest value to ensure that even if the same user submits the same feedback content, its record also has global uniqueness. Then, the digest value and its corresponding data entry are written into the blockchain ledger, broadcast to each node in the blockchain network using consensus algorithm, forming an irreversible transaction record. This record supports future users to initiate responsibility traceability, feedback callback or improvement path verification request for any reply content, and the system can restore the multi-modal composition strategy, feedback source and update chain based on the record in the blockchain, thereby ensuring that the evolution process of the system has complete explainability and data compliance.

[0082] Through the above structural design, the control unit of the application realizes the three-in-one technical effects of feedback-driven content evolution, adaptive strategy updating and compliance traceability, effectively breaking through the problems of static reply strategy, neglect of user differences and inability to track historical updates in existing industrial question and answer systems.

[0083] A specific example is provided below: Taking a certain actual scene as an example, a device inspection officer with an intermediate permission level submits the natural language question “What failures will the frequent start-stop of the device cause?” through the question and answer interface during annual maintenance of the compressor unit. After the system identifies the intent and retrieves the knowledge, it generates a multi-modal reply information containing text description, prediction curve and a historical fault picture segment. However, the user clicks on the prediction curve zero times, stays for less than one second, and then marks “not solved current problem” in the feedback area and mentions “more concerned about the allowed number of times of frequent start-stop in the specification” in the manual supplementary description. The control unit immediately analyzes the interaction behavior and supplementary text, and identifies that the modal combination in the original reply does not fully cover the user's demand focus. Through semantic embedding space calculation, the semantic difference value between the original reply and the user's supplementary expression is only 0.67, which is far below the system threshold, indicating that there is a significant deviation.

[0084] The system thereby generates the language adjustment prompt “supplement the reference description of the regulation clause about frequent start-stop” in the revision parameter set, generates the reply completion suggestion “add a procedure text jump link related to the problem”, and based on the user's low score behavior and the low interaction value of the prediction curve mode, forms the modal weight revision suggestion “downgrade the priority of the prediction curve and upgrade the display weight of the document jump link”. The above revision parameters and the user's access identification information (such as timestamp, permission level and access portal) are bound and coded into a structured feedback sample, and the digest value is calculated by the SHA-256 algorithm and written into the blockchain storage node. The next time the system encounters a similar question and identifies the user, it will automatically prioritize the presentation of structured clause content and highlighted document jump link based on the updated modal configuration strategy, without displaying the prediction curve, significantly improving the response accuracy and user satisfaction of the question and answer. This process not only completes individual optimization, but also forms a complete and auditable reply evolution chain.

[0085] Although the above-mentioned preferred embodiments are disclosed, the application is not intended to be limited thereto, and any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of the application. Therefore, the protection scope of the application should be subject to the scope defined by the claims of the application.

Claims

1. An industrial equipment maintenance intelligent question and answer system based on multi-agent cooperation, characterized in that, The method comprises the following steps: An input unit is configured to receive multi-modal user input information, and parse the multi-modal user input information into structured question information, wherein the multi-modal user input information comprises text input, voice input, image input or device scanning input; A scheduling unit is configured to perform semantic understanding and question classification on the structured question information based on a multi-language intention recognition model and an incremental training mechanism, and obtain a question classification result, wherein the multi-language intention recognition model is a fine-tuned cross-language pre-trained language model; A processing unit is configured to call a domain intelligent agent corresponding to the question classification result to generate an intelligent question and answer processing result, wherein the domain intelligent agent comprises a device maintenance intelligent agent, a management consultation intelligent agent, a digital operation and maintenance intelligent agent or a technical service intelligent agent; the intelligent question and answer processing result comprises a predictive maintenance suggestion, structured reply content and semantic annotation information; A fusion unit is configured to perform multi-hop semantic reasoning based on the intelligent question and answer processing result, and fuse local knowledge base information, graph database information and network search information, to obtain multi-modal reply information, wherein the multi-modal reply information comprises text description, image screenshot, prediction curve and external recommendation link.

2. The multi-agent collaboration based industrial equipment maintenance intelligent question and answer system according to claim 1, characterized in that, Further comprising: A control unit is configured to receive the multi-modal reply information and user feedback information, update the multi-language intention recognition model and knowledge weight configuration based on the user feedback information, and perform permission verification and tamper-proof storage processing on the multi-modal reply information by using a role-based access control mechanism and a blockchain-based operation log mechanism. 3.The industrial equipment maintenance intelligent question-answering system based on multi-agent collaboration of claim 1, wherein, The scheduling unit is specifically configured to: Receive the structured question information output by the input unit, and extract text input fields, voice recognition transcription fields and image recognition tag fields therefrom to form a multi-modal semantic tag sequence; Concatenate the multi-modal semantic tag sequence into a unified input vector according to a preset encoding rule, and input the unified input vector into the multi-language intention recognition model, wherein the multi-language intention recognition model is a fine-tuned cross-language pre-trained language model, and is used to generate an embedding vector containing language-independent semantic features; Based on the semantic content represented by the embedding vector, use a semantic consistency discrimination network to evaluate the stability of the language mixing features of the input information, and output a language consistency score; According to the language consistency score and the user feedback information, only update the classification head parameters in the multi-language intention recognition model by using a streaming incremental fine-tuning mechanism, keep the bottom encoder parameters in a frozen state, and output a question classification result according to the fine-tuned model.

4. The multi-agent collaboration based industrial equipment maintenance intelligent question and answer system according to claim 1, characterized in that, The device maintenance intelligent agent is specifically configured to: Receive the question classification result and the device identification field, fault description field and image recognition tag field in the structured question information, and construct a structured input sample for predictive modeling; Input the structured input sample into a preset fault mode and health management model, wherein the fault mode and health management model generates a prediction curve based on a time series prediction model, and maps the current device state based on a fault label system defined in a knowledge graph; Based on the generated prediction curve and the fault label, a maintenance priority score of each device component is calculated, which is a priority value obtained by fusing the confidence function output and the remaining life estimation result according to a preset weighting coefficient; According to the maintenance priority score, in combination with the hierarchical relationship between the device structure nodes in the knowledge graph, an intelligent question and answer processing result is generated, which includes predictive maintenance suggestions, structured reply content and semantic annotation information.

5. The multi-agent collaboration based industrial equipment maintenance intelligent question and answer system according to claim 1, characterized in that, The fusion unit is specifically used for: Based on the fault label information contained in the intelligent question and answer processing result, a semantic path expansion operation with a limited number of hops is performed in the graph database starting from the knowledge graph node corresponding to the fault label, the semantic path expansion operation including filtering according to the edge type, path weight and cross-node semantic consistency defined in the graph, and constructing a plurality of reasoning paths with fault semantic association; The knowledge fragments corresponding to the nodes involved in each reasoning path are taken as candidate answer sources, and the original content, reference frequency, update timeliness and semantic relevance of the corresponding knowledge fragments in the graph database, local knowledge base and network search information are obtained respectively, and the source confidence score of each source is calculated; According to the source confidence score, content reorganization and deduplication processing are performed on the candidate knowledge fragments of multiple sources according to a preset dynamic fusion strategy, and a structured multi-modal reply information draft is output, which includes text description, image fragments and prediction curve sorted by confidence weighting; Based on the semantic anchor point of the text description, an optical character recognition module is called to perform paragraph-level text positioning and structure recognition on the original document image, and a highlight frame is generated in the image fragment to mark the position of the semantic anchor point in the original text, thereby generating a multi-modal reply information with visual prompts and semantic correspondence.

6. The industrial equipment maintenance intelligent question and answer system based on multi-agent collaboration according to claim 5, characterized in that, The fusion unit is also used for: Based on the question type field in the question classification result, a preset modal configuration rule table is called to dynamically determine the combination mode of each modal element in the multi-modal reply information draft, the combination mode including text description, prediction curve, image fragment and document jump link, wherein when the question type field represents "device fault class”, the prediction curve and image fragment combination is preferentially enabled; when it represents "procedure consultation class”, the text description and document jump link combination is preferentially enabled; After generating the multi-modal reply information draft, a unique traceable label is attached to each modal element in it, the traceable label pointing to the corresponding original knowledge fragment source node, and the node number sequence, edge weight cumulative value and path semantic similarity score in the graph database participating in the reasoning path are recorded, for supporting subsequent path backtracking and result review; When the multi-modal reply information draft is structured and displayed, a reinforcement learning ranking model based on user interaction feedback is introduced, which receives the click behavior, dwell time and feedback score of different modal combinations in historical interactions as reward signals, and dynamically updates the display order weight parameters to form a user personalized multi-modal display configuration; In the final presented multi-modal reply information, the traceable label and original path annotation mechanism are retained, and the jump entry to the original knowledge fragment, graph path and source document is provided through an expandable link.

7. The multi-agent collaboration based industrial equipment maintenance intelligent question and answer system according to claim 2, characterized in that, The control unit is specifically used for: Receiving the semantic difference value between the task expression in the user feedback information and the multi-modal reply information, interface interaction behavior data and access user identification information, wherein the semantic difference value is obtained by calculating the vector cosine similarity of the user natural language input and the system semantic analysis result in the unified embedding space, the interface interaction behavior data includes the click number, dwell time and feedback score of each modal element in the multi-modal reply information, and the access user identification information includes access timestamp, access permission level and access path identification code; Based on the semantic difference value and the interface interaction behavior data, the modal elements with semantic deviation in the reply content are extracted, and a correction parameter set including language adjustment prompt, reply completion suggestion and modal weight correction suggestion is generated, the correction parameter set records the key expression deviation point between the original reply content and the actual user demand, forming a feedback sample that can be used to improve the reply generation process; The modal composition strategy of the multi-modal reply information is reconstructed using the correction parameter set, specifically including: optimizing the text description expression according to the language adjustment prompt; according to the reply completion suggestion, adding the missing image fragment, predicted curve or document jump link; according to the modal weight correction suggestion, updating the modal priority order in the modal configuration rule table to dynamically adjust the generation structure of the multi-modal reply information in subsequent problems of the same type; The access user identification information and each feedback record in the correction parameter set are bound, and a unique digest value is generated using a hash algorithm and timestamp encryption, the digest value is written into the on-chain storage structure together with the multi-modal reply information, the semantic difference value and the interface interaction behavior data, broadcast to the blockchain node network, form an audit entry with the characteristics of non-tamperability, support the source tracing, responsibility identification and evolution verification of any multi-modal reply information.

Images