Device abnormal visual diagnosis method, electronic device and storage medium

Abstract

The application belongs to the field of artificial intelligence, and particularly relates to a device abnormality visual diagnosis method, an electronic device and a storage medium. The method comprises the following steps: acquiring a video containing a device running state field, a text consultation question and device prior metadata, and generating a multi-modal data package; extracting a key frame visual embedding vector from a video frame stream and extracting an audio event label from an audio frame stream; performing coarse detection based on the key frame visual embedding vector, and determining a time interval in which a preset device exists; performing fine positioning based on image data of the key frame in the time interval and a text interaction instruction, and outputting fault positioning information; constructing a multi-modal time sequence semantic graph based on the fault positioning information, the audio event label and the text interaction instruction to perform multi-modal reasoning, and generating fault diagnosis structured data. The method of the application realizes low-latency and high-accuracy diagnosis of a device field video through deep fusion of multi-modal information by a visual large model and a large language model.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence, and specifically relates to a visual diagnostic method for device anomalies, an electronic device, and a storage medium. Background Technology

[0002] Traditional product after-sales customer service platforms mainly rely on plain text semantic automatic replies or static image manual communication or link guidance. When faced with multimodal videos uploaded by users that contain continuous images and audio tracks, it often requires human customer service to check frame by frame, manually take screenshots, join group discussions, and provide text descriptions before a reply can be given. This not only prolongs the response chain but also significantly increases labor costs and makes it difficult to meet users' requirements for 24 / 7 real-time service.

[0003] The emergence of large-scale visual language models has provided new opportunities for cross-modal understanding and diagnostic reasoning. These models can combine natural language instructions with visual perception, supporting more flexible interactions and decision-making. However, their direct application in industrial fields still faces the following challenges:

[0004] 1) Scene recognition deficiency: It cannot automatically filter videos from non-preset devices or irrelevant segments in videos, resulting in wasted computing resources and misdiagnosis;

[0005] 2) Inefficient processing flow: It requires manual downloading of videos, frame-by-frame viewing, and screenshot comparison, resulting in a long response chain and high labor costs;

[0006] 3) Insufficient modal fusion: Most solutions only analyze single-frame visual features and cannot effectively combine temporal and acoustic information such as flashing indicator lights and beep spectrum of the modeling device, thus ignoring the diagnostic value of multimodal cues;

[0007] 4) Difficulty in understanding and locating intent: The platform cannot automatically understand whether the user is asking "troubleshooting" or "function usage", and it also lacks the ability to automatically highlight faulty parts or operation buttons in the video;

[0008] 5) Poor real-time performance and high iteration cost: The inference latency is high, making it difficult to meet the second-level diagnostic requirements; and when new firmware or new alarm codes appear, a large number of samples need to be manually re-labeled and the entire model needs to be retrained, resulting in a long maintenance cycle and high costs.

[0009] Therefore, there is an urgent need for a device intelligent diagnostic solution that can integrate visual, audio, and text information to achieve low latency and high accuracy. Summary of the Invention

[0010] This application provides a method for visual diagnosis of equipment anomalies, an electronic device, and a storage medium. The aim is to align multimodal information such as vision, audio, and text, and to achieve low-latency, high-accuracy diagnosis and visual interaction of on-site equipment video through deep fusion of Visual Model (VLM) and Large Language Model (LLM).

[0011] In a first aspect, embodiments of this application provide a method for visual diagnosis of device anomalies, the method comprising:

[0012] Step S1: Obtain a video containing the on-site operating status of the device, text consultation questions associated with the video, and prior metadata of the device, and generate a unified encapsulated multimodal data packet with a timestamp;

[0013] Step S2: Preprocess the multimodal data packet, extract the visual embedding vector of the key frame from the video frame stream of the multimodal data packet, extract the audio event tag from the audio frame stream of the multimodal data packet, and generate a cross-modal semantic tensor based on timestamp alignment.

[0014] Step S3: Based on the visual embedding vector of the key frame, use a pre-trained coarse-grained visual language model to determine whether a preset device exists in the video frame stream, and determine the time interval in which the preset device exists.

[0015] Step S4: Using a pre-trained fine-grained visual language model, the image data of key frames located within the time interval and the text interaction instructions obtained based on the text consultation question are fused through cross-attention to output fault location information;

[0016] Step S5: Construct a multimodal temporal semantic graph based on the fault location information, the audio event tags, and the text interaction instructions; use a cross-modal Transformer and a causal reasoning large language model to perform chain reasoning to generate structured fault diagnosis data.

[0017] Optionally, extracting the visual embedding vectors of keyframes from the video frame stream of the multimodal data packets includes:

[0018] A lightweight visual language model is used to encode each video frame of the multimodal data packet, and the visual embedding vector of each video frame is output.

[0019] Calculate the cosine similarity of the visual embedding vectors of adjacent video frames, and combine it with optical flow changes to determine motion saliency, thereby selecting keyframes.

[0020] Optionally, extracting audio event tags from the audio frame stream of the multimodal data packet includes:

[0021] The audio frame stream of the multimodal data packet is subjected to source separation to obtain pure human voice and device background sound;

[0022] The background noise of the device is detected based on preset audio events, and audio event tags are obtained.

[0023] Optionally, the step of using a pre-trained coarse-grained visual language model to determine whether a preset device exists in the video frame stream based on the visual embedding vector of the keyframe includes:

[0024] A device category text hint is set for the coarse-grained visual language model, and the coarse-grained visual language model generates a device category query vector based on the device category text hint;

[0025] The visual embedding vector of the keyframe is input into the coarse-grained visual language model. The coarse-grained visual language model calculates the similarity between the visual embedding vector of the keyframe and the device category query vector in the shared semantic space. When the maximum similarity exceeds a preset similarity threshold, it is determined that a preset device exists in the keyframe.

[0026] Optionally, step S4 includes:

[0027] The text interaction instructions in the multimodal data packet are input into the fine-grained visual language model, and the fine-grained visual language model generates a fault query vector based on the text interaction instructions and stores it in the query pool;

[0028] Image data of key frames located within the time interval are input in batches into the fine-grained visual language model. The fine-grained visual language model obtains multi-scale feature maps based on the image data of the key frames. The multi-scale feature maps are then fused with the fault query vector through cross-attention to generate sparse interest points.

[0029] The points of interest are fine-tuned in coordinates using a bounding box regressor, and a multi-channel hotspot heatmap header is set simultaneously to generate a pixel-level fault saliency mask, resulting in fault location information including device bounding boxes, fault hotspot masks, fault semantic labels, fault location confidence, and timestamps.

[0030] Optionally, after step S4 and before step S5, the method further includes:

[0031] Based on the confidence scores of the device bounding box, the fault semantic labels, the audio event labels, and the device prior metadata, a comprehensive confidence score is generated by fusing Bayesian calibration and semantic consistency verification.

[0032] Based on the comprehensive confidence level and dynamic threshold, the fault clue information is divided into high-confidence type and low-confidence type, and the fault clue information includes the fault location information and the audio event tag;

[0033] When the fault clue information is of the high confidence type, proceed to step S5.

[0034] Optionally, step S5 may be followed by:

[0035] Step S6: Based on the fault diagnosis structured data and the equipment prior metadata, a repair solution is retrieved and matched in the pre-built knowledge database, and a diagnostic report integrating visual annotations and structured text descriptions is generated.

[0036] Optionally, step S6 includes:

[0037] Based on the fault diagnosis structured data and the equipment prior metadata, a sparse query vector is constructed.

[0038] Based on the sparse query vector, a search is performed in a pre-constructed knowledge database to obtain a set of candidate entries;

[0039] The knowledge graph inference engine is used to filter out the candidate entries from the candidate entry set that best match the fault diagnosis structured data, and a maintenance plan is obtained based on the candidate entries.

[0040] Secondly, embodiments of this application provide an electronic device, including at least one processor and a memory communicatively connected to the at least one processor. The memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the method described above.

[0041] Thirdly, embodiments of this application provide a computer storage medium storing instructions or programs that, when executed by at least one processor, cause the at least one processor to perform the method described above.

[0042] In this embodiment, a visual diagnostic method for equipment anomalies is provided. First, a video containing the equipment's operating status, text-based consultation questions associated with the video, and prior metadata of the equipment are acquired to generate a unified, encapsulated, timestamped multimodal data packet. Then, time-aligned cross-modal semantic features are extracted from this data packet. A pre-trained visual language model is used to perform two-level processing on keyframes in the video frame stream: first, rapid coarse detection and identification of the target equipment; then, fine-grained analysis of the time period in which the equipment appears to output accurate fault location information. Finally, fault location information, audio event tags, and text interaction commands are fused to construct a multimodal temporal semantic graph and drive a cross-modal Transformer and a large language model to perform chain-like causal reasoning, generating structured fault diagnosis data. This achieves end-to-end intelligent diagnosis from multimodal perception to deep causal analysis. The method in this application quickly filters out non-target scenes using a lightweight model and utilizes multimodal parallel processing and chained reasoning to shorten the diagnostic response time from hours to minutes or even seconds, achieving efficient real-time online diagnosis 24 / 7. By integrating visual, audio, and textual multimodal information and performing multi-level processing such as coarse detection, fine localization, and cross-modal reasoning, the accuracy of fault diagnosis is significantly improved and the false alarm rate is effectively reduced. Attached Figure Description

[0043] Figure 1 An example is shown in the architecture diagram of an intelligent diagnostic question-answering system;

[0044] Figure 2 An exemplary flowchart of a visual diagnostic method for device anomalies is shown;

[0045] Figure 3 An exemplary diagram illustrates the main business process for visual diagnosis of equipment malfunctions.

[0046] Figure 4 An exemplary schematic diagram of the hardware structure of an electronic device is shown. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0048] The technical terms used in this application are explained below:

[0049] 1. FFMPEG: An open-source multimedia processing tool that supports video and audio conversion, editing, streaming, and playback.

[0050] 2. Device SDK: This refers to a software development kit that runs on the device side (such as mobile phones, embedded devices, etc.), providing necessary interfaces and functions to support local application development and integration.

[0051] 3. Circular zero-copy buffer: This is a memory management technique that uses a fixed-size buffer to store data in a circular manner, thereby reducing the overhead of data copying and improving performance.

[0052] 4. VLM-Coarse: This is a coarse version of a visual large language model, typically used to handle simpler or lower-resolution visual and language tasks to improve computational efficiency and response speed.

[0053] 5. VLM-Fine: This is a refined version of the Visual Large Language Model, which is obtained by making targeted adjustments to a large-scale pre-trained VLM using domain-specific data.

[0054] 6. LLM-Edge: This refers to deploying Large Language Models (LLMs) to edge devices for local processing to achieve low latency, save bandwidth, and improve privacy protection.

[0055] 7. Distillation technique: This refers to training a smaller model (student model) to mimic the behavior of a larger, more complex model (teacher model), thereby improving the computational efficiency and inference speed of the model while maintaining performance.

[0056] 8.7 Level B parameter weights: These refer to the approximately 7 billion trainable parameters in the model, used to represent the model's complexity and learning ability, and are typically found in large deep learning models.

[0057] 9. Anchor-free: This refers to object detection where the model does not rely on predefined anchor boxes for position prediction, but directly generates the position and size of the target, thereby simplifying model design and improving flexibility.

[0058] 10. Four-boundary offset: In image processing or object detection, this refers to the offset of the four boundaries of a frame relative to their original positions, used to adjust the accuracy of object position or boundaries.

[0059] 11. Center confidence: In target detection or localization tasks, this refers to the confidence level of the model's prediction of the target's center location, used to assess whether that location is likely the true center of the target.

[0060] 12. KL-divergence loss (Kullback-Leibler Divergence Loss): This is a loss function used to measure the difference between two probability distributions. It is often used to optimize models so that their output distribution is as close as possible to the target distribution.

[0061] 13. Box regressor: In object detection, this refers to a model used to predict the bounding box (such as position and size) of an object, which optimizes the position and size of the box through regression methods.

[0062] 14. NMS-Suppression (Non-Maximum Suppression): This is a technique in object detection used to remove multiple candidate boxes with high overlap and retain the most confident box, thereby reducing redundant detection results.

[0063] 15. Visual hash fingerprint: This method converts an image into a fixed-length hash value for fast and unique identification. It is commonly used in image retrieval, duplicate detection, and other applications.

[0064] 16. Bayesian correction: This method uses Bayesian inference to adjust the probability of the model's prediction results in order to improve the confidence and accuracy of the prediction. It is widely used, especially in dealing with uncertainty and bias.

[0065] 17. Transformer classifier: A classifier based on the Transformer model architecture. It captures important features in the input data through a self-attention mechanism and is widely used in classification tasks such as text and images.

[0066] 18. LLM-Causal: This model combines Large Language Model (LLM) and causal reasoning to make more accurate inferences and decisions by analyzing causal relationships in text.

[0067] 19. IVF-HNSW Two-Level Index: Combining the Inverted File (IVF) and Hierarchical Navigation Small World (HNSW) algorithms, it is used for efficient vector retrieval. The two-level index structure accelerates the similarity search process and is especially suitable for large-scale datasets.

[0068] 20. Heatmap Header: In a deep learning model, this refers to the output layer used to generate heatmaps, typically used for localization tasks such as object detection and keypoint detection. Heatmaps represent the probability distribution of a specific location.

[0069] 21. Lightweight LLM-Router: This is an optimized, low-resource-consumption model router used for intelligent selection and routing tasks among multiple Large Language Models (LLMs), improving computational efficiency and response speed.

[0070] Please refer to Figure 1 , Figure 1 The architecture diagram of the intelligent diagnostic question-answering system is shown. This system includes a data acquisition and encapsulation module 1, a preprocessing module 2, a device coarse inspection module 3, a fault location module 4, a confidence gating module 5, a cross-modal reasoning module 6, a knowledge retrieval module 7, and a diagnosis generation module 8.

[0071] Among them, the data acquisition and encapsulation module 1 is used to acquire video containing the on-site operation status of the equipment, text consultation questions associated with the video, and prior metadata of the equipment, and generate a unified encapsulated multimodal data packet with a timestamp.

[0072] In one embodiment, the intelligent diagnostic question-and-answer system has a video upload entry on the device side. Users can record or photograph the device's operating status to obtain a video containing the device's operating status, and then upload the video to the backend server through the video upload entry. After detecting the uploaded video, the backend server generates a unique upload ID for the uploaded video data. At the same time, it controls the device side to pop up a structured form, obtains data such as device model, serial number, firmware version, and alarm code supplemented by the user in the structured form, forms the device's prior metadata, and marks the device's prior metadata with the same upload ID.

[0073] After receiving the uploaded video, the backend server uses FFmpeg to encapsulate the uploaded video into a standard container format, separates the audio track from the video, extracts video frames and audio frames at a rate of 1 frame per second, and appends the corresponding timestamp to the extracted video frames and audio frames.

[0074] In one embodiment, the intelligent diagnostic question-and-answer system has a command input interface on its device side, allowing users to upload text-based or voice-based questions to the backend server. For voice-based text-based questions, the backend server uses a lightweight sound source separation model to separate the voice-based questions into pure human voice and background sound. The pure human voice is then processed using Automatic Speech Recognition (ASR) technology to obtain an ASR text description. This ASR text description, along with the text-based question, is then processed using a tokenizer to obtain a text token (i.e., a text interaction command).

[0075] Furthermore, the backend server encapsulates the parsed video frame stream, audio frame stream, text interaction commands, and device prior metadata into a multimodal data packet in a unified standard format, and adds a timestamp accurate to the millisecond level to each modal data to ensure their complete synchronization on the timeline.

[0076] For example, multimodal data packets are arranged in a circular zero-copy buffer according to timestamps, and the data is pushed to the preprocessing module 2 when the buffer is full.

[0077] Preprocessing module 2 is used to preprocess multimodal data packets, extracting visual embedding vectors of keyframes from video frame streams and audio event tags from audio frame streams, generating cross-modal semantic tensors based on timestamp alignment, providing standardized input for device coarse inspection module 3. For example, for video frame streams, a lightweight VLM-Coarse model is invoked to encode each video frame, obtaining the visual embedding vectors of each video frame. Simultaneously, the model encoder calculates the cosine similarity of the visual embedding vectors of adjacent video frames and combines optical flow changes to determine motion saliency, filtering out keyframes while retaining the image data of the keyframes. For audio frame streams, sound source separation is performed on the Mel spectrum through masking estimation, obtaining pure human voice and device background sounds (such as buzzers, relay crackling, motor noises, etc.). Automatic speech recognition and semantic parsing are performed on the pure human voice to obtain a structured audio separation map, such as information like "device name, error code, user's stated intent," as well as information such as emotions or doubts that may affect diagnosis. This information can be used for subsequent multimodal causal inference. Background noise from the device is detected based on preset audio events, and audio event tags are obtained, such as alarm beeping, relay noise, etc.

[0078] After preprocessing the multimodal data packets, the visual embedding vectors, audio separation spectrograms, audio event tags, and text interaction commands of the keyframes are concatenated into a unified coordinate system to generate a timestamp-aligned cross-modal semantic tensor. This cross-modal semantic tensor is then pushed to the device coarse inspection module 3 using a zero-copy method.

[0079] Device coarse detection module 3 is used to determine whether a preset device exists in the video frame stream based on the visual embedding vectors of keyframes and to determine the time interval in which the preset device exists. For example, this coarse-grained visual language model is a lightweight visual language foundation model, abbreviated as VLM-Coarse. The main body of this model consists of a 12-layer visual Transformer and a compressed text query encoder. The total number of parameters is controlled at the 90M level, achieving a single-frame inference latency of less than 10ms on edge GPUs or high-performance ARM processors.

[0080] The purpose of determining whether a preset device exists in a video frame stream is to quickly filter out non-target scenes and save computational resources. This process uses a "zero-sample matching" paradigm. During the model loading phase, the system injects multiple preset device category hints, such as "EP500", "AC240L", and "power battery pack", which are stored in the query pool as device category query vectors. At runtime, the cosine similarity between the visual embedding of a keyframe and the device category query vector in the query pool in the shared semantic space is calculated. When the maximum similarity exceeds a preset similarity threshold, the keyframe is determined to "have a preset device", and is then labeled with the upload ID, frame number, and confidence score and written to the inference cache.

[0081] The fault location module 4 utilizes a pre-trained fine-grained visual language model to fuse image data and text interaction commands from key frames within a preset time interval where the device exists through cross-attention, outputting fault location information. For example, this fine-grained visual language model is a fine-tuned visual language model, abbreviated as VLM-Fine. Unlike the aforementioned lightweight coarse-grained visual language model, VLM-Fine's visual trunk retains a 24-layer multi-head attention structure at a 32×32 image block resolution, while the language trunk is loaded with 7B-level parameter weights for further pre-training specific to the device domain. Taking energy storage as an example, VLM-Fine's language trunk can understand long-tail technical terms such as "DC bus voltage indicator," "inverter cooling fan," and "terminal erosion marks."

[0082] After the device coarse inspection module 3 outputs the preset time interval for the device's existence, the system inputs the image data of keyframes within that time interval and the text interaction commands into the fine-grained visual language model. This fine-grained visual language model completes the fine fault localization through sparse query and dense regression. For example, the fine-grained visual language model generates a fault query vector based on the text interaction commands and stores it in a query pool. The language backbone obtains the fault semantic label most similar to the fault query vector. The visual backbone obtains a multi-scale feature map based on the keyframe image data and performs cross-attention fusion with the multi-scale feature map and the fault query vector to generate sparse interest points. Subsequently, the interest points are fine-tuned in coordinates using a bounding box regressor, and a multi-channel hotspot heatmap header is simultaneously set to generate a pixel-level fault saliency mask. Finally, fault localization information including the device bounding box, fault hotspot mask, fault semantic label, fault location confidence, and timestamp is obtained.

[0083] The confidence gating module 5 is used to generate a comprehensive confidence score based on the confidence scores of the device bounding box, fault semantic labels, and audio event labels, combined with prior device metadata, through Bayesian calibration and semantic consistency verification. Based on the comprehensive confidence score and dynamic thresholds, fault clue information (including fault location information and audio event labels) is categorized into high-confidence and low-confidence types. When fault clue information is marked as high-confidence, it is pushed to the cross-modal inference module 6; when it is marked as low-confidence, it is written to the bypass channel along with a reason code. The bypass channel, by default, first triggers a FAQ (Frequently Asked Questions) retrieval to provide immediate responses to common false alarms or missing information. If the FAQ still cannot be matched, the system automatically generates a semi-structured work order and pushes it, along with all original multimodal data packets, to human customer service.

[0084] The cross-modal reasoning module 6 is used to construct a multimodal temporal semantic graph based on fault location information, audio event tags, and text interaction instructions. It uses a cross-modal Transformer and a causal reasoning big language model to perform chain reasoning and generate structured fault diagnosis data.

[0085] For example, the fault location information, including the device bounding box, fault hotspot mask, fault semantic labels, audio event labels, and text interaction instructions, is first mapped onto a unified frame sequence according to timestamps. In each frame, a multimodal node containing visual segments, text interaction instructions, and audio events is constructed. Then, adjacency relationships between nodes are established based on time windows and fault semantic labels, thus forming a multimodal temporal semantic graph. In this way, the time axis is upgraded to a cross-modal graph structure, solving the problem of difficulty in complementing single-modal features in traditional methods. The aligned multimodal temporal semantic graph is flattened into a sequence and input into a cross-modal Transformer. The encoder side of the cross-modal Transformer employs multi-head cross-attention, allowing visual, text, and acoustic queries to interact with each other. The decoder side is supplemented with a multi-task classification head, predicting ternary classification results for fault type, fault cause, and fault severity, respectively. To compensate for long-range dependencies that may be missed in a single Transformer test, after obtaining the ternary classification results, the system calls LLM-Causal to perform chained multimodal causal reasoning, generating structured fault diagnosis data that includes fault phenomena, fault causes, and solutions.

[0086] The knowledge retrieval module 7 is used to retrieve and match maintenance solutions from a pre-built knowledge database based on the fault diagnosis structured data and equipment prior metadata output by the cross-modal reasoning module 6. For example, firstly, a sparse query vector is constructed based on the fault diagnosis structured data and equipment prior metadata; then, a search is performed in the pre-built knowledge database based on the sparse query vector to obtain a set of candidate entries; finally, a knowledge graph inference engine is used to filter out the candidate entries that best match the fault diagnosis structured data from the candidate entry set, and a maintenance solution is obtained based on this candidate entry. This maintenance solution includes maintenance steps, required tools, estimated working hours, safety warnings, etc. Further, the knowledge retrieval module 7 organizes the fault diagnosis structured data, maintenance solution, cross-modal reasoning link, and confidence vector into knowledge retrieval structured data and outputs it to the diagnosis generation module 8.

[0087] The diagnostic generation module 8 is used to process the structured knowledge retrieval data output by the knowledge retrieval module 7 through a text generator and an annotation overlay device, and generate a diagnostic report that integrates visual annotations and structured text descriptions.

[0088] In one embodiment, the intelligent diagnostic question-answering system further includes a FAQ / human assistance module and a continuous learning closed-loop module. The FAQ / human assistance module is used to triage low-confidence or abnormally processed fault clue information through an intent splitter, retrieve answers from the FAQ corpus based on classification confidence, or direct the information to a human work order channel. This low-confidence or abnormally processed fault clue information includes fault clue information marked as low-confidence by the confidence gating module 5, fault clue information for which a repair solution cannot be matched in the knowledge retrieval module 7, and fault clue information for which structured fault diagnosis data cannot be generated in the cross-modal reasoning module 6 due to semantic conflicts or other reasons. The continuous learning closed-loop module is used to collect low-confidence clues and manually reviewed samples, and after semi-automatic annotation, update the visual language model, cross-modal reasoning model, and knowledge graph through low-rank fine-tuning technology to achieve adaptive evolution of the system.

[0089] Please refer to Figure 2 , Figure 2 A flowchart of a visual diagnostic method for equipment anomalies is shown, which includes:

[0090] Step S201: Obtain a video containing the on-site operating status of the device, text consultation questions associated with the video, and prior metadata of the device, and generate a unified encapsulated multimodal data packet with a timestamp.

[0091] In one embodiment, the edge SDK deployed on the device side of the intelligent diagnostic question-and-answer system acquires video footage of the device's operating status recorded or captured by the user using a camera device such as a mobile phone or tablet. It then initiates a segmented-resume upload process to upload the video to the backend server. Upon detecting the uploaded video, the backend server generates a unique upload ID for the video data and simultaneously controls the device to display a structured form. This form retrieves data such as the device model, serial number (SN), firmware version, and alarm code provided by the user, forming prior metadata for the device. This prior metadata is then marked with the same upload ID.

[0092] After receiving the uploaded video, the backend server uses FFmpeg to encapsulate the uploaded video into a standard container format, separates the audio track from the video, extracts video frames and audio frames at a rate of 1 frame per second, and appends the corresponding timestamp to the extracted video frames and audio frames.

[0093] In one embodiment, the edge SDK also acquires text-based or voice-based inquiries from users (e.g., "Why does this inverter keep beeping?"), uploads them along with video to the backend server, and assigns the same upload ID to each inquiry. For voice-based text inquiries, the backend server uses a lightweight sound source separation model to separate the voice inquiries into pure human voice and background sound. The pure human voice is then processed using automatic speech recognition technology to obtain an ASR text description. This ASR text description, along with the text-based inquiries, is then processed using a tokenizer to obtain text interaction instructions.

[0094] Furthermore, the backend server encapsulates the parsed video frame stream, audio frame stream, text interaction commands, and device prior metadata into a multimodal data packet in a unified standard format, and adds a timestamp accurate to the millisecond level to each modal data to ensure complete synchronization of video frames, audio frames, and text interaction commands on the timeline.

[0095] Step S202: Preprocess the multimodal data packets by extracting the visual embedding vectors of keyframes from the video frame stream of the multimodal data packets, extracting audio event tags from the audio frame stream of the multimodal data packets, and generating a cross-modal semantic tensor based on timestamp alignment.

[0096] In one embodiment, upon receiving a multimodal data packet containing video and audio frame streams, the VLM-Coarse model is first invoked to encode each video frame, obtaining the visual embedding vector for each video frame. Simultaneously, the model encoder calculates the cosine similarity of the visual embedding vectors of adjacent video frames and, combined with optical flow changes, determines motion saliency, selecting keyframes. The image data of these keyframes is also retained.

[0097] In step S201, the audio frame stream is segmented according to a time window equal to that of the video frame stream. For each segmented audio frame, a dual-branch time-frequency masking network based on the Demucs-v3 structure is used to perform source separation on the Mel spectrum through masking estimation, resulting in clean human voice and device background noise. Automatic speech recognition and semantic parsing are performed on the clean human voice (e.g., streaming semantic parsing via a locally hosted LLM-Edge) to obtain a structured audio separation map. Device background noise is detected based on preset audio events to obtain audio event labels. Taking an energy storage scenario as an example, in the background noise channel, the system deploys an audio event detection network specifically trained for energy storage scenarios. Its front end uses a unified Mel spectrum, and multi-head attention is introduced at the top layer to focus on narrowband high-energy events such as buzzers, relay crackling, and motor noises. When the confidence level of a certain type of event exceeds a threshold, the model immediately records the audio event label on the timeline.

[0098] After preprocessing the multimodal data packets, the visual embedding vectors, audio separation spectra, audio event tags, and text interaction instructions of the keyframes are concatenated into a unified coordinate system to generate a cross-modal semantic tensor based on timestamp alignment.

[0099] In this step, the preprocessing of video and audio frame streams is carried out in parallel, which can greatly reduce the preprocessing delay and keep the cross-modal error within a small range. This lays a semantically consistent, temporally consistent, and noise-controlled input foundation for subsequent equipment inspection, precise positioning, and cross-modal reasoning.

[0100] Step S203: Based on the visual embedding vector of the key frame, use a pre-trained coarse-grained visual language model to determine whether a preset device exists in the video frame stream, and determine the time interval in which the preset device exists.

[0101] In one embodiment, the coarse-grained visual language model is a lightweight visual language foundation model, abbreviated as VLM-Coarse. The main body of this model consists of a 12-layer visual Transformer and a compressed text query encoder. The total number of parameters is controlled at the 90M level, enabling single-frame inference latency of less than 10ms on edge GPUs or high-performance ARM processors.

[0102] One method for determining the presence of a preset device in a video frame stream is as follows: Device category text hints (such as "EP500", "AC240L", "power battery pack", etc.) are injected during the loading of the coarse-grained visual language model. These hints are stored in a query pool as text vectors. The coarse-grained visual language model embeds and edits the device category text hints using a text editor to generate device category query vectors. During runtime, the coarse-grained visual language model calculates the similarity between the visual embedding vector of a keyframe and the device category query vector in the shared semantic space. When the maximum similarity exceeds a preset similarity threshold, the system determines that a preset device exists in that keyframe. When multiple consecutive frames exceed the similarity threshold, the system generates a time interval on the timeline indicating the device's presence. When the coarse-grained visual language model does not detect any keyframes exceeding the similarity threshold in the entire video frame stream, it directly triggers a bypass, transferring the work order to the FAQ / human channel to avoid wasting subsequent computational resources.

[0103] In one embodiment, in order to further reduce the computing power overhead at the edge, a two-level early stop mechanism is set when determining whether a preset device exists in the video frame stream. Specifically: (1) If the Hamming distance of the visual hash fingerprint of two adjacent frames is lower than a certain threshold, the system regards it as a static picture and automatically jumps to the next dynamic segment where the content of the next picture changes significantly before starting detection; (2) If the frames within a consecutive preset time period do not reach the preliminary similarity threshold (which is lower than the aforementioned similarity threshold), the inference thread enters a light sleep and continues to infer only the first frame per second until a suspected target is detected again.

[0104] In one embodiment, the coarse-grained visual language model is pre-trained from a visual language model with a higher number of parameters using a "distillation technique". The training steps are as follows: (1) Select a visual language model (e.g., Stable Diffusion 3.5) as the base model; (2) Distill on an industry dataset. Taking energy storage equipment as an example, the images of energy storage equipment in the dataset are all labeled with energy storage-specific terms, which may include professional terms such as "PCS", "BMS", and "EMS". Based on the above model, the model is further fine-tuned. For example, the private dataset includes images of key parts such as indicator lights, displays, and terminals, as well as corresponding specialized terminology labels, in order to construct an attention mechanism for perceiving energy storage status and external structure.

[0105] Step S204: Using a pre-trained fine-grained visual language model, the image data of key frames within the time interval and the text interaction instructions obtained based on text consultation questions are fused through cross-attention to output fault location information.

[0106] In one embodiment, the fine-grained visual language model is a fine-tuned visual language model, abbreviated as VLM-Fine. VLM-Fine achieves precise fault localization through two steps: sparse query and dense regression. First, the fine-grained visual language model generates fault query vectors based on text interaction commands and stores them in a query pool. The language backbone obtains the most similar fault semantic labels (e.g., alarm lights, inverter fans, etc.) based on the fault query vectors. The visual backbone obtains multi-scale feature maps based on keyframe image data, and performs cross-attention fusion with the multi-scale feature maps and fault query vectors to generate sparse interest points. Subsequently, the interest points are fine-tuned in coordinates by a bounding box regressor, and a multi-channel hotspot heatmap header is simultaneously set to generate a pixel-level fault saliency mask. For example, the bounding box regressor adopts an anchor-free design, directly predicting the four-boundary offset and center confidence, and is equipped with a 16×16 channel hotspot heatmap header for generating a pixel-level fault saliency mask. The heatmap header is jointly trained with the bounding box regressor using KL-divergence loss, thereby obtaining the device bounding box and fault hotspot mask simultaneously in the same forward pass.

[0107] To ensure the consistency and accuracy of visual localization results, the system performs Non-Maximum Suppression (NMS) on bounding boxes with confidence scores below a preset confidence threshold after inference and checks back the visual hash fingerprints from the coarse detection stage (i.e., the stage of determining whether a preset device exists in the video frame stream). If it finds that adjacent frames on the same timeline have highly similar images but yield significantly different localization results, a self-supervised consistency regularization is triggered, and the predicted values ​​of the two frames are averaged and written back to the cache. This approach can eliminate occasional drift under conditions of low-light flicker or motion blur. Finally, the fine-grained visual language model produces fault localization information including device bounding boxes, fault hotspot masks, fault semantic labels, fault location confidence scores, and timestamps.

[0108] Step S205: Construct a multimodal temporal semantic graph based on fault location information, audio event tags, and text interaction instructions. Use cross-modal Transformer and causal reasoning big language model to perform chain reasoning to generate structured fault diagnosis data.

[0109] For example, a multimodal temporal semantic graph is first constructed based on fault location information, audio event labels, and text interaction instructions. Specifically, the fault location information, including the device bounding box, fault hotspot mask, and fault semantic labels, as well as the audio event labels and text interaction instructions, are first mapped onto a unified frame sequence according to timestamps. In each frame, a multimodal node containing visual segments, text interaction instructions, and audio events is constructed. Then, adjacency relationships between nodes are established based on time windows and fault semantic labels (e.g., nodes with adjacent time windows are automatically connected, and nodes with the same source fault are automatically connected). In this way, the time axis is upgraded to a cross-modal graph structure, solving the problem of the difficulty in complementing single-modal features in traditional methods.

[0110] Furthermore, the aligned multimodal temporal semantic graph is flattened into a sequence and input into a cross-modal Transformer. The encoder side of this cross-modal Transformer employs multi-head cross-attention, allowing visual, textual, and acoustic information to interact with each other. The decoder side is supplemented with a multi-task classification head, predicting ternary classification results for fault type, fault severity, and fault cause. To maintain the robustness of the cross-modal Transformer in scenarios with unknown firmware versions, pre-defined fault video logs are used as the label source during model training, and soft label distillation is introduced. After the cross-modal Transformer encoding-decoding is completed, the ternary classification results for fault type, fault cause, and fault severity, along with their confidence vectors, are obtained.

[0111] To compensate for long-range dependencies that might be missed in a single Transformer iteration, the system calls LLM-Causal for chained multimodal causal inference after obtaining the ternary classification result, as follows:

[0112] (1) The fault semantic label, structured label, audio event label and Transformer confidence vector are concatenated into a natural language prompt. The structured label is obtained by performing tone and keyword-triggered event detection and semantic understanding on the pure human voice separated from the text consultation question in speech form and the pure human voice separated from the audio frame stream.

[0113] (2) Generate the first round of "hypothesis-reason" pairs using LLM-Causal and output the credibility score;

[0114] (3) Select the top k hypotheses with the highest credibility and quickly match them with the equipment-fault-solution triples in the preset knowledge database. Feed the successfully matched triples back to LLM-Causal as new context, prompting the model to supplement or correct the previous conclusions in the second round of reasoning.

[0115] (4) If the results of the two rounds of reasoning are contradictory in key fields (equipment components, alarm codes, causal order), the system uses a proof-refutation chain to allow the model to provide an explanation and automatically selects the conclusion with the highest information gain.

[0116] To enhance model interpretability, the above reasoning process and logic chain will be directly displayed on the user's end. The final conclusion will be output after completing step (4). Through the above chain-like multimodal causal reasoning, the model can integrate the ternary classification results lacking a logic chain into a complete logic chain of fault phenomena, fault causes, and solutions. For example, "alarm sound + red flashing + inverter cooling fan stopped" can be integrated into "fan stuck (fault phenomenon)". Inverter over-temperature protection (cause of failure) DC-side current limiting (solution)".

[0117] In one embodiment, to obtain a more comprehensive and user-friendly output, a repair solution is retrieved and matched from a pre-built knowledge database based on the fault diagnosis structured data and equipment prior metadata output in step S205, and a diagnostic report integrating visual annotations and structured text descriptions is generated. Specifically, a sparse query vector is first constructed based on the fault diagnosis structured data and equipment prior metadata, and then the sparse query vector is projected onto a continuous semantic space through an embedded retrieval engine. The system uses a multi-task distilled encoder to perform cosine matching between the sparse query vector and knowledge entries in the pre-built knowledge database to obtain a candidate entry set. For example, the knowledge database uses an IVF-HNSW two-layer index to store millions of multi-variable relation knowledge entries. For example, the knowledge entry is stored as a five-element relation of "equipment model-component-alarm code-fault phenomenon-solution". Then, the candidate entry set is input into the knowledge graph inference engine for structural consistency verification. In the knowledge graph inference engine, each knowledge graph node covers categories such as device model, hardware component, symptoms, alarm codes, repair steps, and risk warnings. Edge types are encoded as relationships such as belonging, causing, mitigating, and replacing. The knowledge graph inference engine searches along the association edges of candidate entries for the subgraph with the highest isomorphism to the fault diagnosis structured data. If a match is found, the repair steps, time assessments, and safety warnings in the matching subgraph are extracted and written to the diagnostic cache as formal search results. If no match is found, zero-sample completion logic is triggered, and LLM-Causal generates suggested text based on the closest solution and marks it as low-confidence pending manual review. Through a three-stage process of index hot routing, vector retrieval, and graph verification, a rapid closed loop of knowledge-driven processing is achieved. After obtaining the repair solution, the system organizes the fault diagnosis structured data, repair solution, cross-modal inference links, and confidence vectors into knowledge retrieval structured data output.

[0118] Furthermore, a text generator and annotator overlay are used to process the knowledge retrieval structured data to generate a diagnostic report that integrates visual annotations and structured text descriptions. For example, the knowledge retrieval structured data is embedded into the prompt template of the large language model in a five-segment format of "problem restatement - fault location - cause explanation - step-by-step operation - risk reminder". The large language model generates structured text descriptions based on this format. The annotation overlay back to the fault hotspot mask and device bounding box output by VLM-Fine on the time axis and copies a transparent canvas on the original video based on the key frame. In one embodiment, the system automatically selects the annotation style according to the confidence level of fault location: (1) when the confidence level is higher than the threshold, a solid red box is used and "fault hotspot" transparent coloring is overlaid; (2) when the confidence level is between the high and low thresholds, a dashed yellow box is used and "please check" prompt is attached; (3) when the confidence level is lower than the low threshold, no annotation is made and only the ambiguous reason is explained in the text. Finally, the keyframe with the highest confidence level is selected as the static cover, and a device bounding box and hotspot mask are superimposed on it; the structured text description in five-segment format is split into short labels, corresponding to the screen origin number, and the full description pops up when clicked.

[0119] In one embodiment, after step S204 and before step S205, the following steps are included: Normalizing the confidence scores of the device bounding box, fault semantic labels, audio event labels, and device prior metadata, and then feeding them into a lightweight Bayesian calibration network for calibration and semantic consistency verification, generating a comprehensive confidence score ranging from 0 to 1. Then, based on the comprehensive confidence score and a dynamic threshold, the fault clue information is divided into high-confidence and low-confidence types. This dynamic threshold varies based on scene temperature, noise environment, and shooting duration. When the fault clue information is of the high-confidence type, step S205 is executed; when the fault clue information is marked as low-confidence, the fault clue information, along with a cause code, is written into the bypass channel. The bypass channel, by default, first triggers a FAQ retrieval to provide immediate responses to common misshooting or missing information; if the FAQ still cannot be matched, the system automatically generates a semi-structured work order and pushes it, along with all original multimodal data packets, to customer service. By relying on this gating process based on Bayesian correction and dynamic thresholds, this application reduces the false trigger rate to below 2.7% without sacrificing diagnostic recall, while maintaining the proportion of work orders that truly require manual intervention in the range of 8-12%, significantly reducing the manpower cost of 24 / 7 after-sales support.

[0120] Please refer to Figure 3 , Figure 3A schematic diagram of the main business process for visual diagnosis of equipment anomalies is shown. When no preset equipment is detected in the video frame stream, when the comprehensive confidence of the fault clue information does not reach the dynamic threshold T, and when no repair solution is matched in the knowledge base retrieval, the FAQ / human channel will be entered. In one embodiment, the FAQ / human channel first calls the lightweight LLM-Router to perform joint semantic encoding on the text interaction instructions, the prior metadata of the equipment, and the failure reason codes of each front-end model, and then sends the encoding result to the four-class intent splitter: (1) Clearly define the solution to the equipment fault; (2) Consultation on general usage methods; (3) Consultation on after-sales process and warranty policy; (4) Irrelevant equipment or suspected false shooting. The splitter performs the following actions according to the classification confidence: if the confidence is greater than a certain threshold (e.g., 0.85), the FAQ corpus is directly retrieved and the answer is returned; otherwise, the human work order channel is entered. Through the above design, the FAQ / human assistance unit can ensure that common problems are solved with one click, while also achieving rapid triage and work order closure for difficult and low-confidence samples. It does not waste model computing resources and ensures that human input is accurately placed in the most needed situations, providing a last reliable guarantee for the entire multimodal fault diagnosis system.

[0121] In other embodiments, the system also collects low-confidence clues and manually verified samples, which are then semi-automatically labeled and updated using low-rank fine-tuning technology to achieve adaptive evolution of the system.

[0122] The device anomaly visual diagnosis method provided in this application first acquires video containing the device's operating status, textual consultation questions associated with the video, and prior metadata of the device, generating a unified encapsulated multimodal data packet with a timestamp. Then, it extracts time-aligned cross-modal semantic features from the data packet and uses a pre-trained visual language model to perform two-level processing on key frames in the video frame stream: first, it quickly identifies and locates the target device; then, it performs fine-grained analysis on the time period in which the device appears to output accurate fault location information. Finally, it fuses fault location information, audio event tags, and text interaction commands to construct a multimodal temporal semantic graph and drives a cross-modal Transformer and a large language model to perform chain-like causal reasoning, generating structured fault diagnosis data. This achieves end-to-end intelligent diagnosis from multimodal perception to deep causal analysis. The method in this application quickly filters out non-target scenes using a lightweight model and utilizes multimodal parallel processing and chained reasoning to shorten the diagnostic response time from hours to minutes or even seconds, achieving efficient real-time online diagnosis 24 / 7. By integrating visual, audio, and textual multimodal information and performing multi-level processing such as coarse detection, fine localization, and cross-modal reasoning, the accuracy of fault diagnosis is significantly improved and the false alarm rate is effectively reduced.

[0123] According to an embodiment of this application, an electronic device is provided, such as... Figure 4The diagram shown is a hardware structure schematic of an electronic device according to an embodiment of this application. The electronic device 100 includes a processor 10, a memory 20, and a communication interface 30. The processor 10, memory 20, and communication interface 30 are connected via lines. Figure 4 In the embodiment shown, the processor 10, memory 20, and communication interface 30 are connected to each other via a bus.

[0124] The memory 20 is used to store software programs, computer-executable program instructions, etc. The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and application programs required for at least one function; the data storage area may store data created based on the use of the electronic device 100, etc.

[0125] The memory 20 can be a read-only memory (ROM), or other types of static storage devices that can store static information and instructions, or random access memory (RAM), or other types of dynamic storage devices that can store information and instructions, or electrically erasable programmable read-only memory (EEPROM). The specific type is not limited here.

[0126] For example, the aforementioned memory 20 can be Double Data Rate Synchronous Dynamic Random Access Memory (DDR SDRAM). This memory 20 can exist independently but is connected to the processor 10. Optionally, the memory 20 can also be integrated with the processor 10, for example, integrated within one or more chips.

[0127] In some embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and this remote memory may be connected to the electronic device 100 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0128] The processor 10 connects various parts of the electronic device 100 using various interfaces and lines. By running or executing software programs stored in the memory 20 and calling data stored in the memory 20, it performs various functions of the electronic device 100 and processes data, such as implementing the methods described in any embodiment of this application.

[0129] The processor 10 can be a field programmable gate array (FPGA), a digital signal processor (DSP), a central processing unit (CPU), or the like.

[0130] Processor 10 can be a single-core processor or a multi-core processor. For example, processor 10 can be composed of multiple FPGAs or multiple DSPs. Furthermore, processor 10 can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions). Processor 10 can be a standalone semiconductor chip or integrated with other circuits into a single semiconductor chip. For example, it can form a system-on-a-chip (SoC) with other circuits (such as encoding / decoding circuits, hardware acceleration circuits, or various bus and interface circuits), or it can be integrated as a built-in processor within an application-specific integrated circuit (ASIC). This ASIC with integrated processor can be packaged separately or together with other circuits.

[0131] The communication interface 30 can use a transceiver device, such as a transceiver, to enable communication between the electronic device 100 and other devices or communication networks.

[0132] This application also provides a non-volatile computer-readable storage medium storing computer-executable instructions that are executed by one or more processors, for example, to perform the steps of the device anomaly visual diagnosis method described above.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them; under the concept of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of this application as described above, which are not provided in detail for the sake of brevity; although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for visual diagnosis of equipment malfunctions, characterized in that, The method includes: Step S1: Obtain a video containing the on-site operating status of the device, text consultation questions associated with the video, and prior metadata of the device, and generate a unified encapsulated multimodal data packet with a timestamp; Step S2: Preprocess the multimodal data packet, extract the visual embedding vector of the key frame from the video frame stream of the multimodal data packet, extract the audio event tag from the audio frame stream of the multimodal data packet, and generate a cross-modal semantic tensor based on timestamp alignment. Step S3: Based on the visual embedding vector of the key frame, use a pre-trained coarse-grained visual language model to determine whether a preset device exists in the video frame stream, and determine the time interval in which the preset device exists. Step S4: Using a pre-trained fine-grained visual language model, the image data of key frames located within the time interval and the text interaction instructions obtained based on the text consultation question are fused through cross-attention to output fault location information; Step S5: Construct a multimodal temporal semantic graph based on the fault location information, the audio event tags, and the text interaction instructions; use a cross-modal Transformer and a causal reasoning large language model to perform chain reasoning to generate structured fault diagnosis data.

2. The method according to claim 1, characterized in that, The extraction of keyframe visual embedding vectors from the video frame stream of the multimodal data packets includes: A lightweight visual language model is used to encode each video frame of the multimodal data packet, and the visual embedding vector of each video frame is output. Calculate the cosine similarity of the visual embedding vectors of adjacent video frames, and combine it with optical flow changes to determine motion saliency, thereby selecting keyframes.

3. The method according to claim 1, characterized in that, Extracting audio event tags from the audio frame stream of the multimodal data packets includes: The audio frame stream of the multimodal data packet is subjected to source separation to obtain pure human voice and device background sound; The background noise of the device is detected based on preset audio events, and audio event tags are obtained.

4. The method according to claim 1, characterized in that, The method of determining whether a preset device exists in the video frame stream using the visual embedding vector based on the keyframes and a pre-trained coarse-grained visual language model includes: A device category text hint is set for the coarse-grained visual language model, and the coarse-grained visual language model generates a device category query vector based on the device category text hint; The visual embedding vector of the keyframe is input into the coarse-grained visual language model. The coarse-grained visual language model calculates the similarity between the visual embedding vector of the keyframe and the device category query vector in the shared semantic space. When the maximum similarity exceeds a preset similarity threshold, it is determined that a preset device exists in the keyframe.

5. The method according to claim 1, characterized in that, Step S4 includes: The text interaction instructions in the multimodal data packet are input into the fine-grained visual language model, and the fine-grained visual language model generates a fault query vector based on the text interaction instructions and stores it in the query pool; Image data of key frames located within the time interval are input in batches into the fine-grained visual language model. The fine-grained visual language model obtains multi-scale feature maps based on the image data of the key frames. The multi-scale feature maps are then fused with the fault query vector through cross-attention to generate sparse interest points. The points of interest are fine-tuned in coordinates using a bounding box regressor, and a multi-channel hotspot heatmap header is set simultaneously to generate a pixel-level fault saliency mask, resulting in fault location information including device bounding boxes, fault hotspot masks, fault semantic labels, fault location confidence, and timestamps.

6. The method according to claim 5, characterized in that, After step S4 and before step S5, the following steps are also included: Based on the confidence scores of the device bounding box, the fault semantic labels, the audio event labels, and the device prior metadata, a comprehensive confidence score is generated by fusing Bayesian calibration and semantic consistency verification. Based on the comprehensive confidence level and dynamic threshold, the fault clue information is divided into high-confidence type and low-confidence type, and the fault clue information includes the fault location information and the audio event tag; When the fault clue information is of the high confidence type, proceed to step S5.

7. The method according to any one of claims 1 to 6, characterized in that, Following step S5, the following is also included: Step S6: Based on the fault diagnosis structured data and the equipment prior metadata, a repair solution is retrieved and matched in the pre-built knowledge database, and a diagnostic report integrating visual annotations and structured text descriptions is generated.

8. The method according to claim 7, characterized in that, Step S6 includes: Based on the fault diagnosis structured data and the equipment prior metadata, a sparse query vector is constructed. Based on the sparse query vector, a search is performed in a pre-constructed knowledge database to obtain a set of candidate entries; The knowledge graph inference engine is used to filter out the candidate entries from the candidate entry set that best match the fault diagnosis structured data, and a maintenance plan is obtained based on the candidate entries.

9. An electronic device, characterized in that, The method includes at least one processor and a memory communicatively connected to the at least one processor, the memory storing instructions executable by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1 to 8.

10. A computer storage medium, characterized in that, The computer storage medium stores instructions or programs that, when executed by at least one processor, cause the at least one processor to perform the method as described in any one of claims 1 to 8.

Images