Device fault diagnosis method and device, electronic device, and storage medium

Abstract

The application discloses a device fault diagnosis method and device, electronic equipment and storage medium, and relates to the technical field of fault diagnosis, to solve the problem that related technologies reduce the accuracy of fault diagnosis. The method comprises the following steps: performing semantic analysis on language data representing device faults, and generating a semantic confidence according to the semantic analysis result; in the case where the semantic confidence is greater than a first threshold, acquiring image data of the device, and positioning a target area in the image data according to a fault object in the semantic analysis result; performing cross-modal matching between the visual features of the target area and the semantic analysis result to obtain a visual confidence, and calculating a target confidence according to the semantic confidence and the visual confidence; and if the target confidence is greater than a second threshold, it is determined that the device has a fault.

Description

Technical Field

[0001] This application relates to the field of fault diagnosis technology, and in particular to a method, apparatus, electronic device and storage medium for diagnosing equipment faults. Background Technology

[0002] In the field of equipment fault diagnosis, related technologies disclose a scheme for multimodal diagnosis through language description and equipment images, including: extracting fault information from language description through natural language processing, identifying abnormal regions in equipment images through object detection models, fusing the fault information extracted from the language modality with the abnormal regions detected by the image modality (such as feature splicing, weighted summation or multimodal attention mechanism), and finally determining whether the equipment has a fault based on the fused features.

[0003] However, in actual repair scenarios, the quality of multimodal data submitted by users varies greatly. For example, verbal descriptions may be vague, ambiguous, or lack key information, while image data may have unclear abnormal areas or artifacts due to factors such as shooting angle, lighting conditions, and occlusion. Directly fusing multimodal data will mix noisy features from low-quality modalities with effective features from high-quality modalities, resulting in information interference in the fused features and thus reducing the accuracy of fault diagnosis. Summary of the Invention

[0004] This application provides a method, apparatus, electronic device, and storage medium for diagnosing equipment faults, which can solve the problem that related technologies reduce the accuracy of fault diagnosis.

[0005] In a first aspect, this application provides a method for diagnosing equipment faults, the method comprising: Semantic analysis is performed on the language data characterizing equipment failure, and semantic confidence is generated based on the semantic analysis results, wherein the language data includes speech data and / or text data; If the semantic confidence level is greater than a first threshold, image data of the device is acquired, and the target area in the image data is located based on the faulty object in the semantic analysis result. The visual features of the target region are matched with the semantic analysis results across modalities to obtain visual confidence, and the target confidence is calculated based on the semantic confidence and the visual confidence. If the target confidence level is greater than the second threshold, then the device is determined to be faulty.

[0006] The above method first performs semantic analysis on the language data and generates semantic confidence scores. Only when the confidence score exceeds a first threshold is the reliable semantic analysis result used to guide the localization of the target region in the image, thereby filtering out low-quality language descriptions that are vague, ambiguous, or lack information, and improving the efficiency of computing resource utilization. Next, the visual features of the target region are matched with the semantic analysis results across modalities to obtain visual confidence scores, which are then fused with semantic confidence scores to calculate the target confidence score, realizing a progressive fault discrimination from single-modal initial screening to multi-modal joint verification. Only when the target confidence score is higher than a second threshold is the equipment fault finally confirmed, effectively reducing the risk of false alarms and false negatives, and significantly improving the accuracy and robustness of fault detection.

[0007] In some embodiments, when the language data includes speech data, the step of performing semantic analysis on the language data to generate semantic confidence includes: The speech data is converted into text data, and the environmental noise energy value is extracted from the speech data; Semantic extraction is performed on the converted text data to obtain fault semantic vectors; Calculate the similarity between the fault semantic vector and each preset fault description vector in the preset fault description library, wherein the preset fault description library includes at least one preset fault description and a corresponding preset fault description vector; The highest similarity obtained is used as the matching degree between the fault semantic vector and the preset fault description library; The semantic confidence score is calculated based on the matching degree and the environmental noise energy value, wherein the environmental noise energy value is negatively correlated with the semantic confidence score.

[0008] In some embodiments, the step of performing cross-modal matching between the visual features of the target region and the semantic analysis results to obtain visual confidence includes: Extract the visual features of the target region and map the visual features to a target feature vector with the same dimension as the fault semantic vector in the semantic analysis result; The cross-modal similarity between the target feature vector and the fault semantic vector is calculated to obtain the visual confidence score.

[0009] In some embodiments, after determining that the device is faulty, the method further includes: The time-varying impact coefficient and safety risk value of the fault are obtained, wherein the time-varying impact coefficient increases with the increase of the fault residence time, and the safety risk value is used to characterize the spatial impact range of the fault; The target confidence level, the time-varying influence coefficient, and the safety risk value are weighted and fused to obtain the fault level. Based on the fault level, a repair work order for the fault is generated.

[0010] In some embodiments, generating a repair work order for the fault based on the fault level includes: The priority of the work order is determined based on the numerical range of the fault level. The location description of the faulty object in the semantic analysis results is converted into spatial coordinates, and the target maintenance department corresponding to the faulty object is determined. The maintenance work order is generated based on the work order priority, the spatial coordinates, and the target maintenance department.

[0011] In some embodiments, determining the work order priority based on the numerical range of the fault level includes: If the fault level is greater than the first level threshold, then the work order priority is determined to be high priority; If the fault level is greater than the second level threshold and less than or equal to the first level threshold, then the work order priority is determined to be medium priority. If the fault level is less than or equal to the second level threshold, then the work order priority is determined to be low priority.

[0012] In some embodiments, the language data is generated in the following manner: The inspection visual data of the device is acquired, and anomaly detection is performed on the inspection visual data, wherein the visual data includes inspection image data and / or inspection video data; If an anomaly is detected in the inspection visual data, the language data is generated based on the anomaly.

[0013] Secondly, this application provides a device for diagnosing equipment faults, the device comprising: A generation module is used to perform semantic analysis on language data representing equipment failures and generate semantic confidence based on the semantic analysis results, wherein the language data includes speech data and / or text data; The positioning module is used to acquire image data of the device when the semantic confidence is greater than a first threshold, and to locate the target area in the image data based on the fault object in the semantic analysis result. The matching module is used to perform cross-modal matching between the visual features of the target region and the semantic analysis results to obtain visual confidence, and to calculate the target confidence based on the semantic confidence and the visual confidence. The fault diagnosis module is used to determine that the device is faulty if the target confidence level is greater than a second threshold.

[0014] Thirdly, this application provides an electronic device, comprising: Memory, used to store computer programs; When the processor executes the computer program stored in the memory, it implements the device fault diagnosis method of the first aspect described above.

[0015] Fourthly, this application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the device fault diagnosis method of the first aspect described above.

[0016] Fifthly, this application provides a computer program product comprising: computer program code, which, when executed on a computer, causes the computer to perform the device fault diagnosis method described in the first aspect.

[0017] The technical effects of each of the second to fifth aspects mentioned above, as well as the technical effects that may be achieved by each aspect, are described above with reference to the technical effects that may be achieved by the various possible solutions in the first aspect, and will not be repeated here. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the system architecture applicable to the embodiments of this application; Figure 2 A flowchart illustrating a device fault diagnosis method provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a device for diagnosing equipment faults provided in an embodiment of this application; Figure 4 This is a schematic diagram of an electronic device structure provided in an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The specific operational methods in the method embodiments can also be applied to the device embodiments or system embodiments. It should be noted that in the description of this application, "multiple" is understood as "at least two". "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing together, or B existing alone. A connected to B can represent: A and B directly connected, or A and B connected through C. Furthermore, in the description of this application, terms such as "first" and "second" are used only for distinguishing the purpose of description and should not be construed as indicating or implying relative importance or order.

[0020] The following is a brief introduction to the system architecture to which the technical solution of this application is applicable. It should be noted that the system architecture described below is for illustrative purposes only and not for limitation. In specific implementations, the technical solution provided in this application can be flexibly applied according to actual needs.

[0021] Figure 1 This is a schematic diagram of the system architecture applicable to the embodiments of this application. The system mainly includes a user layer, a data layer, and a service layer.

[0022] The user layer includes mobile or web-based entry points and an API gateway (Application Programming Interface). Users can transmit multimodal data from the device to the service layer via the API gateway through mobile or web-based devices. This multimodal data includes, but is not limited to, text, voice, image, and video data. It should be noted that text, voice, and image data are typically uploaded when a device malfunctions; while image and video data can also be uploaded at preset intervals, for example, in device inspection scenarios, in addition to uploads during malfunctions.

[0023] The data layer includes an equipment knowledge base, a location mapping library, and a maintenance responsibility matrix library. The equipment knowledge base stores standard status data and historical fault data for equipment components; the location mapping library stores the mapping relationship between semantic locations and spatial coordinates (e.g., building information model coordinates); and the maintenance responsibility matrix library stores the correspondence between equipment types or components and maintenance departments.

[0024] The service layer includes a multimodal processing engine, a location mapping module, and a dispatch decision module, which are used to receive multimodal data from the user layer and execute the equipment repair and / or inspection processes.

[0025] The repair reporting process is as follows: The multimodal processing engine performs semantic analysis on the voice, text, and image data in the multimodal data to determine the fault object and fault phenomenon described by the user, and verifies the authenticity and fault level of the fault in stages. Then, the multimodal processing engine calls the location mapping module to convert the fault object location description in the multimodal data into spatial coordinates by querying the location mapping library (e.g., Information Center 404 is converted to X=120.3, Y=50.1, Z=4.0). Next, the multimodal processing engine calls the dispatch decision module to determine the target maintenance department responsible for the fault object by querying the maintenance responsibility matrix library (e.g., precision air conditioning corresponds to the data center operations and maintenance department). Finally, the multimodal processing engine calls the dispatch decision module to generate a corresponding maintenance work order based on the fault level, spatial coordinates, and target maintenance department, and pushes the maintenance work order to the work order management system via the dispatch decision module, where maintenance personnel from the corresponding department carry out subsequent fault repair work. Simultaneously, the fault object, fault phenomenon, and corresponding maintenance work order are stored in the equipment knowledge base to enrich historical fault data.

[0026] The inspection process is as follows: If image data is received, the multimodal processing engine extracts features from the image data to obtain the status features of the equipment components (such as instrument readings, indicator light colors, mechanical positions, and other visual features). Then, these status features are compared with the standard status data of the equipment components pre-stored in the equipment knowledge base. It should be noted that the standard status data is not the original image, but rather a standard feature template that has been labeled or modeled (such as feature vectors under normal conditions, feature descriptors of reference images, or parameter ranges of components under normal conditions). If the comparison results are consistent (i.e., the similarity between the status features and the standard feature template exceeds a preset threshold), the equipment is determined to be operating normally; if the comparison is inconsistent, the equipment is determined to be operating abnormally, and fault description data containing abnormal features, timestamps, and image data is generated, subsequently automatically triggering the aforementioned repair process.

[0027] If video data is received, the multimodal processing engine first decodes and samples the video to obtain continuous time-series image frames. Then, a temporal anomaly detection algorithm (such as one based on inter-frame difference or temporal convolutional networks) is used to analyze changes in the device's state in the dynamic images to determine if any abnormal behavior exists. If an anomaly is detected (e.g., component movement lag, abnormal temperature fluctuations, instantaneous sparks), the image frame at the time of the anomaly and its temporal context are extracted to generate fault description data containing the fault type, timestamp, and video segment, automatically triggering the aforementioned repair process. Even if no temporal anomaly is detected, a repair request can still be triggered if the static frames in the video do not match the standard state data.

[0028] The technical solution provided in this application will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0029] Figure 2 This is a flowchart illustrating a device fault diagnosis method provided in an embodiment of this application, applied to... Figure 1 Taking the service layer as an example, the process includes the following steps: S201, Perform semantic analysis on the language data representing equipment failure, and generate semantic confidence based on the semantic analysis results; S202, if the semantic confidence is greater than the first threshold, acquire the image data of the device, and locate the target area in the image data based on the fault object in the semantic analysis result; S203, perform cross-modal matching between the visual features of the target region and the semantic analysis results to obtain the visual confidence score, and calculate the target confidence score based on the semantic confidence score and the visual confidence score; S204. If the target confidence level is greater than the second threshold, then the device is determined to be faulty.

[0030] Language data includes speech data and / or text data.

[0031] In this embodiment, when it is determined that the language data includes speech data, the multimodal processing engine first performs speech recognition on the speech data (for example, using the Whisper model optimized for industrial noise) to convert the speech data into text data and extract the environmental noise energy value in the speech data (this energy value can be used to assist in judging the confidence level of the speech).

[0032] Subsequently, the text data obtained from speech recognition and the text data directly input by the user from the language data are jointly fed into the natural language processing model. The natural language processing model can employ BERT (Bidirectional Encoder Representations from Transformers), which extracts semantics from the text, generating a unified structured fault semantic vector. This vector extracts the fault object (including component name and location description) and fault phenomena (such as abnormal noise, excessive temperature, jamming, etc.). Then, this structured fault semantic vector is matched against a pre-defined fault description library, and the semantic confidence is determined based on the matching results. The pre-defined fault description library includes at least one pre-defined fault description and its corresponding pre-defined fault description vector.

[0033] When the semantic confidence score is greater than a first threshold (the specific value can be set according to the application scenario and is not limited here), if the multimodal data uploaded by the user includes image data of the device at the time of the fault, then the image data is acquired, and the target region in the image data is located based on the fault object in the semantic analysis results. For example, the image data is detected by an object detection model, where the object detection model adopts YOLOv8. This model outputs multiple candidate regions and their category labels. The bounding box with the category label of the fault object name and the highest confidence score is determined as the component region, and this component region is segmented from the original image as the target region.

[0034] Therefore, the visual features of the target region are matched with the structured fault semantic vector in the semantic analysis results across modalities to obtain the visual confidence score.

[0035] The target confidence score is calculated based on semantic and visual confidence scores. For example, a first weight for semantic confidence and a second weight for visual confidence are determined, and the semantic and visual confidence scores are weighted and fused according to these weights to obtain the target confidence score. The sum of the first and second weights is 1. The weight settings can be adjusted according to the criticality of the equipment: for non-critical equipment, the first weight can be greater than the second weight, allowing the language modality to play a dominant role in fault response for faster processing of repair requests; for critical equipment, since visual evidence is generally more reliable, the weight of the image modality can be increased, making the second weight greater than the first weight to enhance the accuracy of cross-modal verification. Specific weight values ​​can be set according to the actual application scenario and are not limited here.

[0036] If the target confidence level is greater than the second threshold, the system determines that the device is faulty. The specific value of the second threshold can be set according to the strictness of the system's fault confirmation, and is not limited here.

[0037] If the semantic confidence score is greater than the first threshold but the user-uploaded multimodal data does not include image data, the system will directly use the semantic confidence score as the target confidence score and compare it with the second threshold. If the target confidence score is greater than the second threshold, the system will determine that the device is faulty; otherwise, the system will consider the fault confidence score to be insufficient.

[0038] When the semantic confidence is less than or equal to the first threshold, it indicates that the reliability of the fault described by language alone is low. At this time, the system outputs a prompt that requires supplementary information, such as prompting the user to supplement image data, provide a more detailed fault description, or re-describe the fault phenomenon.

[0039] This embodiment adopts a dynamic phased sequence of language first and then images, which is reasonable based on a comprehensive consideration of human cognitive logic and computational efficiency optimization.

[0040] In real-world equipment repair scenarios, users naturally interact by first providing a verbal or written description, and then requesting photos or video frames as evidence based on the description. This embodiment simulates the human diagnostic thought process, aligns with users' behavioral habits when reporting repairs, and provides a more natural interactive experience.

[0041] Meanwhile, the computational overhead of semantic analysis of text or speech is far lower than that of object detection and feature extraction of high-resolution images. By using lightweight semantic analysis as the first stage, invalid repair requests (i.e., cases where the semantic confidence is less than or equal to the first threshold) can be quickly filtered out, thereby significantly reducing the overall computational load and response latency of the system and optimizing the allocation of system resources while ensuring diagnostic accuracy.

[0042] Using the above method, semantic analysis is performed on language data to generate semantic confidence scores. Cross-modal verification with images is triggered only when the semantic confidence score meets the standard, which significantly reduces the system's computational load and response latency. When the semantic confidence score meets the standard and image data is available, the visual features of the target region in the image are matched with the semantic vectors across modalities to obtain the visual confidence score. This visual confidence score is then weighted and fused with the semantic confidence score to obtain the target confidence score, effectively avoiding the mixing of noisy features from low-quality modalities with effective features from high-quality modalities. Finally, the fault authenticity is verified based on the target confidence score, making the final fault judgment result more accurate and reliable.

[0043] In some embodiments, when the language data includes speech data, semantic analysis is performed on the language data to generate semantic confidence scores, including but not limited to: First, the speech data is converted into text data, and then the environmental noise energy value is extracted from the speech data. The environmental noise energy value is a quantitative indicator of the background noise intensity in the acquired speech data, reflecting the level of noise in the equipment operating environment or recording environment. The environmental noise energy value is obtained through the following methods: The collected speech data is processed into frames (e.g., frame length 25ms, frame shift 10ms), and a speech activity detection algorithm is used to identify silent frames (i.e., signal segments that do not contain human speech). The short-time energy (decibels quantized value) of the silent frames is extracted, and the average short-time energy of all silent frames is calculated. This average value is used as the environmental noise energy value.

[0044] Then, using a natural language processing model, semantic extraction is performed on the converted text data to obtain fault semantic vectors. Simultaneously, the fault object (including component name and location description) and fault phenomenon are extracted from the text.

[0045] Subsequently, the similarity between the fault semantic vector and each preset fault description vector in the preset fault description library is calculated. The preset fault description library includes at least one preset fault description and its corresponding preset fault description vector. The similarity can be calculated using cosine similarity, Euclidean distance, Manhattan distance, or dot product similarity, depending on the specific application scenario; no limitation is made here. The highest calculated similarity is then taken as the matching degree between the fault semantic vector and the preset fault description library.

[0046] Finally, based on the matching degree (sim max ) and environmental noise energy value (E noise ) Calculate the semantic confidence C1, where E noise It is negatively correlated with C1, and the formula for calculating C1 is as follows: C1=α1×sim max -β1×log(E noise ), where α1 and β1 are weighting coefficients, satisfying α1 + β1 = 1. The value of β1 is dynamically adjusted according to the signal-to-noise ratio (SNR) to reflect the impact of noise on semantic reliability. In this embodiment, the SNR can be calculated based on the energy of the speech data and the noise energy. The adjustment rules for the weighting coefficients are as follows: When SNR>20dB (high signal-to-noise ratio, quiet environment), β1=0.2, and α1=0.8. When 20dB≥SNR>10dB (medium signal-to-noise ratio, with some ambient noise), take β1=0.5, then α1=0.5; When 10dB≥SNR (low signal-to-noise ratio, noisy environment), β1=0.8, and α1=0.2. The specific adjustment rules can be determined according to the actual situation, and are not limited here.

[0047] The above method extracts environmental noise energy values ​​from speech data and dynamically adjusts the weighting coefficients in the semantic confidence calculation formula based on the signal-to-noise ratio (SNR). When the environment is noisy, the noise energy term carries a higher weight in the confidence calculation, effectively lowering the semantic confidence and reducing invalid repair requests and misjudgments caused by environmental interference. In high SNR environments, the system places greater trust in the semantic matching results, maintaining efficient and accurate fault semantic understanding. This improves the anti-interference capability and robustness of fault diagnosis.

[0048] In some embodiments, when the language data includes text data, semantic analysis is performed on the language data to generate semantic confidence scores, including but not limited to: By using a natural language processing model, semantic extraction is performed on the text data to obtain fault semantic vectors. At the same time, the fault objects (including component names and location descriptions) and fault phenomena are extracted from the text.

[0049] Subsequently, the similarity between the fault semantic vector and each preset fault description vector in the preset fault description library is calculated. The highest similarity obtained is then used as the semantic confidence score, or the highest normalized similarity score is used as the semantic confidence score.

[0050] Using the above method, for scenarios with plain text input, the semantic confidence is directly determined by the semantic matching degree between the text description and the preset fault database, thereby enabling a rapid assessment of the reliability of the fault description and providing reliable semantic confidence input for subsequent fault verification.

[0051] In some embodiments, it is exemplarily illustrated that visual features of the target region are matched with semantic analysis results across modalities to obtain visual confidence, including but not limited to: First, visual features of the target region are extracted and mapped to target feature vectors of the same dimension as the fault semantic vectors in the semantic analysis results.

[0052] For example, visual features (such as color distribution, texture patterns, and shape contours) of the target region are extracted using a convolutional neural network. These features are then input into a cross-modal mapping model, which converts the visual features into text phrases describing the state of the region (such as wear marks on the bearing surface or cracks on the blade edge). Subsequently, the text phrases are encoded using the same natural language processing model as in the semantic analysis stage to obtain a target feature vector with the same dimensions as the fault semantic vector.

[0053] Then, the cross-modal similarity between the target feature vector and the fault semantic vector in the semantic analysis result is calculated to obtain the visual confidence. The cross-modal similarity can be calculated using cosine similarity, Euclidean distance, Manhattan distance, or dot product similarity, depending on the actual application scenario. No specific method is specified here.

[0054] The above method converts the visual features of the target region in the image into descriptive text phrases, and then encodes them using the same natural language processing model as the semantic analysis stage. This allows image information and text information to be aligned and compared in the same semantic vector space, avoiding noise interference caused by direct splicing of features from different modalities, thereby improving the accuracy and interpretability of cross-modal fault verification.

[0055] In some embodiments, exemplarily illustrated, after determining that the device is faulty, the method further includes: First, the time-varying impact coefficient and safety risk value of the fault are obtained. The time-varying impact coefficient increases with the fault dwell time and is used to reflect the constraint of the time dimension on the urgency of the fault. The safety risk value is used to characterize the spatial impact range of the fault and is used to reflect the constraint of the spatial dimension on the severity of the fault.

[0056] For example, the time-varying influence coefficient I(t) can be calculated based on the time difference t between the fault occurrence time (obtained from the equipment log) and the current time. This embodiment uses an exponential growth model, i.e., I(t) = e^(-t / t). Kt Here, K is a time-sensitive factor used to adjust the amplification rate of t with respect to I(t). The value of K can be set according to the equipment type or operation and maintenance strategy. For example, a larger value is used for vulnerable parts or equipment with high safety requirements, and a smaller value is used for durable parts or non-core equipment. The specific value depends on the situation and is not limited here. Thus, through the exponential growth of I(t), the actual operation and maintenance needs of longer fault dwell time and higher urgency can be effectively reflected.

[0057] The safety risk value R is linked to historical fault data in the equipment knowledge base to quantify the spatial impact range of the fault. The spatial impact range can include dimensions such as the area that the fault may affect, the number of upstream and downstream devices involved, and whether it affects personnel safety.

[0058] For example, if a search of the equipment knowledge base reveals that a similar failure in the equipment's history only affected a single device and posed no safety risk, then R=0.2; if it affected multiple devices within a 10-meter radius but resulted in no personal injury, then R=0.6; if it caused minor personal injury or serious equipment damage, then R=0.8; and if it caused serious personal injury or significant property damage, then R=0.9 or 1. Specific quantitative standards can be set according to the enterprise's operation and maintenance rules and are not limited here.

[0059] Then, the target confidence S final We perform a weighted fusion of I(t) and R to obtain the fault level L = α² × S final +β2×I(t)+γ×R. Where α2, β2, and γ are weighting coefficients, each determined through backtracking optimization using historical fault data. For example, historical fault samples and their manually labeled true levels are collected, the mean absolute error between the model output and the true levels is calculated, and the coefficients are adjusted with the goal of minimizing the error. To adapt to changes in equipment operating status and maintenance strategies, new fault samples can be collected every six months, and α2, β2, and γ can be fine-tuned using gradient descent to achieve dynamic updates of the coefficients.

[0060] Finally, based on L, a repair work order for the fault is generated. For example, based on the numerical range of L, the work order priority (such as high, medium, low) is determined, and a complete repair work order is generated by combining information such as the fault object, location description, and responsible department.

[0061] Using the above method, I(t) and R are introduced to quantify the urgency and impact range of the fault from both temporal and spatial dimensions, respectively, and combined with S obtained from multimodal diagnosis. finalWeighted fusion improves the reliability of fault level assessment; it enables the maintenance work order generated based on L to reasonably reflect the true urgency of the fault, which is conducive to the optimal scheduling of maintenance resources.

[0062] In some embodiments, it is illustrated, by way of example, that a fault repair work order is generated based on L, including but not limited to: First, the multimodal processing engine calls the dispatch decision module to determine the work order priority based on the numerical range of L. In actual operation and maintenance scenarios, maintenance resources are limited and cannot process all repair requests simultaneously. By comprehensively reflecting the time urgency and spatial hazard of the fault through L, and converting it into a clear priority, the work order management system can be guided to intelligently schedule work orders according to the principle of prioritizing high-priority faults and delaying low-priority faults. This avoids delaying high-urgent faults and preventing low-urgent faults from monopolizing resources, thereby improving the overall utilization efficiency of maintenance resources and ensuring the continuous operation of critical equipment and on-site safety.

[0063] The multimodal processing engine calls the location mapping module, which queries the location mapping library to convert the location description of the faulty object in the semantic analysis results into spatial coordinates, enabling maintenance personnel to quickly navigate to the fault site.

[0064] The multimodal processing engine calls the dispatch decision module to determine the target maintenance department corresponding to the faulty object by querying the maintenance responsibility matrix library. Spatial coordinates refer to the physical location coordinates of the equipment in the building information model, usually represented as (X, Y, Z) triples.

[0065] Finally, the multimodal processing engine calls the dispatch decision module to generate a repair work order based on the work order priority, spatial coordinates, and target repair department. The generated work order may include: work order number, fault description, priority level, spatial coordinates, target repair department, and suggested response time. This work order is then pushed to the work order management system for processing by the appropriate repair personnel.

[0066] By using the above method, semantic location descriptions are converted into precise spatial coordinates, enabling maintenance personnel to quickly locate the specific location of faulty equipment. At the same time, by combining the work order priority determined by the fault level and the target maintenance department determined by the maintenance responsibility matrix, a fully automated process from fault confirmation to work order dispatch is realized, which significantly improves the efficiency of operation and maintenance response and the accuracy of work order dispatch.

[0067] In some embodiments, it is illustrated, by way of example, that the priority of a work order is determined based on the numerical range in which L is located, including but not limited to: If L is greater than the first-level threshold (e.g., the fault may cause significant property damage or personal injury, and the delay time is relatively long), the work order priority is determined to be high priority, and a red emergency work order label is generated; if L is greater than the second-level threshold, but less than or equal to the first-level threshold (e.g., the fault has affected production but will not cause an immediate safety accident), the work order priority is determined to be medium priority, and an orange high-priority work order label is generated; if L is less than or equal to the second-level threshold (e.g., a minor fault that can be included in the routine maintenance plan), the work order priority is determined to be low priority, and a blue regular work order label is generated.

[0068] The aforementioned first-level and second-level thresholds can be dynamically adjusted based on factors such as the enterprise's operation and maintenance strategy, the criticality level of equipment, and the historical distribution of fault levels. For example, for core equipment, the first-level and second-level thresholds can be appropriately lowered to allow more work orders to enter the high-priority channel; for non-core equipment, the thresholds can be appropriately raised to avoid frequent interruptions to routine maintenance.

[0069] In this system, red, orange, and blue serve as visual identifiers for work orders, allowing maintenance personnel to quickly identify the urgency of faults in the work order list and clearly distinguish priorities. Red indicates an immediate response, orange indicates a response within a short timeframe, and blue indicates that the work can be handled according to the standard plan. The color-coding strictly corresponds to priority, which helps improve maintenance scheduling efficiency and communication clarity. Specific visual identifiers can be determined based on the actual application scenario and are not limited here.

[0070] By using the above method, the fault level is quantified into an executable work order priority, which allows maintenance personnel to intuitively judge the urgency of the handling based on the work order color, optimizes the scheduling efficiency of maintenance resources, and ensures that high-risk and high-urgency faults are handled first.

[0071] In some embodiments, the language data is exemplarily described as being generated in the following manner: Acquire the inspection visual data of the equipment and perform anomaly detection on the inspection visual data, wherein the visual data includes inspection image data and / or inspection video data.

[0072] Specifically, the inspection device collects image or video data of the equipment according to a preset cycle (such as once a day, once an hour, etc.) or under specific triggering conditions (such as an alarm for abnormal equipment operation status) and uploads it to the service layer. The inspection device can be an inspection robot equipped with a camera, a drone, a fixed monitoring camera, or a smart terminal device handheld by the inspection personnel.

[0073] If an anomaly is detected in the inspection visual data, language data is generated based on the anomaly. The anomaly detection methods include: For inspection image data, the multimodal processing engine compares it with the status of standard equipment components stored in the equipment knowledge base. If the comparison is inconsistent (such as incorrect indicator light color, instrument reading outside the normal range, cracks or foreign objects on the surface of the component, incorrect switch position, etc.), an anomaly is determined. For inspection video data, the multimodal processing engine performs time-series anomaly detection (such as analyzing whether the motion trajectory changes abruptly in consecutive frames, whether the vibration amplitude increases suddenly, whether the rotation is stuck, etc.). If an abnormal pattern is detected, an anomaly is determined.

[0074] Therefore, based on the detected anomalies (such as the red fault indicator light on the power distribution cabinet illuminating, the motor vibration amplitude exceeding the limit, or cracks appearing on the conveyor belt rollers), a corresponding text description is automatically generated. This text description is used as linguistic data to characterize the equipment fault and triggers the subsequent repair process.

[0075] By employing the methods described above, inspection images or videos of the equipment are acquired. A multimodal processing engine is then used to perform standard state comparisons on the images and temporal anomaly detection on the videos. This enables the automatic identification of equipment anomalies and the generation of standardized fault text descriptions without relying on manual descriptions, thereby triggering the repair reporting process. This reduces the subjective judgment bias of manual inspections, achieves seamless integration of inspection and repair reporting, and improves the intelligence level of equipment operation and maintenance and fault response efficiency.

[0076] It should be understood that, although Figure 2 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 2 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0077] In some embodiments, such as Figure 3 As shown, a device for diagnosing equipment faults is provided, comprising: a generation module 301, a positioning module 302, a matching module 303, and a fault diagnosis module 304, wherein: The generation module 301 is used to perform semantic analysis on the language data representing equipment faults and generate semantic confidence based on the semantic analysis results. The language data includes speech data and / or text data. The positioning module 302 is used to acquire image data of the device when the semantic confidence is greater than a first threshold, and locate the target area in the image data based on the fault object in the semantic analysis result. The matching module 303 is used to perform cross-modal matching between the visual features of the target region and the semantic analysis results to obtain the visual confidence score, and to calculate the target confidence score based on the semantic confidence score and the visual confidence score. The fault diagnosis module 304 is used to determine that the device is faulty if the target confidence level is greater than the second threshold.

[0078] Specific limitations regarding the equipment fault diagnosis device can be found in the limitations of the equipment fault diagnosis method described above, and will not be repeated here. Each module in the aforementioned equipment fault diagnosis device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0079] In some embodiments, an electronic device is provided that can perform the functions of the aforementioned device fault diagnosis device, with reference to... Figure 4 Electronic devices include: At least one processor 401 and a memory 402 connected to at least one processor 401. In this embodiment, the specific connection medium between the processor 401 and the memory 402 is not limited. Figure 4 The example shown is the connection between processor 401 and memory 402 via bus 400. Bus 400 is... Figure 4 The connections between other components are indicated by thick lines and are for illustrative purposes only, not as limiting information. The 400 bus can be divided into address bus, data bus, control bus, etc., for ease of representation. Figure 4 The term is represented by a single thick line, but this does not imply that there is only one bus or one type of bus. Alternatively, processor 401 can also be called a controller; there is no restriction on the name.

[0080] In this embodiment, the memory 402 stores instructions executable by at least one processor 401. By executing the instructions stored in the memory 402, the at least one processor 401 can perform the device fault diagnosis method described above. The processor 401 can implement... Figure 3 The functions of each module in the device shown.

[0081] The processor 401 is the control center of the device. It can connect to various parts of the control device through various interfaces and lines. By running or executing instructions stored in memory 402 and calling data stored in memory 402, the processor can perform various functions and process data, thereby monitoring the device as a whole.

[0082] In some embodiments, processor 401 may include one or more processing units. Processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into processor 401. In some embodiments, processor 401 and memory 402 may be implemented on the same chip; in some embodiments, they may be implemented on separate chips.

[0083] Processor 401 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit, field-programmable gate array or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the device fault diagnosis method disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0084] Memory 402, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 402 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. Memory 402 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. In the embodiments of this application, memory 402 may also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0085] By designing and programming the processor 401, the code corresponding to the device fault diagnosis method described in the foregoing embodiments can be embedded into the chip, thereby enabling the chip to execute the code during operation. Figure 2 The steps of the device fault diagnosis method in the illustrated embodiment are as follows. How to design and program the processor 401 is a technique well-known to those skilled in the art and will not be described further here.

[0086] In some embodiments, a computer storage medium is provided, comprising computer program code that, when executed on a computer, causes the computer to perform any of the device fault diagnosis methods described above. Since the principle by which the computer storage medium solves the problem is similar to that of a device fault diagnosis method, the implementation of the computer storage medium can be found in the implementation of the method, and repetitions will not be repeated.

[0087] In specific implementation, computer storage media can include: Universal Serial Bus Flash Drive (USB), portable hard drive, Read-Only Memory (ROM), Random Access Memory (RAM), magnetic disk or optical disk, and other storage media that can store program code.

[0088] In some embodiments, a computer program product is also provided, comprising: computer program code, which, when executed on a computer, causes the computer to perform any of the device fault diagnosis methods described above. Since the principle by which the above-described computer program product solves the problem is similar to that of a device fault diagnosis method, the implementation of the above-described computer program product can be found in the implementation of the method, and repeated details will not be elaborated further.

[0089] Computer program products may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0090] The methods in this application can be implemented, in whole or in part, by software, hardware, firmware, or any combination thereof. When implemented in software, they can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions of this application are performed, in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, network equipment, user equipment, core network equipment, OAM, or other programmable device.

[0091] Computer programs or instructions can be stored in or transferred from one computer-readable storage medium to another. For example, a computer program or instructions can be transferred from one website, computer, server, or data center to another via wired or wireless means. A computer-readable storage medium can be any usable medium that a computer can access, or a data storage device such as a server or data center that integrates one or more usable media. Usable media can be magnetic media, such as floppy disks, hard disks, and magnetic tapes; optical media, such as digital video discs; or semiconductor media, such as solid-state drives. The computer-readable storage medium can be volatile or non-volatile, or may include both types.

[0092] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0093] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 Devices that specify the functions in one or more boxes.

[0094] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction device, which is implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0095] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0096] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for diagnosing equipment faults, characterized in that, The method includes: Semantic analysis is performed on the language data characterizing equipment failure, and semantic confidence is generated based on the semantic analysis results, wherein the language data includes speech data and / or text data; If the semantic confidence level is greater than a first threshold, image data of the device is acquired, and the target area in the image data is located based on the faulty object in the semantic analysis result. The visual features of the target region are matched with the semantic analysis results across modalities to obtain visual confidence, and the target confidence is calculated based on the semantic confidence and the visual confidence. If the target confidence level is greater than the second threshold, then the device is determined to be faulty.

2. The method according to claim 1, characterized in that, When the language data includes speech data, the step of performing semantic analysis on the language data to generate semantic confidence includes: The speech data is converted into text data, and the environmental noise energy value is extracted from the speech data; Semantic extraction is performed on the converted text data to obtain fault semantic vectors; Calculate the similarity between the fault semantic vector and each preset fault description vector in the preset fault description library, wherein the preset fault description library includes at least one preset fault description and a corresponding preset fault description vector; The highest similarity obtained is used as the matching degree between the fault semantic vector and the preset fault description library; The semantic confidence score is calculated based on the matching degree and the environmental noise energy value, wherein the environmental noise energy value is negatively correlated with the semantic confidence score.

3. The method according to claim 1, characterized in that, The step of performing cross-modal matching between the visual features of the target region and the semantic analysis results to obtain visual confidence includes: Extract the visual features of the target region and map the visual features to a target feature vector with the same dimension as the fault semantic vector in the semantic analysis result; The cross-modal similarity between the target feature vector and the fault semantic vector is calculated to obtain the visual confidence score.

4. The method according to claim 1, characterized in that, After determining that the device is faulty, the process further includes: The time-varying impact coefficient and safety risk value of the fault are obtained, wherein the time-varying impact coefficient increases with the increase of the fault residence time, and the safety risk value is used to characterize the spatial impact range of the fault; The target confidence level, the time-varying influence coefficient, and the safety risk value are weighted and fused to obtain the fault level. Based on the fault level, a repair work order for the fault is generated.

5. The method according to claim 4, characterized in that, The step of generating a repair work order for the fault based on the fault level includes: The priority of the work order is determined based on the numerical range of the fault level. The location description of the faulty object in the semantic analysis results is converted into spatial coordinates, and the target maintenance department corresponding to the faulty object is determined. The maintenance work order is generated based on the work order priority, the spatial coordinates, and the target maintenance department.

6. The method according to claim 5, characterized in that, The step of determining the work order priority based on the numerical range of the fault level includes: If the fault level is greater than the first level threshold, then the work order priority is determined to be high priority; If the fault level is greater than the second level threshold and less than or equal to the first level threshold, then the work order priority is determined to be medium priority. If the fault level is less than or equal to the second level threshold, then the work order priority is determined to be low priority.

7. The method according to claim 1, characterized in that, The language data is generated in the following manner: The inspection visual data of the device is acquired, and anomaly detection is performed on the inspection visual data, wherein the visual data includes inspection image data and / or inspection video data; If an anomaly is detected in the inspection visual data, the language data is generated based on the anomaly.

8. A device for diagnosing equipment faults, characterized in that, The device includes: The generation module is used to perform semantic analysis on the language data representing equipment failure and generate semantic confidence based on the semantic analysis results, wherein the language data includes speech data and / or text data; The positioning module is used to acquire image data of the device when the semantic confidence is greater than a first threshold, and to locate the target area in the image data based on the fault object in the semantic analysis result. The matching module is used to perform cross-modal matching between the visual features of the target region and the semantic analysis results to obtain visual confidence, and to calculate the target confidence based on the semantic confidence and the visual confidence. The fault diagnosis module is used to determine that the device is faulty if the target confidence level is greater than a second threshold.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, when executing a computer program stored in the memory, implements the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 7.

Images