Method, device, management system and electronic device for handling device failure

Abstract

The present disclosure provides a device fault processing method, device, management system and electronic device, the processing method comprising constructing a digital twin of a target device and visualizing the digital twin in a mixed reality device; collecting a plurality of modal actual correlation parameters; based on the actual correlation parameters and a pre-trained preset fault recognition model, obtaining a target fault recognition result; displaying the target fault recognition result in the digital twin; in response to an external interactive operation, obtaining target fault analysis data based on a pre-constructed preset fault physical knowledge graph. In the present disclosure, a real-time interaction and mutual enhancement closed-loop processing scheme of fault recognition between the "physical world-digital twin-fault recognition model-MR interaction-human expert" is constructed, ensuring the accuracy and efficiency of fault recognition in any target device, and based on the display and interaction in the MR environment, the interactive experience of the fault processing scene is effectively improved.

Description

Technical Field

[0001] This disclosure relates to the field of fault detection technology, and in particular to a method, apparatus, management system and electronic equipment for handling equipment faults. Background Technology

[0002] Early fault identification in industrial equipment (such as fans, pumps, gearboxes, motors, etc.) is a huge challenge for the industry. Existing equipment fault identification schemes mainly have the following limitations: (1) Vibration signal-based schemes: are sensitive to the location of sensor placement, making it difficult to identify early and weak faults, and the diagnostic results lack intuitiveness and cannot directly guide maintenance. (2) Deep learning-based schemes: are mostly "black box" models, heavily reliant on a large amount of labeled data, have poor generalization ability for new fault types with few or zero samples, and their decision-making process lacks physical causal support, resulting in low reliability. Therefore, existing equipment fault identification schemes generally fail to meet the needs of actual scenarios. Summary of the Invention

[0003] The technical problem to be solved by this disclosure is to overcome the shortcomings of existing fault identification schemes, such as difficulty in identifying early and weak faults and poor model generalization ability. The purpose is to provide a method, device, management system and electronic equipment for handling equipment faults.

[0004] This disclosure solves the above-mentioned technical problems through the following technical solution:

[0005] In a first aspect, this disclosure provides a method for handling equipment malfunctions, the method comprising:

[0006] A digital twin of the target device is constructed and visualized in a mixed reality device;

[0007] The actual correlation parameters of the target device in the preset operation process are collected;

[0008] Based on the actual correlation parameters and the pre-trained preset fault identification model, the target fault identification result is obtained;

[0009] The target fault identification results are displayed in the digital twin according to preset display rules to obtain the target display content;

[0010] In response to receiving an external interactive operation based on the target displayed content, target fault analysis data matching the target fault identification result is obtained based on a pre-constructed preset fault physical knowledge graph.

[0011] Optionally, the step of constructing a digital twin of the target device includes:

[0012] Obtain the three-dimensional physical model of the target device;

[0013] In the mixed reality device, the three-dimensional physical model is spatially registered with the target device in the real environment to construct the digital twin of the target device in the mixed reality space;

[0014] And / or,

[0015] Different types of sensing devices are deployed at different preset locations in the real environment where the target device is located;

[0016] The step of acquiring the actual correlation parameters of the multimodal modes of the target device during a preset operating process includes:

[0017] Using different types of sensing devices, the actual correlation parameters of the target device in the preset operation process are collected respectively in the multimodal mode;

[0018] And / or,

[0019] The steps for constructing the preset fault physical knowledge graph include:

[0020] Obtain several dimensional characterization parameters for each physical component in the target device;

[0021] To obtain the causal relationships between the different physical components;

[0022] By using the representation parameters of several dimensions as entity nodes and the causal relationships as causal relationship edges, a structured preset fault physical knowledge graph is constructed.

[0023] Optionally, the step of constructing the preset fault identification model includes:

[0024] Obtain the first number of first historical sample data from different sample devices;

[0025] In response to the first quantity being less than a preset quantity, a preset sample construction method is used to expand and construct a second quantity of second historical sample data corresponding to different sample devices;

[0026] The first historical sample data includes the first sample association parameters of the multimodal nature of the sample device and the corresponding first sample device fault label data; the second historical sample data includes the second sample association parameters of the multimodal nature of the sample device and the corresponding second sample device fault label data.

[0027] Alternatively, the first historical sample data includes the first sample fusion parameters obtained by fusing the first sample association parameters of the multimodal data of the sample device, and the corresponding first sample device fault label data; the second historical sample data includes the second sample fusion parameters obtained by fusing the second sample association parameters of the multimodal data of the sample device, and the corresponding second sample device fault label data.

[0028] A sample training set is obtained based on the first historical sample data and the second historical sample data;

[0029] The preset model is trained based on the sample training set to obtain the preset fault identification model used to predict the fault condition in any target device.

[0030] Optionally, the step of expanding to obtain a second number of second historical sample data corresponding to different sample devices using a preset sample construction method includes:

[0031] By employing a preset digital simulation algorithm and the physical laws followed by the sample devices, the second number of second historical sample data corresponding to different sample devices is obtained;

[0032] And / or,

[0033] The model training process of the preset fault identification model also includes:

[0034] The sample device is encoded according to several preset physical principles to obtain corresponding encoded data, and the encoded data is used as a regularization constraint term of the preset model for model training.

[0035] And / or,

[0036] The loss functions in the preset model include a physical consistency loss function and a prediction minimization loss function.

[0037] Optionally, the step of displaying the target fault identification result in the digital twin according to a preset display rule to obtain the target display content includes:

[0038] Obtain the faulty component and corresponding fault identification information from the target fault identification result;

[0039] The faulty component is highlighted in the digital twin using a first preset display method;

[0040] And / or,

[0041] In the first preset area of ​​the floating display interface in the digital twin, the fault identification information is displayed using a matching second preset display method;

[0042] And / or,

[0043] The processing method further includes:

[0044] In the second preset area of ​​the floating display interface in the digital twin, the actual parameter data of each modality are displayed using their respective matching third preset display methods;

[0045] And / or,

[0046] The processing method further includes:

[0047] Based on the aforementioned external interaction operations, determine the actual interaction requirements;

[0048] Based on the actual interaction requirements, the preset fault physical knowledge graph is invoked to generate causal tracing path information for describing the reverse reasoning that caused the fault, and the causal tracing path information is displayed in the third preset area of ​​the floating display interface in the digital twin using a fourth preset display method.

[0049] Optionally, the processing method further includes:

[0050] The target fault analysis data is determined or corrected;

[0051] Obtain the processed target fault analysis data;

[0052] And / or,

[0053] The processed target fault analysis data is used as the actual fault label for the corresponding multimodal actual correlation parameter;

[0054] New sample training data is formed based on the actual correlation parameters of the multimodal model and the corresponding actual fault labels;

[0055] In response to the cumulative number of new sample training data reaching a preset value, the model is updated and iterated to obtain a new preset fault identification model;

[0056] And / or,

[0057] Based on the processed target fault analysis data, the confidence weight values ​​of the corresponding causal relationship edges in the preset fault physical knowledge graph are adjusted to update the preset fault physical knowledge graph.

[0058] And / or,

[0059] The processing method further includes:

[0060] Analyze several identified reference cases within a preset time period;

[0061] In response to a systematic deviation between the actual characteristic frequency and the theoretical characteristic frequency of a type of fault in any of the reference cases, a frequency compensation coefficient is generated.

[0062] The preset fault identification model is corrected based on the frequency compensation coefficient.

[0063] Optionally, the step of obtaining the target fault identification result based on the actual correlation parameters and the pre-trained preset fault identification model includes:

[0064] The actual correlation parameters of different modalities are asynchronously fused in the time-frequency domain to obtain actual fused parameter data;

[0065] The actual fusion parameter data is input into the pre-trained preset fault identification model to obtain the target fault identification result;

[0066] And / or,

[0067] The preset model includes a meta-learner;

[0068] And / or,

[0069] The target equipment includes rotating machinery;

[0070] And / or,

[0071] The external interaction operations include gestures and / or voice interaction methods.

[0072] A second aspect of this disclosure provides an apparatus for handling equipment malfunctions, the apparatus comprising:

[0073] A digital twin processing module is used to construct a digital twin of the target device and visualize the digital twin in a mixed reality device;

[0074] A multimodal parameter acquisition module is used to acquire the actual correlation parameters of the target device in the preset operation process.

[0075] The fault identification module is used to obtain the target fault identification result based on the actual correlation parameters and the pre-trained preset fault identification model;

[0076] The fault display module is used to display the target fault identification result in the digital twin according to a preset display rule to obtain the target display content;

[0077] The interaction processing module is used to respond to external interactive operations received based on the target displayed content, and obtain target fault analysis data that matches the target fault identification result based on a pre-constructed preset fault physical knowledge graph.

[0078] Optionally, the digital twin processing module includes:

[0079] A 3D model acquisition unit is used to acquire a 3D physical model of the target device;

[0080] A digital twin construction unit is used to register the three-dimensional physical model with the target device in the real environment in the mixed reality device to construct the digital twin of the target device in the mixed reality space.

[0081] And / or,

[0082] Different types of sensing devices are deployed at different preset locations in the real environment where the target device is located;

[0083] The multimodal parameter acquisition module is used to acquire the actual correlation parameters of the target device in the preset operation process using different types of sensing devices.

[0084] And / or,

[0085] The identification device further includes a knowledge graph construction module, which comprises:

[0086] The parameter acquisition unit is used to acquire several dimensional characterization parameters of each physical component in the target device;

[0087] A causal relationship acquisition unit is used to acquire causal relationships between different physical components;

[0088] The knowledge graph construction unit is used to construct a structured physical knowledge graph of the preset fault by using the representation parameters of several dimensions as entity nodes and the causal relationships as causal relationship edges.

[0089] Optionally, the identification device further includes a prediction model building module, which includes:

[0090] The first sample module unit is used to acquire a first number of first historical sample data from different sample devices;

[0091] The sample expansion unit is used to respond to the first quantity being less than a preset quantity by using a preset sample construction method to expand and construct a second quantity of second historical sample data corresponding to different sample devices;

[0092] The first historical sample data includes the first sample association parameters of the multimodal nature of the sample device and the corresponding first sample device fault label data; the second historical sample data includes the second sample association parameters of the multimodal nature of the sample device and the corresponding second sample device fault label data.

[0093] Alternatively, the first historical sample data includes the first sample fusion parameters obtained by fusing the first sample association parameters of the multimodal data of the sample device, and the corresponding first sample device fault label data; the second historical sample data includes the second sample fusion parameters obtained by fusing the second sample association parameters of the multimodal data of the sample device, and the corresponding second sample device fault label data.

[0094] The training set acquisition unit is used to obtain a sample training set based on the first historical sample data and the second historical sample data;

[0095] The model training unit is used to train a preset model based on the sample training set to obtain the preset fault identification model for predicting fault conditions in any target device.

[0096] Optionally, the sample expansion unit is used to expand to obtain the second number of second historical sample data corresponding to different sample devices by using a preset digital simulation algorithm and the physical laws followed by the sample devices;

[0097] And / or,

[0098] The model training unit is also used to encode several preset physical principles followed by the sample device to obtain corresponding encoded data, and use the encoded data as a regularization constraint term of the preset model for model training.

[0099] And / or,

[0100] The loss functions in the preset model include a physical consistency loss function and a prediction minimization loss function.

[0101] Optionally, the fault display module is further configured to acquire the faulty component and corresponding fault identification information in the target fault identification result; and to highlight the faulty component in the digital twin using a first preset display method.

[0102] And / or,

[0103] The fault display module is also used to display the fault identification information in a first preset area of ​​the floating display interface in the digital twin using a matching second preset display method;

[0104] And / or,

[0105] The fault display module is also used to display the actual parameter data of each modality using a third preset display method that matches each modality in the second preset area of ​​the floating display interface in the digital twin.

[0106] And / or,

[0107] The interaction processing module is also used to determine the actual interaction requirements based on the external interaction operation;

[0108] Based on the actual interaction requirements, the preset fault physical knowledge graph is invoked to generate causal tracing path information for describing the reverse reasoning that caused the fault, and the fault display module is invoked.

[0109] The fault display module is also used to display the causal tracing path information in the third preset area of ​​the floating display interface in the digital twin using a fourth preset display method.

[0110] Optionally, the identification device further includes:

[0111] The fault analysis data processing module is used to determine or correct the target fault analysis data;

[0112] The analysis data output module is used to acquire the processed target fault analysis data;

[0113] And / or,

[0114] The new sample data formation module is used to take the processed target fault analysis data as the actual fault label of the corresponding multimodal actual correlation parameter; and to form new sample training data based on the multimodal actual correlation parameter and the corresponding actual fault label.

[0115] The model update module is used to update and iterate the model in response to the cumulative number of new sample training data reaching a preset value, so as to obtain a new preset fault identification model.

[0116] And / or,

[0117] The knowledge graph update module is used to adjust the confidence weight values ​​of the corresponding causal relationship edges in the preset fault physical knowledge graph based on the processed target fault analysis data, so as to update the preset fault physical knowledge graph.

[0118] And / or,

[0119] The identification device further includes:

[0120] The reference case analysis module is used to analyze several identified reference cases within a preset time period.

[0121] The compensation coefficient generation module is used to generate frequency compensation coefficients in response to a systematic deviation between the actual characteristic frequency and the theoretical characteristic frequency of a type of fault in any of the reference cases.

[0122] The model correction module is used to correct the preset fault identification model based on the frequency compensation coefficient.

[0123] Optionally, the fault identification module is used to asynchronously fuse the actual correlation parameters of different modalities in the time-frequency domain to obtain actual fused parameter data; and input the actual fused parameter data into the pre-trained preset fault identification model to obtain the target fault identification result;

[0124] And / or,

[0125] The preset model includes a meta-learner;

[0126] And / or,

[0127] The target equipment includes rotating machinery;

[0128] And / or,

[0129] The external interaction operations include gestures and / or voice interaction methods.

[0130] A third aspect of this disclosure provides a management system for equipment failure, the management system including a mixed reality device and an equipment failure processing apparatus as provided in the second aspect.

[0131] A fourth aspect of this disclosure provides an electronic device including a memory, a processor, and a computer program stored in the memory and for running on the processor, wherein the processor, when executing the computer program, implements a device malfunction handling method as provided in the first aspect.

[0132] A fifth aspect of this disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a device malfunction handling method as provided in the first aspect.

[0133] A sixth aspect of this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements a device malfunction handling method as provided in the first aspect.

[0134] Based on common knowledge in the field, the above-mentioned preferred conditions can be combined arbitrarily to obtain various preferred embodiments of this disclosure.

[0135] The positive and progressive effects of this disclosure are as follows:

[0136] In this disclosure, a digital twin of a device is constructed and visualized within an MR (Mixed Reality) device. Multimodal operating parameters are then synchronously collected using different types of sensors. Based on this modal data, the faulty components and fault conditions within the target device are accurately identified dynamically and promptly. This information is then displayed in a targeted, intuitive, and clear manner within the MR environment. Furthermore, user interactions via gestures, voice, and other means can trigger the invocation of a pre-defined fault physical knowledge graph, enabling rapid and accurate causal reasoning to trace the corresponding fault. This essentially constructs a closed-loop fault identification solution that facilitates real-time interaction and mutual reinforcement between the physical world, digital twin, fault identification model, MR interaction, and human experts. This effectively ensures the accuracy and efficiency of fault identification (such as early-stage faults) in any target device. The MR environment-based display and interaction provide a more intuitive understanding of the fault, significantly enhancing the interactive experience in fault handling scenarios.

[0137] Among them, by deeply integrating technologies such as mixed reality interaction, physical mechanism guidance and small sample meta-learning, dynamic identification, precise location and causal tracing of equipment faults (such as early and weak faults in rotating machinery) can be achieved. This effectively solves the core problems of traditional data-driven methods, such as the scarcity of early fault samples, weak fault characteristics and the "black box" nature of diagnostic results, which makes it difficult for field engineers to understand and trust them. Attached Figure Description

[0138] Figure 1 A flowchart of a method for handling equipment malfunctions provided in Embodiment 1 of this disclosure;

[0139] Figure 2 This is a first flowchart of a method for handling equipment malfunctions provided in Embodiment 2 of this disclosure;

[0140] Figure 3 This is a second flowchart of the equipment failure handling method provided in Embodiment 2 of this disclosure;

[0141] Figure 4 A schematic diagram of the module of the device for handling equipment failures provided in Embodiment 3 of this disclosure;

[0142] Figure 5 A schematic diagram of the module of the device for handling equipment failure provided in Embodiment 4 of this disclosure;

[0143] Figure 6 A schematic diagram of the module of the equipment fault management system provided in Embodiment 5 of this disclosure;

[0144] Figure 7 A schematic diagram of the structure of an electronic device provided in Embodiment 6 of this disclosure. Detailed Implementation

[0145] The present disclosure is further illustrated below by way of embodiments, but the present disclosure is not limited to the scope of the embodiments described herein.

[0146] The prefixes such as "first" and "second" used in this disclosure are merely for distinguishing different descriptive objects and do not limit the position, order, priority, quantity, or content of the described objects. The use of ordinal numbers and other prefixes used to distinguish descriptive objects in this disclosure does not constitute a limitation on the described objects. The description of the described objects is given in the claims or the context of the embodiments, and should not be construed as an unnecessary limitation. Furthermore, in the description of this embodiment, unless otherwise stated, "multiple" means two or more.

[0147] Example 1

[0148] like Figure 1 As shown, the equipment failure handling method of this embodiment includes:

[0149] S101. Construct a digital twin of the target device and visualize the digital twin in a mixed reality device;

[0150] The target equipment includes, but is not limited to, rotating machinery, which includes fans, pumps, gearboxes, motors, etc.

[0151] Mixed reality devices include, but are not limited to, mixed reality (MR) head-mounted displays.

[0152] S102. Collect the actual correlation parameters of the target device in the preset operation process;

[0153] Specifically, the actual correlation parameters of the multimodal model include the vibration parameters, acoustic parameters, and thermal imaging parameters of various components in the target device (such as bearings, gears, and other key components).

[0154] S103. Based on the actual correlation parameters and the pre-trained preset fault identification model, the target fault identification result is obtained;

[0155] S104. Display the target fault identification results in the digital twin using preset display rules to obtain the target display content;

[0156] S105. In response to receiving an external interactive operation based on the target displayed content, obtain target fault analysis data that matches the target fault identification result based on a pre-built preset fault physical knowledge graph.

[0157] Among them, the target fault analysis data corresponds to early faults in the target equipment, fault conditions in any operating process before the current moment, etc.

[0158] This solution constructs a digital twin of the device and visualizes it within the MR device. Then, multimodal operating parameters are synchronously collected using different types of sensors. Based on this modal data, the solution dynamically and promptly identifies faulty components and fault conditions within the target device, and presents this information in a targeted, intuitive, and clear manner within the MR environment. Simultaneously, it can receive user interactions via gestures, voice, and other means, triggering the invocation of a pre-defined fault physical knowledge graph. This allows for rapid and accurate causal reasoning to trace the corresponding fault. Essentially, this constructs a closed-loop fault identification solution that enables real-time interaction and mutual reinforcement between the physical world, digital twin, fault identification model, MR interaction, and human experts. This effectively ensures the accuracy and efficiency of fault identification (such as early-stage faults) in any target device. Furthermore, the MR environment-based display and interaction provide a more intuitive understanding of the fault, effectively enhancing the interactive experience in fault handling scenarios.

[0159] Example 2

[0160] The equipment failure handling method in this embodiment is a further improvement on Embodiment 1, specifically:

[0161] In a feasible solution, such as Figure 2 As shown, step S101 includes:

[0162] S1011. Obtain the three-dimensional physical model of the target device;

[0163] Among them, a high-precision 3D scanner is used to scan the target device (such as a gearbox) to obtain the physical structure of the target device, so as to obtain a digital three-dimensional physical model quickly and accurately.

[0164] S1012. In a mixed reality device, a three-dimensional physical model is registered with the target device in the real environment to construct a digital twin of the target device in the mixed reality space.

[0165] Specifically, in mixed reality devices, QR code marking technology is used to perform high-precision spatial coordinate registration between the three-dimensional physical model of the target device and the actual objects on site (e.g., spatial positioning error <5mm).

[0166] In this solution, a digital twin that is precisely aligned with the physical device is constructed through high-precision 3D scanning and mixed reality spatial registration technology to ensure the accuracy and efficiency of the digital twin determination, thereby ensuring the accuracy and reliability of subsequent fault identification and display processing.

[0167] In one feasible solution, different types of sensing devices are deployed at different preset locations in the real environment where the target device is located;

[0168] like Figure 2 As shown, step S102 includes:

[0169] S1021. Using different types of sensing devices, the actual correlation parameters of the target device in the preset operation process are collected.

[0170] Specifically, different types of sensing devices include vibration sensors, acoustic sensors (such as high-precision microphone arrays), and infrared sensors (such as thermal imaging cameras) to achieve non-contact dynamic working condition perception in different dimensions.

[0171] Vibration sensors are used to collect vibration signals of corresponding components during a preset operating process;

[0172] The acoustic sensor collects acoustic signals from the corresponding component during the preset operating process;

[0173] Infrared sensors are used to obtain thermal imaging signals of corresponding components during preset operation.

[0174] Among them, a multimodal sensor network, including vibration sensors, acoustic sensors, and infrared sensors, is deployed, and microsecond-level time synchronization is achieved through a precision clock protocol, so as to lay a precise data foundation for subsequent fusion analysis.

[0175] In one feasible solution, ICP accelerometers are installed in key components such as bearing housings and gearbox housings, with a sampling frequency ≥25.6 kHz; at least four microphone arrays are deployed to form an acoustic camera around the equipment, with a sampling frequency ≥48 kHz; and an infrared thermal imager is fixed opposite the critical temperature rise area of ​​the equipment, with a resolution of at least 640x480 and a frame rate ≥30 Hz. All sensors are connected to the same network via a precision clock protocol to achieve microsecond-level time synchronization, ensuring strict alignment of multimodal data on the timeline.

[0176] In this solution, in addition to vibration sensors, acoustic sensors are added to the equipment to use the acoustic signals of corresponding components to compensate for the shortcomings of vibration signals in low-frequency and far-field propagation; and infrared sensors are used to collect local temperature rise anomalies in corresponding components caused by friction, efficiency reduction, etc., so as to achieve multimodal comprehensive perception of any monitored component in the equipment; based on these multimodal data (such as asynchronously fusing the time and frequency domains of the three modes and inputting them into the preset fault identification model), the accuracy and reliability of subsequent fault identification results are guaranteed.

[0177] Of course, you can also choose any two types of sensing devices according to actual needs, or add other types of sensing devices besides these three types. The specifics can be determined or adjusted according to the needs of the actual scenario.

[0178] In one feasible solution, the pre-defined model training process for the fault identification model includes:

[0179] The sample devices are encoded according to several preset physical principles to obtain corresponding encoded data, and the encoded data is used as a regularization constraint term for the preset model for model training.

[0180] Specifically, sample equipment, such as rotating machinery, follows several pre-defined physical principles, including Newton's second law, rotor dynamics equations, and formulas for calculating gear meshing frequency.

[0181] In this scheme, the basic physical mechanism followed by the sample devices is processed, that is, the data is encoded by a preset encoding method to obtain encoded data, and then the encoded data is used as the regularization constraint of the model, thereby effectively preventing overfitting of the model training and ensuring the generalization ability of the model.

[0182] In a feasible approach, the pre-defined model includes, but is not limited to, a meta-learner, which uses the physical mechanisms followed by the sample devices as prior knowledge for the meta-learner. More specifically, the meta-learner includes MAML (Model Independent Meta-Learning) models.

[0183] The loss functions in the preset model include the physical consistency loss function and the prediction minimization loss function.

[0184] Specifically, a physical consistency loss function is introduced to constrain the feature frequencies learned by the model from the data with the theoretical fault frequencies calculated based on rotational speed and bearing parameters. During training, a large number of small-sample diagnostic tasks are constructed from historical data. In the internal optimization of each task, the data fitting loss function (such as MSE) and the physical consistency loss function are jointly optimized. In the external optimization, the parameters of the meta-learner are updated so that the model has the ability to quickly adapt to new tasks and conform to physical laws with only a few samples.

[0185] The formula for the physical consistency loss function is as follows:

[0186] L_physics=λ * Σ |f_model - f_theoretical|

[0187] Where L_physics is the physical consistency loss, f_model is the dominant fault frequency identified by the model from the data, f_theoretical is the theoretical fault characteristic frequency calculated by physical formulas based on equipment speed, bearing geometric parameters, etc., and λ is the adaptive weight, the initial value of which can be 0.5 or dynamically adjusted according to expert feedback.

[0188] In a feasible solution, the meta-learning training process is as follows:

[0189] (1) Task Construction: Construct a large number of "small sample diagnostic tasks" from historical sample data. Each task includes:

[0190] Support set: 1 normal sample + 1~3 samples of a certain type of fault;

[0191] Query set: A set of unlabeled samples used to evaluate and update the model;

[0192] (3) Inner loop optimization: For each task, the model performs gradient descent in a small number of steps (e.g., 5 steps) on the support set. The optimization objective is: L_total = L_mse + L_physics; where L_total is the total loss, L_mse is the minimum prediction loss, and L_physics is the physical consistency loss.

[0193] (3) Outer loop optimization: Calculate the total loss on the query set of all tasks, and update the initial parameters of the meta-learner through backpropagation so that it can quickly adapt to new tasks.

[0194] In this scheme, during the inner loop optimization process of the pre-defined model, in addition to minimizing the prediction loss function, a physical consistency loss function is also introduced. This physical consistency loss function penalizes model parameter update directions that violate known physical laws. For example, if the fault feature frequency predicted by the model is significantly inconsistent with the theoretical fault frequency calculated based on rotational speed and structural parameters, it will be penalized. This allows the model to quickly learn fault modes that conform to physical laws from a very small number of samples, thereby effectively ensuring the efficiency, accuracy, and stability of model training, especially for fault handling scenarios with very few samples.

[0195] In a feasible approach, the steps for constructing a pre-defined physical knowledge graph of faults include:

[0196] Obtain several dimensional characterization parameters for each physical component in the target device;

[0197] To obtain the causal relationships between different physical components;

[0198] By using several dimensions of representation parameters as entity nodes and causal relationships as causal relationship edges, a structured pre-defined fault physical knowledge graph is constructed.

[0199] Each physical component is characterized by several dimensions, including the failure modes, theoretical characteristic frequencies, vibration signals, acoustic signals, thermal characteristics, and equipment physical parameters (such as rotational speed and geometric dimensions) of key components such as bearings and gears. These parameters are treated as entity nodes. Causal relationships between them are defined according to physical laws (such as "failure mode-excitation-characteristic frequency"). For example, the failure modes (such as imbalance, misalignment, bearing spalling) are associated with vibration characteristic frequencies (1x, 2x, bearing passing frequency) and time-domain characteristics (peak value, kurtosis, envelope spectrum) and equipment physical parameters (mass, stiffness, damping) to obtain a structured fault physics knowledge graph, which is stored in a graph database (such as Neo4j) and assigned an initial confidence level.

[0200] Therefore, it can be understood that the process of pre-setting a fault physics knowledge map includes:

[0201] Node creation: Create four types of entity nodes, including fault modes, characteristic frequencies, time-domain / frequency-domain characteristics, and physical parameters;

[0202] Relationship definition: Defines the causal relationship edges between nodes;

[0203] For example: (bearing outer ring failure) --[excitation]--> (bearing outer ring passing frequency) --[calculation formula]-->(speed, number of rollers, contact angle);

[0204] (Gear tooth breakage) --[leads to]--> (Frequency modulation sideband) --[manifests as]--> (Sideband energy in the spectrum);

[0205] Knowledge storage: Use a graph database (such as Neo4j) to store nodes and relation edges, and assign an initial confidence weight (between 0 and 1) to each relation edge.

[0206] The fault physics knowledge graph constructed in this scheme serves as the external memory of the meta-learner, used for causal reasoning and hypothesis verification during the diagnostic process.

[0207] In a feasible solution, such as Figure 3 As shown, the steps for constructing a preset fault identification model include:

[0208] S201. Obtain the first number of first historical sample data for different sample devices;

[0209] S202. In response to the first quantity being less than the preset quantity, a preset sample construction method is used to expand and construct the second quantity of second historical sample data corresponding to different sample devices.

[0210] The constructed second historical sample data is a result that conforms to real physical laws and has good reliability.

[0211] The first historical sample data includes the first sample association parameters of the multimodal sample device and the corresponding first sample device fault label data; the second historical sample data includes the second sample association parameters of the multimodal sample device and the corresponding second sample device fault label data.

[0212] Alternatively, the first historical sample data includes the first sample fusion parameters obtained by fusing the first sample association parameters of the multimodal sample devices, and the corresponding first sample device fault label data; the second historical sample data includes the second sample fusion parameters obtained by fusing the second sample association parameters of the multimodal sample devices, and the corresponding second sample device fault label data.

[0213] Among them, the first sample equipment fault label data and the second sample equipment fault label data are sample fault information of the sample equipment, including bearing inner ring wear, etc.

[0214] S203. Obtain a sample training set based on the first historical sample data and the second historical sample data;

[0215] S204. Train the preset model based on the sample training set to obtain a preset fault identification model for predicting fault conditions in any target device.

[0216] In scenarios where samples are scarce or even zero, such as to address the problem of scarce samples in early equipment failures, it is necessary to expand the available sample data to ensure the feasibility, stability, and accuracy of model training.

[0217] In this solution, when the existing sample size is less than the preset number, a preset sample construction method is automatically used to expand the sample data to obtain a number of samples that meet the conditions. This allows the solution to work reliably under conditions of scarce data, effectively ensuring the efficiency, feasibility, and reliability of model training. In other words, through physically guided meta-learning, the solution can quickly adapt to and identify new faults even with only a very small number of samples (e.g., 1-5 samples), effectively solving the pain point of scarce fault samples in industry.

[0218] In one feasible embodiment, step S103 includes:

[0219] The actual correlation parameters of different modalities are asynchronously fused in the time-frequency domain to obtain the actual fused parameter data;

[0220] The actual fusion parameter data is input into a pre-trained preset fault identification model to obtain the target fault identification result.

[0221] In one feasible solution, the step of expanding to obtain a second number of second historical sample data corresponding to different sample devices using a preset sample construction method includes:

[0222] By employing a pre-defined digital simulation algorithm and the physical laws followed by the sample devices, a second number of second historical sample data corresponding to different sample devices are obtained.

[0223] The specific type of digital simulation algorithm used is not limited, as long as it can be implemented; therefore, it will not be elaborated here.

[0224] In this scheme, data simulation is used to simulate the historical sample data corresponding to the sample device. These simulated extended historical sample data all conform to the physical laws followed by the sample device, have high reliability, and ensure the efficiency of sample data extension processing.

[0225] In one feasible embodiment, step S104 includes:

[0226] Obtain the faulty component and its corresponding fault identification information from the target fault identification results;

[0227] In the digital twin, the faulty component is highlighted using the first preset display method.

[0228] In the first preset area of ​​the floating display interface in the digital twin, the fault identification information is displayed using a matching second preset display method;

[0229] In the second preset area of ​​the floating display interface in the digital twin, the actual parameter data of each modality are displayed using their respective matching third preset display method;

[0230] The first preset display method includes highlighting and other methods, while the second and third preset display methods are various types of visual charts and other methods.

[0231] The location and size of the first and second preset areas within the field of vision can be determined or adjusted according to actual needs.

[0232] This solution can automatically determine which specific component is faulty and the specific fault situation based on the target fault identification results. For example, if the faulty component is identified as a bearing, and the outer ring of the bearing is suspected to be faulty, the diagnostic result is visualized in the MR environment. Field engineers can directly see the displayed faulty equipment content by wearing MR glasses, such as "suspected bearing outer ring fault, confidence level 75%", instead of a simple text alarm.

[0233] More specifically, within the MR field of view of the MR glasses, a virtual, highlighted faulty component (such as a flashing red bearing model) is precisely superimposed on the corresponding location of the real device, allowing relevant personnel to intuitively and clearly understand the faulty object. Simultaneously, multimodal data features related to the fault, such as characteristic frequency peaks in the vibration spectrum, energy distribution of acoustic signals, and hot spots in thermal imaging, are displayed synchronously around the virtual faulty component in the form of floating visual charts, further enhancing the intuitive presentation and improving the interactivity and usability.

[0234] In a feasible solution, the specific fault conditions are intuitively presented in the MR field of view, which can facilitate direct guidance on-site for the repair and other operations of the corresponding faulty components.

[0235] In this solution, based on the intuitive display of the fault, it can be linked to the actual physical scenario to directly guide relevant personnel to repair the fault remotely, thereby further ensuring the efficiency and convenience of fault handling and effectively optimizing the existing fault handling process.

[0236] In one feasible solution, the processing method further includes:

[0237] Determine the actual interaction requirements based on external interactive operations;

[0238] External interaction operations include gestures, voice interaction, and other methods.

[0239] Based on actual interaction needs, a preset fault physical knowledge graph is invoked to generate causal tracing path information for describing the reverse reasoning that caused the fault. The causal tracing path information is then displayed in the third preset area of ​​the floating display interface in the digital twin using the fourth preset display method.

[0240] In this solution, after viewing the virtual fault details displayed in the MR device, relevant personnel can interact with specific objects (virtual faulty components, virtual fault information) in the MR environment through gestures, voice, etc. Based on the specific interactive operations, the corresponding interaction requirements are analyzed, and a pre-built fault physical knowledge graph is invoked for auxiliary analysis and processing. This efficiently and with high quality outputs causal tracing path information describing the reverse reasoning that caused the fault, and further displays it in the MR environment of the MR device. This not only allows viewing but also enables further fault analysis and processing in conjunction with the pre-built fault physical knowledge graph, effectively optimizing the fault analysis process and ensuring its convenience and reliability. The diagnostic results are supported by physical mechanisms and can be intuitively traced through the MR environment, breaking the "black box" problem of deep learning and greatly improving the interpretability and credibility of the results.

[0241] In one feasible solution, the processing method further includes:

[0242] To determine or correct the target fault analysis data;

[0243] Acquire the processed target fault analysis data;

[0244] In this solution, the usability of target fault analysis data can be directly determined, or it can be corrected to ensure high-quality target fault analysis data. This enables the use of MR technology to transparently deliver key information to human experts, allowing them to make final decisions based on their experience and intuition, and thereby continuously optimize the fault analysis results for the equipment.

[0245] In one feasible solution, the processing method further includes:

[0246] To determine or correct the target fault analysis data;

[0247] The processed target fault analysis data is used as the actual fault label for the corresponding multimodal actual correlation parameter;

[0248] New sample training data is generated based on the actual correlation parameters of the multimodal and the corresponding actual fault labels.

[0249] When the number of new sample training data reaches a preset value, the model is updated and iterated to obtain a new preset fault identification model.

[0250] In this solution, new model training samples can be constructed based on target fault analysis data determined through human feedback. This allows for periodic or ad-hoc iterative training of the model, further optimizing the training to obtain a more reliable and stable pre-set fault identification model. In essence, on-site feedback from human experts is structurally integrated into model optimization, enabling the corresponding fault identification system to continuously evolve and improve its intelligence level over time and with the accumulation of experience.

[0251] Specifically, when relevant personnel see characteristic frequency peaks in the vibration spectrum, energy distribution of acoustic signals, and hot spots in thermal imaging, which appear as floating visual charts surrounding a virtual faulty component, they can point to a characteristic frequency peak and ask, "What is the source of this frequency component?" The system will then call up the fault physics knowledge graph, automatically generate a causal tracing path, and display it in the MR field of view in the form of a graph, for example: "Rotational speed (1x) → Gear meshing frequency → Bearing outer ring passing frequency".

[0252] The engineer's on-site confirmation or correction feedback on the cause-and-effect tracing path (for example, the engineer confirms that the bearing is faulty after disassembling the equipment, or finds that the false alarm is caused by a loose sensor, etc.) will be recorded in real time and fed back to the meta-learner as a "golden sample".

[0253] This feedback can be used to fine-tune the diagnostic model and update and correct the weights in the physical consistency loss function, thereby enabling human expert knowledge to calibrate the physical model. This forms a continuous learning closed loop of diagnosis → MR visualization → manual verification → model enhancement, effectively ensuring high accuracy in fault identification in equipment.

[0254] In one feasible solution, the processing method further includes:

[0255] To determine or correct the target fault analysis data;

[0256] Based on the processed target fault analysis data, the confidence weight values ​​of the corresponding causal relationship edges in the preset fault physical knowledge graph are adjusted to update the preset fault physical knowledge graph.

[0257] In this scheme, the pre-constructed physical knowledge graph of the fault can be updated in a timely manner based on the target fault analysis data determined by human feedback, thus further ensuring the quality of the causal tracing path information determined in subsequent model training and interactive calls.

[0258] In one feasible solution, the processing method further includes:

[0259] Analyze several identified reference cases within a preset time period;

[0260] In response to a systematic deviation between the actual characteristic frequency and the theoretical characteristic frequency of a certain type of fault in any reference case, frequency compensation coefficients are generated.

[0261] The preset fault identification model is corrected based on the frequency compensation coefficient.

[0262] This can be determined through a pre-constructed mapping table of deviation-frequency compensation coefficients; of course, other feasible methods can also be used, which will not be elaborated here.

[0263] In this solution, after obtaining several reference cases determined by experts over a period of time, case analysis is performed, and based on the deviation between the actual characteristic frequency and the theoretical characteristic frequency of a certain type of fault, the corresponding frequency compensation coefficient is automatically generated to further optimize the preset fault identification model, thereby correcting the preset fault identification model and ensuring the accuracy and reliability of the output results in subsequent actual fault identification scenarios.

[0264] The online diagnostic and interactive MR tracing process of this embodiment is further explained below:

[0265] (1) Automatic triggering of fault identification, real-time data fusion and fault identification processing

[0266] The system continuously monitors the multimodal data stream. When the adaptive threshold (such as vibration kurtosis value > 4.5) of any one or more modes of signal is exceeded, the entire fault identification process is automatically triggered.

[0267] Specifically, it is necessary to extract data from a time window (e.g., 2 seconds) backward from the current time point as input data for the meta-learner (fusion parameter data of the actual correlation parameters of the multimodal models) to ensure comprehensive, timely and accurate model fault identification, especially for early fault identification.

[0268] (2) Visualization of fault identification results in MR environment

[0269] Engineers wearing MR glasses can see the following in the MR field of view:

[0270] Faulty parts highlighted: suspected faulty bearings or gears on the equipment are superimposed on the actual parts with a semi-transparent red pulse flashing effect.

[0271] Multimodal data panel: Next to the faulty component, three parallel spectrum graphs are displayed in a floating manner: Vibration spectrum: spectral peaks that match the theoretical fault frequency are highlighted in yellow; Acoustic spectrum: displays an acoustic energy cloud map, with abnormal frequency bands highlighted; Thermal imaging sequence: plays a dynamic GIF image (an animated graph) showing the temperature rise of the component over the past 30 seconds.

[0272] Fault identification conclusion card: Displays an example such as "early spalling of bearing outer ring, confidence level 72%", and lists the main evidence.

[0273] (3) Causal tracing of human-machine collaboration

[0274] Gesture interaction: An engineer taps in the air to detect an unexplained peak in the vibration spectrum;

[0275] Causal tracing: The system queries the physical knowledge graph of the fault in real time, generates and displays a reverse reasoning chain;

[0276] For example, the current peak frequency (157Hz) → possible source: bearing outer ring passing frequency (theoretical value 156.7Hz) → affected component: bearing model 6312 → related failure modes: outer ring peeling, poor lubrication;

[0277] Voice interaction: Engineer's voice input: "Confirm diagnosis, the cause of the fault is poor lubrication."

[0278] Online model enhancement: The system packages the current data segment (vibration, acoustics, thermal imaging) with the fault label confirmed by the engineer (poor lubrication) as a new gold sample. This gold sample is fed into the incremental learning queue of the meta-learner for iterative model updates. Simultaneously, the system automatically adjusts the confidence weight of the relationship edge between poor lubrication and abnormal temperature rise in the knowledge graph (e.g., increasing it from 0.7 to 0.8) to further ensure the accuracy and reliability of subsequent model updates and the back-inference chain obtained by the system from querying the fault physics knowledge graph.

[0279] (4) Closed-loop feedback and model evolution

[0280] Periodic model fine-tuning: Every time the system accumulates 10 gold samples confirmed by on-site experts, it automatically initiates a lightweight meta-learner fine-tuning process to update the model parameters, making it more sensitive to these confirmed failure modes.

[0281] Physical model calibration: The system periodically analyzes all expert-confirmed cases. If a systematic deviation is found between the actual characteristic frequency of a certain type of fault and the characteristic frequency calculated theoretically (for example, due to installation errors), a frequency compensation coefficient will be automatically generated and applied to the subsequent calculation of physical consistency loss L_physics to achieve self-correction of the physical model and further ensure the stability and generalization ability of the model.

[0282] The implementation principle of the equipment failure handling scheme in this embodiment will be explained in detail below with specific examples:

[0283] A 1.5MW wind turbine generator set at a wind farm had its gearbox (model: FL600) exhibit slightly increased high-frequency vibration during a routine inspection, but this did not reach the alarm threshold. Traditional fault monitoring systems were unable to determine if an early-stage fault was present.

[0284] Based on this, the fault handling process for gearboxes in wind power plants in this embodiment is as follows:

[0285] (1) Initialization phase

[0286] (11) Constructing a digital twin

[0287] A handheld 3D scanner is used to scan the gearbox and create a precise 3D model that includes components such as the main shaft, planetary gears, parallel shaft gears, and bearings. In the MR device, the virtual model is aligned with the physical object with high precision by using pre-attached QR code markers.

[0288] (12) Constructing a fault knowledge graph

[0289] Node relationships:

[0290] (Planetary gear peeling) --[excitation]--> (Planetary gear passing frequency);

[0291] (Planetary gear passing frequency) --[Calculation formula]--> (Revolution frequency × Number of teeth × Characteristic coefficient);

[0292] (Planetary gear peeling) --[Accompanying characteristics]--> (Vibration spectrum sidebands);

[0293] (Planetary gear peeling) --[Temperature characteristics]--> (Local temperature rise of 0.5-1.5℃);

[0294] (2) Deploy sensors

[0295] Vibration sensors: Two triaxial accelerometers are installed in the bearing housings of the input and output shafts of the gearbox;

[0296] Acoustic array: A four-microphone array is arranged inside the cabin to form an acoustic imaging system;

[0297] Thermal imager: A fixed infrared thermal imager is installed opposite the gearbox observation window.

[0298] (3) Fault handling and MR interactive tracing

[0299] (31) Fault identification

[0300] The system detected that the kurtosis value of the planetary vibration signal increased from 2.1 to 3.8, and the acoustic signal energy increased by 6dB in the 1250Hz frequency band. At this time, it automatically triggered the fusion of these multimodal data and input them into the meta-learner for fault identification.

[0301] The model selects three historical planetary gear failure samples from the support set, combines them with current multimodal data, and performs rapid adaptation under physical constraints, outputting the result: "Early planetary gear spalling, confidence level 68%"; physical consistency verification: the detected frequency component has an error of <2% compared to the theoretical planetary gear passing frequency.

[0302] (32) MR interactive interface display

[0303] In the 3D model of the gearbox, the planetary gear section flashes orange. The right panel displays the vibration spectrum: a clear spectral peak appears at the frequency through which the planetary gear passes; the acoustic cloud map: energy is concentrated in the 1250Hz frequency band; and the thermal imaging sequence: the temperature rise in the planetary gear area is 0.8℃.

[0304] (33) Interaction between field engineers

[0305] Engineers use gestures to select abnormal peaks displayed in the MR environment and ask, "What is the source of this frequency component?"; they query the constructed fault physics knowledge graph in real time, automatically generating a causal tracing path and displaying it in the MR field of view in graph form, such as "Real-world diagnostic criteria: frequency matching degree 89%, consistent temperature change trend, and consistent acoustic energy distribution"; engineers confirm via voice, such as "Record the current information as the gold sample" to update the meta-learner and ensure the timely and reliable updating of the preset fault recognition model.

[0306] In addition, based on a 68% confidence level and clear fault location, the electric field was scheduled to conduct an unpacking inspection on the third day during a period of low wind; the inspection results confirmed that an early spalling area of ​​3mm×5mm appeared on the surface of the planetary gear;

[0307] The resulting economic benefits are: (1) Avoiding losses: The fault was detected 30 days in advance, avoiding a potential serious gearbox damage accident; among which, the maintenance cost savings were RMB 150,000 (early maintenance) compared to RMB 800,000 (gearbox replacement); (3) Power generation revenue: A planned shutdown of 6 hours was compared to an unexpected shutdown of at least 3 days; among which, the power generation loss was reduced by approximately RMB 250,000.

[0308] Furthermore, the complete data package for this fault identification was marked as "early planetary gear peeling - confirmed case", which can be used by the meta-learner for incremental learning during the nighttime maintenance window; and the weight of "planetary gear peeling → temperature rise feature" in the knowledge graph was increased from 0.65 to 0.73 to ensure further timely optimization and improvement of the model.

[0309] Example 3

[0310] like Figure 4 As shown, the device malfunction handling apparatus of this embodiment includes:

[0311] Digital twin processing module 1 is used to construct a digital twin of the target device and visualize the digital twin in a mixed reality device;

[0312] Multimodal parameter acquisition module 2 is used to acquire the actual correlation parameters of the target device in the preset operation process.

[0313] Fault identification module 3 is used to obtain the target fault identification result based on the actual correlation parameters and the pre-trained preset fault identification model;

[0314] Fault display module 4 is used to display the target fault identification results in the digital twin according to preset display rules to obtain the target display content;

[0315] The interaction processing module 5 is used to respond to external interactive operations received based on the target displayed content, and obtain target fault analysis data that matches the target fault identification results based on the pre-built preset fault physical knowledge graph.

[0316] This solution constructs a digital twin of the device and visualizes it within the MR device. Then, multimodal operating parameters are synchronously collected using different types of sensors. Based on this modal data, the solution dynamically and promptly identifies faulty components and fault conditions within the target device, and presents this information in a targeted, intuitive, and clear manner within the MR environment. Simultaneously, it can receive user interactions via gestures, voice, and other means, triggering the invocation of a pre-defined fault physical knowledge graph. This allows for rapid and accurate causal reasoning to trace the corresponding fault. Essentially, this constructs a closed-loop fault identification solution that enables real-time interaction and mutual reinforcement between the physical world, digital twin, fault identification model, MR interaction, and human experts. This effectively ensures the accuracy and efficiency of fault identification (such as early-stage faults) in any target device. Furthermore, the MR environment-based display and interaction provide a more intuitive understanding of the fault, effectively enhancing the interactive experience in fault handling scenarios.

[0317] For the system embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The system embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure solution according to actual needs.

[0318] Example 4

[0319] like Figure 5 As shown, the equipment failure handling device in this embodiment is a further improvement on embodiment 3, including:

[0320] In one feasible solution, the digital twin processing module 1 includes:

[0321] The 3D model acquisition unit is used to acquire the 3D physical model of the target device;

[0322] The digital twin building unit is used to register the spatial coordinates of a 3D physical model with the target device in the real environment in a mixed reality device, so as to build a digital twin of the target device in the mixed reality space.

[0323] In one feasible solution, different types of sensing devices are deployed at different preset locations in the real environment where the target device is located;

[0324] The multimodal parameter acquisition module 2 is used to acquire the actual correlation parameters of the target device in the preset operation process by using different types of sensing devices;

[0325] In one feasible embodiment, the identification device further includes a knowledge graph construction module 6, which comprises:

[0326] The parameter acquisition unit is used to acquire several dimensional characterization parameters of each physical component in the target device.

[0327] The causal relationship acquisition unit is used to acquire the causal relationships between different physical components;

[0328] The knowledge graph construction unit is used to construct a structured, pre-defined physical knowledge graph of faults by using several dimensions of representation parameters as entity nodes and causal relationships as causal relationship edges.

[0329] In one feasible embodiment, the identification device further includes a prediction model building module 7, which comprises:

[0330] The first sample module unit is used to acquire a first number of first historical sample data from different sample devices;

[0331] The sample expansion unit is used to expand and construct a second number of second historical sample data corresponding to different sample devices in response to the first number being less than the preset number, using a preset sample construction method;

[0332] The first historical sample data includes the first sample association parameters of the multimodal sample device and the corresponding first sample device fault label data; the second historical sample data includes the second sample association parameters of the multimodal sample device and the corresponding second sample device fault label data.

[0333] Alternatively, the first historical sample data includes the first sample fusion parameters obtained by fusing the first sample association parameters of the multimodal sample devices, and the corresponding first sample device fault label data; the second historical sample data includes the second sample fusion parameters obtained by fusing the second sample association parameters of the multimodal sample devices, and the corresponding second sample device fault label data.

[0334] The training set acquisition unit is used to obtain a sample training set based on the first historical sample data and the second historical sample data.

[0335] The model training unit is used to train a preset model based on a sample training set to obtain a preset fault identification model for predicting fault conditions in any target device.

[0336] In one feasible scheme, the sample expansion unit is used to expand to obtain a second number of second historical sample data corresponding to different sample devices by using a preset digital simulation algorithm and the physical laws followed by the sample devices;

[0337] In one feasible scheme, the model training unit is also used to encode several preset physical principles followed by the sample device to obtain corresponding encoded data, and use the encoded data as a regularization constraint term of the preset model for model training.

[0338] In a feasible approach, the loss function in the pre-defined model includes a physical consistency loss function and a prediction minimization loss function.

[0339] In one feasible solution, the fault display module 4 is also used to acquire the faulty component and the corresponding fault identification information in the target fault identification result; and to highlight the faulty component in the digital twin using a first preset display method;

[0340] In one feasible solution, the fault display module 4 is also used to display fault identification information in a first preset area of ​​the floating display interface in the digital twin using a matching second preset display method;

[0341] In one feasible solution, the fault display module 4 is also used to display the actual parameter data of each modality in the second preset area of ​​the floating display interface in the digital twin using a third preset display method that matches each modality.

[0342] In one feasible solution, the interaction processing module 5 is also used to determine the actual interaction requirements based on external interaction operations; according to the actual interaction requirements, it calls the preset fault physical knowledge graph to generate causal tracing path information for describing the reverse reasoning that caused the fault, and calls the fault display module 4.

[0343] The fault display module 4 is also used to display the causal tracing path information in the third preset area of ​​the floating display interface in the digital twin using the fourth preset display method.

[0344] In one feasible embodiment, the identification device further includes:

[0345] The fault analysis data processing module 8 is used to determine or correct the target fault analysis data;

[0346] Analysis data output module 9 is used to acquire processed target fault analysis data;

[0347] In one feasible embodiment, the identification device further includes:

[0348] The new sample data formation module 10 is used to take the processed target fault analysis data as the actual fault label of the corresponding multimodal actual correlation parameter; and to form new sample training data based on the multimodal actual correlation parameter and the corresponding actual fault label.

[0349] The model update module 11 is used to update and iterate the model in response to the cumulative number of new sample training data reaching a preset value, so as to obtain a new preset fault identification model.

[0350] In one feasible embodiment, the identification device further includes:

[0351] The knowledge graph update module 12 is used to adjust the confidence weight values ​​of the corresponding causal relationship edges in the preset fault physical knowledge graph based on the processed target fault analysis data, so as to update the preset fault physical knowledge graph.

[0352] In one feasible embodiment, the identification device further includes:

[0353] The reference case analysis module 13 is used to analyze several identified reference cases within a preset time period;

[0354] The compensation coefficient generation module 14 is used to generate frequency compensation coefficients in response to a systematic deviation between the actual characteristic frequency and the theoretical characteristic frequency of a type of fault in any reference case.

[0355] Model correction module 15 is used to correct the preset fault identification model based on the frequency compensation coefficient.

[0356] In one feasible solution, the fault identification module 3 is used to asynchronously fuse the actual correlation parameters of different modalities in the time-frequency domain to obtain actual fused parameter data; and input the actual fused parameter data into a pre-trained preset fault identification model to obtain the target fault identification result.

[0357] In one feasible approach, the pre-defined model includes a meta-learner;

[0358] In one feasible solution, the target equipment includes rotating machinery.

[0359] In one feasible approach, external interaction includes gestures and / or voice interaction.

[0360] For the system embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The system embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure solution according to actual needs.

[0361] Example 5

[0362] like Figure 6 As shown, the device fault management system 100 of this embodiment includes a mixed reality device 200 and a device fault processing device 300 as in embodiment 3 or 4.

[0363] The equipment fault management system in this embodiment integrates the aforementioned equipment fault processing device. It is equivalent to constructing a closed-loop processing system for fault identification that enables real-time interaction and mutual enhancement between the "physical world - digital twin - fault identification model - MR interaction - human experts". This effectively ensures the accuracy and efficiency of fault identification (such as early faults) in any target device. Furthermore, the display and interaction based on the MR environment make the fault more intuitive to understand, effectively improving the interactive experience of the fault handling scenario, thereby effectively improving the overall product performance of the equipment fault management system.

[0364] Example 6

[0365] Figure 7 This is a schematic diagram of the structure of an electronic device according to an example embodiment of the present disclosure. The electronic device includes a memory, a processor, and a computer program stored in the memory and used to run on the processor. When the processor executes the computer program, it implements the method described in any of the above embodiments. Figure 7 The electronic device 90 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0366] like Figure 7 As shown, the electronic device 90 can be manifested as a general-purpose computing device, such as a server device. The components of the electronic device 90 may include, but are not limited to: at least one processor 91, at least one memory 92, and a bus 93 connecting different system components (including memory 92 and processor 91).

[0367] Bus 93 includes a data bus, an address bus, and a control bus.

[0368] The memory 92 may include volatile memory, such as random access memory (RAM) 921 and / or cache memory 922, and may further include read-only memory (ROM) 923.

[0369] The memory 92 may also include a program tool 925 (or utility) having a set (at least one) program module 924, such program module 924 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0370] The processor 91 executes various functional applications and data processing, such as the methods provided in any of the above embodiments, by running computer programs stored in the memory 92.

[0371] Electronic device 90 can also communicate with one or more external devices 94 (e.g., keyboard, pointing device, etc.). This communication can be performed through input / output (I / O) interface 95. Furthermore, electronic device 90 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public network, such as the Internet) via network adapter 96. As shown, network adapter 96 communicates with other modules of electronic device 90 via bus 93. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with electronic device 90, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems.

[0372] It should be noted that although several units / modules or sub-units / modules of the electronic device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0373] Example 7

[0374] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method provided in any of the above embodiments.

[0375] The readable storage medium may be more specifically adopted, including but not limited to: portable disk, hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.

[0376] Example 8

[0377] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the method described in any of the above embodiments.

[0378] The program code for executing the computer program product of this disclosure can be written in any combination of one or more programming languages, and the program code can be executed entirely on a user device, partially on a user device, as a stand-alone software package, partially on a user device and partially on a remote device, or entirely on a remote device.

[0379] While specific embodiments of this disclosure have been described above, those skilled in the art should understand that these are merely illustrative examples, and the scope of protection of this disclosure is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of this disclosure, but all such changes and modifications fall within the scope of protection of this disclosure.

Claims

1. A method for handling equipment malfunctions, characterized in that, The processing method includes: A digital twin of the target device is constructed and visualized in a mixed reality device; The actual correlation parameters of the target device in the preset operation process are collected; Based on the actual correlation parameters and the pre-trained preset fault identification model, the target fault identification result is obtained; The target fault identification results are displayed in the digital twin according to preset display rules to obtain the target display content; In response to receiving an external interactive operation based on the target displayed content, target fault analysis data matching the target fault identification result is obtained based on a pre-constructed preset fault physical knowledge graph.

2. The equipment failure handling method as described in claim 1, characterized in that, The steps for constructing a digital twin of the target device include: Obtain the three-dimensional physical model of the target device; In the mixed reality device, the three-dimensional physical model is spatially registered with the target device in the real environment to construct the digital twin of the target device in the mixed reality space; And / or, Different types of sensing devices are deployed at different preset locations in the real environment where the target device is located; The step of acquiring the actual correlation parameters of the multimodal modes of the target device during a preset operating process includes: Using different types of sensing devices, the actual correlation parameters of the target device in the preset operation process are collected respectively in the multimodal mode; And / or, The steps for constructing the preset fault physical knowledge graph include: Obtain several dimensional characterization parameters for each physical component in the target device; To obtain the causal relationships between the different physical components; By using the representation parameters of several dimensions as entity nodes and the causal relationships as causal relationship edges, a structured preset fault physical knowledge graph is constructed.

3. The equipment failure handling method as described in claim 2, characterized in that, The steps for constructing the preset fault identification model include: Obtain the first number of first historical sample data from different sample devices; In response to the first quantity being less than a preset quantity, a preset sample construction method is used to expand and construct a second quantity of second historical sample data corresponding to different sample devices; The first historical sample data includes the first sample association parameters of the multimodal nature of the sample device and the corresponding first sample device fault label data; the second historical sample data includes the second sample association parameters of the multimodal nature of the sample device and the corresponding second sample device fault label data. Alternatively, the first historical sample data includes the first sample fusion parameters obtained by fusing the first sample association parameters of the multimodal data of the sample device, and the corresponding first sample device fault label data; the second historical sample data includes the second sample fusion parameters obtained by fusing the second sample association parameters of the multimodal data of the sample device, and the corresponding second sample device fault label data. A sample training set is obtained based on the first historical sample data and the second historical sample data; The preset model is trained based on the sample training set to obtain the preset fault identification model used to predict the fault condition in any target device.

4. The equipment failure handling method as described in claim 3, characterized in that, The step of expanding to obtain a second number of second historical sample data corresponding to different sample devices using a preset sample construction method includes: By employing a preset digital simulation algorithm and the physical laws followed by the sample devices, the second number of second historical sample data corresponding to different sample devices is obtained; And / or, The model training process of the preset fault identification model also includes: The sample device is encoded according to several preset physical principles to obtain corresponding encoded data, and the encoded data is used as a regularization constraint term of the preset model for model training. And / or, The loss functions in the preset model include a physical consistency loss function and a prediction minimization loss function.

5. The method for handling equipment failure as described in any one of claims 1 to 4, characterized in that, The step of displaying the target fault identification result in the digital twin according to a preset display rule to obtain the target display content includes: Obtain the faulty component and corresponding fault identification information from the target fault identification result; The faulty component is highlighted in the digital twin using a first preset display method; And / or, In the first preset area of ​​the floating display interface in the digital twin, the fault identification information is displayed using a matching second preset display method; And / or, The processing method further includes: In the second preset area of ​​the floating display interface in the digital twin, the actual parameter data of each modality are displayed using their respective matching third preset display methods; And / or, The processing method further includes: Based on the aforementioned external interaction operations, determine the actual interaction requirements; Based on the actual interaction requirements, the preset fault physical knowledge graph is invoked to generate causal tracing path information for describing the reverse reasoning that caused the fault, and the causal tracing path information is displayed in the third preset area of ​​the floating display interface in the digital twin using a fourth preset display method.

6. The method for handling equipment failure as described in any one of claims 1 to 4, characterized in that, The processing method further includes: The target fault analysis data is determined or corrected; Obtain the processed target fault analysis data; And / or, The processed target fault analysis data is used as the actual fault label for the corresponding multimodal actual correlation parameter; New sample training data is formed based on the actual correlation parameters of the multimodal model and the corresponding actual fault labels; In response to the cumulative number of new sample training data reaching a preset value, the model is updated and iterated to obtain a new preset fault identification model; And / or, Based on the processed target fault analysis data, the confidence weight values ​​of the corresponding causal relationship edges in the preset fault physical knowledge graph are adjusted to update the preset fault physical knowledge graph. And / or, The processing method further includes: Analyze several identified reference cases within a preset time period; In response to a systematic deviation between the actual characteristic frequency and the theoretical characteristic frequency of a type of fault in any of the reference cases, a frequency compensation coefficient is generated. The preset fault identification model is corrected based on the frequency compensation coefficient.

7. The method for handling equipment malfunctions as described in claim 3 or 4, characterized in that, The step of obtaining the target fault identification result based on the actual correlation parameters and the pre-trained preset fault identification model includes: The actual correlation parameters of different modalities are asynchronously fused in the time-frequency domain to obtain actual fused parameter data; The actual fusion parameter data is input into the pre-trained preset fault identification model to obtain the target fault identification result; And / or, The preset model includes a meta-learner; And / or, The target equipment includes rotating machinery; And / or, The external interaction operations include gestures and / or voice interaction methods.

8. A device for handling equipment malfunctions, characterized in that, The processing device includes: A digital twin processing module is used to construct a digital twin of the target device and visualize the digital twin in a mixed reality device; A multimodal parameter acquisition module is used to acquire the actual correlation parameters of the target device in the preset operation process. The fault identification module is used to obtain the target fault identification result based on the actual correlation parameters and the pre-trained preset fault identification model; The fault display module is used to display the target fault identification result in the digital twin according to a preset display rule to obtain the target display content; The interaction processing module is used to respond to external interactive operations received based on the target displayed content, and obtain target fault analysis data that matches the target fault identification result based on a pre-constructed preset fault physical knowledge graph.

9. A management system for equipment failure, characterized in that, The management system includes a mixed reality device and a device for handling device malfunctions as described in claim 8.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and for running on the processor, characterized in that, When the processor executes the computer program, it implements the device fault handling method according to any one of claims 1 to 7.

Images