An AR intelligent glasses-based fault handling assistance method and system

Abstract

The application provides a fault processing auxiliary method and system based on AR intelligent glasses, relates to the technical field of auxiliary fault processing, and the method comprises the following steps: scanning a target equipment unique ID by using AR intelligent glasses, calling a maintenance work flow from a pre-constructed risk library according to an analysis result, analyzing maintenance steps in the maintenance work flow and marking risk steps and adding a blocking process; if the current step is a risk step, triggering blocking, releasing the blocking after receiving a confirmation feedback from a maintenance end, and continuing to execute the risk step in the maintenance work flow. By adding the blocking process before the risk step and releasing the blocking process after receiving the confirmation feedback from the maintenance end, the application can effectively improve the safety of the maintenance process.

Description

Technical Field

[0001] This application relates to the technical field of assisting in fault handling, and in particular to a fault handling assisting method and system based on AR smart glasses. Background Technology

[0002] In a power system, the distribution network, as the power network directly facing users, is crucial for ensuring the normal order of social production and life. The distribution network comprises numerous devices, such as transformers, switchgear, and cables. During long-term operation, these devices inevitably experience various faults, requiring timely maintenance to restore normal operation, reduce power outage time and scope, and improve power supply reliability. Currently, the maintenance of distribution network equipment mainly relies on manual operation. Maintenance personnel typically rely on experience and familiarity with the equipment, following established maintenance procedures step by step.

[0003] Chinese invention patent application CN115510253A provides an AR-assisted maintenance method and terminal for substation equipment based on a three-dimensional knowledge graph. After a fault is detected, the patent determines the target maintenance plan for the user to view. However, for some high-risk maintenance steps, the patent only emphasizes their importance and precautions through text or pictures, and cannot forcibly intervene or pause when maintenance personnel actually perform the steps. Maintenance personnel may skip the necessary inspection and confirmation steps and directly proceed to the next step due to negligence or eagerness to complete the task, thus creating potential safety hazards. Summary of the Invention

[0004] To improve safety during fault diagnosis and repair, this application provides a fault handling assistance method and system based on AR smart glasses.

[0005] Firstly, this application provides a fault handling assistance method based on AR smart glasses, employing the following technical solution:

[0006] A fault handling assistance method based on AR smart glasses includes the following steps:

[0007] AR smart glasses are used to scan the unique ID of the target device. Based on the parsing result of the unique ID, the maintenance workflow of the target device is retrieved from the pre-built risk database. The maintenance workflow is broken down into maintenance steps. Maintenance steps with risks are marked as risk steps, and blocking processes are added before the risk steps.

[0008] If the current step is a risky step, a blocking process is triggered. Once the maintenance end confirms the risky step, the blocking is lifted, and the risky step in the maintenance workflow continues to be executed.

[0009] The application adopts the unique ID of the AR intelligent glasses scanning device, analyzes it, obtains an analysis result, retrieves the target device's maintenance workflow in a pre-constructed risk library according to the analysis result, then splits the maintenance workflow into maintenance steps, marks the steps with risks in the maintenance steps as risk steps, adds a blocking process before the risk steps, and the blocking process makes the risk steps unable to be activated before the risk steps are confirmed, so as to avoid operation skipping or illegal execution as much as possible, force the maintenance end to execute according to the preset maintenance workflow, reduce human randomness, and improve the safety of the maintenance process.

[0010] Optionally, before continuing to execute the risk steps in the maintenance workflow, the method further comprises:

[0011] Obtaining training data, the training data including historical data under normal working conditions and safety thresholds of various historical data, the historical data including historical device parameters, historical operation data and historical environment data, training the pre-constructed machine learning model with the training data to obtain a trained machine learning model;

[0012] Collecting real-time data of the target device, the real-time data including real-time device parameters, real-time operation data and real-time environment data, inputting the real-time data into the trained machine learning model, and outputting safety thresholds of various real-time data;

[0013] If the real-time data exceeds the range of the safety thresholds of the real-time data, the blocking process is triggered.

[0014] The application trains the machine learning model with the historical data and the safety thresholds of various historical data, processes the real-time data with the trained machine learning model to obtain the safety thresholds of various real-time data, and predicts the safety thresholds based on the trained machine learning model, which is more suitable for actual working conditions than fixed thresholds, improves the accuracy of various safety thresholds, and then determines whether the real-time data is safe, and then triggers the blocking process when the real-time data exceeds the safety threshold. Once the process is blocked, the maintenance personnel must pause the operation and troubleshoot the problem, so that the risk is actively eliminated rather than passively bypassed, further improving the safety of the maintenance process.

[0015] Optionally, the method further comprises:

[0016] Dividing the historical data in the training data with a preset time window to obtain divided historical data, denoted as first data; obtaining the historical data of the target device in a preset time window before the current time, denoted as second data;

[0017] The KL divergence of each first data and second data in a preset time window is calculated, a weighted quantile sum algorithm is adopted, a drift index is calculated based on all the KL divergences in the preset time window, and an alarm signal is sent when the drift index is greater than a preset drift threshold.

[0018] The application quantifies the degree of change of data features by comparing the distribution difference between the training data and the second data, and then aggregates the drift indexes calculated by the plurality of KL divergences to reflect the degree of change of the overall data distribution, and triggers an alarm when the drift index exceeds the preset threshold, indicating that there is a potential risk.

[0019] Optionally, after the blockage is removed, the method further comprises:

[0020] The real-time operation picture of the risk step by the maintenance end is acquired in real time through the AR smart glasses, the standard operation picture and the non-standard operation picture about the risk step in the risk library are called, the standard operation picture and the non-standard picture are taken as training samples, the CNN model is trained by using the training samples, the trained CNN model is obtained, the real-time operation picture is input into the trained CNN model, and a classification result is obtained; when the classification result does not meet the expectation, an alarm bounding box is pushed to the maintenance end through the AR smart glasses, and the maintenance end is instructed to re-maintain the risk step.

[0021] The application acquires the operation picture of the risk step by the maintenance end in real time through the AR smart glasses, captures the maintenance action and details in time, avoids the delay of information as much as possible, and monitors the maintenance process in real time. When the classification result of the CNN model does not meet the expectation, the application immediately pushes an alarm bounding box to the maintenance end through the AR smart glasses, so that the maintenance personnel can quickly understand the problem, without spending extra time to interpret complex reports or instructions, thereby improving the maintenance efficiency. The alarm bounding box pushed by the AR smart glasses can clearly indicate the problem position and nature, help the maintenance personnel to quickly take correct measures, avoid accidents as much as possible, and ensure the safe performance of the maintenance work.

[0022] The application trains the CNN model by calling the standard operation picture and the non-standard operation picture about the risk step in the risk library, which helps to improve the distinguishing ability of the CNN model for the standard operation and the non-standard operation. The application compares and analyzes the real-time operation picture with the standard operation picture, and classifies and judges by using the CNN model, so as to strictly control whether the maintenance process meets the standard requirement. When the classification result does not meet the expectation, the maintenance end is required to re-maintain in time, so as to correctly handle each risk step, improve the overall maintenance quality, and reduce the risk of safety accidents caused by improper operation.

[0023] Optionally, when the maintenance end re-inspects the risk step, the method further comprises:

[0024] The computer vision algorithm is used to capture the hand key points of the maintenance personnel, the hand key points are used to reconstruct the hand movement trajectory of the maintenance personnel by using the SLAM technology, the hand movement trajectory is time-aligned with the pre-stored standard operation trajectory in the risk library by using the DTW technology, and an initial similarity score is obtained.

[0025] The computer vision algorithm is used to capture the hand key points of the maintenance personnel, the hand key points are used to reconstruct the hand movement trajectory of the maintenance personnel by using the SLAM technology, the hand movement trajectory is time-aligned with the pre-stored standard operation trajectory in the risk library by using the DTW technology, and an initial similarity score is obtained.

[0026] Subsequently, the reconstructed hand movement trajectory is time-aligned with the pre-stored standard operation trajectory in the risk library by using the DTW (dynamic time warping) technology, an initial similarity score is obtained, when the initial similarity score is lower than the pre-set similarity score threshold, it is determined that the operation of the maintenance personnel does not conform to the standard, a red warning trajectory is superimposed in the AR interface, the AR technology can combine virtual information with a real scene, so that the maintenance personnel can intuitively see the difference between the hand movement trajectory and the standard trajectory during operation, the red warning trajectory can quickly attract the attention of the maintenance personnel, timely remind them that the operation is deviated and needs to be adjusted immediately, so that the maintenance personnel can find and correct problems in time during operation, improve the maintenance efficiency, and reduce the waste of time and resources.

[0027] By strictly monitoring the operation trajectory of the maintenance personnel and timely warning the operation that does not conform to the standard, the application can effectively prevent safety accidents caused by improper operation, and long-term application of the method is helpful for the maintenance personnel to gradually develop a standard safety operation consciousness.

[0028] Optionally, before superimposing the red warning trajectory in the AR interface, the method further comprises:

[0029] The risk step is decomposed to obtain a plurality of key links in the risk step, key points of each key link are extracted, a path planning algorithm is used, a maintenance posture path is generated based on all the key points, a final similarity score of a hand movement trajectory and the maintenance posture path is calculated, if the final similarity score is lower than a similarity score threshold, a red warning trajectory is superimposed in an AR interface; if the final similarity score is not lower than the similarity score threshold, a signal of successful maintenance is output.

[0030] The application decomposes the risk step to obtain a plurality of key links, each key link represents a step that needs to be focused on and accurately executed in the maintenance process, and then the key points of each key link are extracted, the key positions and action points that the maintenance personnel need to pay attention to in each key link are determined, which helps them to more accurately complete the task of each link and improves the standardization and accuracy of operation.

[0031] Since the installation environments of the target devices are different, the application generates a maintenance posture path based on all the key points by using a path planning algorithm, the maintenance posture path comprehensively considers the logical relationship and operation sequence between the key links and the reasonable movement trajectory of the hand in space, then the application calculates the final similarity score of the hand movement trajectory and the maintenance posture path, quantifies the difference between the actual operation of the maintenance personnel and the standard path, and then analyzes whether the actual operation of the maintenance personnel is risky in combination with the actual situation. If the final similarity score is lower than the similarity score threshold, a red warning trajectory is superimposed in the AR interface.

[0032] Optionally, after triggering the blocking process, the method further includes:

[0033] The duration of the blocking process is counted, when the duration of the blocking process exceeds a preset time threshold, an assistance request package is generated, the assistance request package is sent to a remote expert terminal through a 5G network, an AR collaboration channel is established, and the assistance request package includes: three-dimensional point cloud data of the target device in the risk step, real-time data of the target device, and a hash value of the historical operation record;

[0034] The expert calls a space anchor to lock the position of the risk step, and a three-dimensional arrow is marked at the position of the risk step in the AR interface of the maintenance terminal to guide the maintenance personnel to operate.

[0035] The application can automatically generate an assistance request package when the duration of the blocking process exceeds a preset time threshold, which helps to timely detect long-term stagnation problems in the maintenance process and avoid delaying the overall maintenance schedule due to long blocking time. The assistance request package contains three-dimensional point cloud data of the target device in the risk step, providing spatial information of the risk step for remote experts. Experts can more intuitively understand the specific location and surrounding environment of the risk step based on the three-dimensional point cloud data, so as to more accurately determine the problem and develop more effective solutions. Real-time data of the target device in the assistance request package enables experts to real-time master the running state and parameter changes of the device, combined with the situation of the risk step, experts can analyze the correlation between device failure and current operation, and provide more targeted guidance for maintenance personnel to avoid misjudgment and misoperation caused by incomplete information. The hash value of the historical operation record in the assistance request package provides data integrity and traceability for the maintenance process. Experts can verify the hash value to confirm the authenticity and integrity of the historical operation record, understand the operation of the maintenance personnel in the previous step, and better analyze the causes of the problem to improve the accuracy of diagnosis.

[0036] Subsequently, the application sends the assistance request package to the remote expert terminal through the 5G network and establishes an AR collaboration channel. With the help of the low-latency AR collaboration channel, experts can participate in the collaboration of the maintenance site in real time, shortening the response and intervention time of experts and improving the efficiency of problem solving.

[0037] Subsequently, the expert calls the spatial anchor to lock the position of the risk step. This operation can provide a precise reference point for the maintenance personnel in the AR environment. Then, the expert labels a three-dimensional arrow at the position of the risk step in the AR interface of the maintenance terminal, so that the maintenance personnel can operate according to the direction and position of the three-dimensional arrow, reducing the blindness and uncertainty of the operation and improving the accuracy and standardization of the operation.

[0038] Optionally, the method further comprises:

[0039] The spatial coordinates and rotation matrix of the three-dimensional arrow are recorded through the spatial anchor to generate a dynamic guidance sequence containing a time stamp;

[0040] When the maintenance personnel moves, the SLAM technology is used to real-time match the position of the risk step with the coordinate value of the spatial anchor, and the pose of the three-dimensional arrow is dynamically updated through the rigid transformation algorithm.

[0041] Optionally, when multiple experts intervene at the same time, a consensus algorithm is used to detect conflicts of the three-dimensional arrows labeled by each expert, including:

[0042] The Euclidean distance between the three-dimensional arrows labeled by each expert is calculated, and if the Euclidean distance is less than a preset distance threshold, an alarm signal is output.

[0043] The application records the spatial coordinates and rotation matrix of the three-dimensional arrow through the spatial anchor point, and generates a dynamic guidance sequence containing a timestamp. The spatial coordinates and rotation matrix can accurately describe the position and attitude of the three-dimensional arrow in three-dimensional space, and the dynamic guidance sequence containing the timestamp makes the guidance sequence have order in the time dimension, so that the maintenance personnel can receive the guidance in time sequence, reducing the degree of information confusion and improving the coherence and accuracy of the operation steps.

[0044] When the maintenance personnel move, the SLAM (Simultaneous Localization and Mapping) technology is used to match the position of the risk step with the coordinate value of the spatial anchor point in real time, and the pose of the three-dimensional arrow is dynamically updated through the rigid transformation algorithm. The SLAM technology can perceive the position changes of the maintenance personnel and the risk step in real time, and the rigid transformation algorithm can quickly adjust the position and direction of the three-dimensional arrow according to these changes, so that it always accurately points to the risk step, providing continuous and effective guidance for the maintenance personnel, adapting to the dynamic changes of personnel and equipment in the maintenance process.

[0045] When multiple experts intervene at the same time, the application detects conflicts between the three-dimensional arrows labeled by each expert through a consensus algorithm, that is, the Euclidean distance between the three-dimensional arrows labeled by each expert is calculated, and if the Euclidean distance is less than a preset distance threshold, it is determined that there is a conflict. The above scheme can timely find the differences and contradictions between the guidance labeled by different experts, and try to avoid the maintenance personnel from feeling confused due to receiving multiple inconsistent guidance, thereby improving the consistency and accuracy of the guidance information.

[0046] In a second aspect, the application provides a fault handling auxiliary system based on AR smart glasses, which adopts the following technical scheme:

[0047] A fault handling auxiliary system based on AR smart glasses, comprising a memory and a processor,

[0048] The memory stores a computer readable storage medium;

[0049] The processor processes the computer program stored on the computer readable storage medium to implement the method of the first aspect.

[0050] In summary, the application has the following at least one beneficial technical effect:

[0051] 1.The application adopts the unique ID of the AR smart glasses scanning device and analyzes it to obtain an analysis result, retrieves the target device's maintenance workflow in the pre-constructed risk library according to the analysis result, then splits the maintenance workflow into maintenance steps, marks the steps with risks in the maintenance steps as risk steps, adds a blocking process before the risk steps, and makes the risk steps unable to be activated before the risk steps are confirmed, so as to avoid operation skipping or illegal execution as much as possible, force the maintenance end to execute according to the preset maintenance workflow, reduce human randomness, and improve the safety of the maintenance process.

[0052] 2.The application trains a machine learning model using historical data and various safety thresholds of historical data, processes real-time data using the trained machine learning model, obtains safety thresholds of various real-time data, and predicts safety thresholds based on the trained machine learning model, which are more accurate than fixed thresholds, thereby improving the accuracy of various safety thresholds and determining whether real-time data is safe. Subsequently, when the real-time data exceeds the safety threshold, a blocking process is triggered. Once the process is blocked, the maintenance personnel must pause the operation and troubleshoot the problem, so that the risk is actively eliminated rather than passively bypassed, further improving the safety of the maintenance process. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 is a flowchart of embodiment 1 of the application;

[0054] Figure 2 is a flowchart of embodiment 2 of the application;

[0055] Figure 3 is a flowchart of embodiment 3 of the application;

[0056] Figure 4 is a flowchart of embodiment 4 of the application. DETAILED DESCRIPTION

[0057] The following will be combined Figures 1 to 4 to further illustrate the application.

[0058] The following explains the related terms in the technical solution of the application:

[0059] AR smart glasses are a kind of wearable devices with functions such as image recognition, data processing and display interaction. It can capture the image of the target device through the built-in camera and extract the unique identification information on the target device, i.e. the unique ID, using advanced image recognition algorithms. This unique ID can be in the form of a two-dimensional code, a bar code, an RFID tag or a specific device serial number, etc., which is used to accurately identify and distinguish different target devices. The application adopts the form of two-dimensional code, and in other embodiments, other forms of unique ID can also be selected according to the needs.

[0060] The risk library is a pre-established database that stores relevant information of various target devices, including the maintenance workflow of the target device.

[0061] The maintenance workflow is a series of ordered maintenance steps and operation specifications formulated for the target device, which describes the entire process from the beginning to the end of the maintenance, covering the specific operation content, required tools, time requirements, etc. of each maintenance step.

[0062] The blocking process is a mechanism for controlling the execution of the maintenance workflow. When the maintenance workflow executes to the risk step, the blocking process is triggered to suspend the continuation of the maintenance workflow until the set conditions are met to unblock. The blocking process can include waiting for the confirmation feedback of the maintenance personnel, obtaining additional approval permission, completing related safety checks, etc.

[0063] Embodiment 1: The embodiment discloses a fault handling auxiliary method based on AR smart glasses, referring to Figure 1 , the method comprises: S11 adding blocking and S12 step analysis, using AR smart glasses to scan the unique ID of the target device, retrieving the maintenance workflow from the pre-constructed risk library according to the analysis result, analyzing the maintenance steps in the maintenance workflow and marking the risk steps, adding the blocking process; if it is a risk step, trigger the blocking, and unblock after the maintenance end confirms the feedback, continue to execute the risk step in the maintenance workflow, the execution process of each step in the embodiment is as follows:

[0064] S11 adds blocking, the maintenance personnel wears AR smart glasses, aims the camera of the AR smart glasses at the unique ID mark on the target device, then the AR smart glasses automatically start the scanning program, quickly identify and read the information in the unique ID, then analyze the information in the unique ID to obtain the analysis result of the unique ID.

[0065] The analysis result of the unique ID is used as a query condition to query the pre-constructed risk library, the target device related information corresponding to the unique ID is found by matching the unique ID with the device ID field in the risk library, and the maintenance workflow of the target device is obtained in the target device related information. Then, according to the pre-set risk assessment rules and standards, the risk of each maintenance step is assessed, and the maintenance step meeting the risk condition is marked as a risk step.

[0066] The risk assessment can consider multiple factors, such as the complexity of the step, the operation difficulty, the involved dangerous substances or energy, the historical failure record of the device, etc. If any of the above conditions is met or any of the pre-set risk assessment rules is violated, the maintenance step is considered as a risk step.

[0067] In other embodiments, the risk step in the maintenance workflow can also be determined by manually labeling each maintenance step as a risk step in advance.

[0068] The blocking process is inserted in the execution logic of the risk step, and a blocking function is set before the running node of the risk step. The maintenance workflow is paused before the risk step is executed by calling the blocking function until a signal to unblock is received.

[0069] In other embodiments, the blocking process before the risk step can also be implemented by modifying the execution state of the workflow, setting a flag, or setting a blocking condition.

[0070] In step S12, it is determined whether the current maintenance step is a risk step according to the label added in step S11. If so, the blocking process is triggered. After the blocking process is triggered, the maintenance terminal needs to wait for the confirmation feedback of the risk step. The maintenance terminal sends the confirmation information through the AR smart glasses or a mobile terminal. The confirmation information includes the understanding of the risk step, the safety measures taken, the operation preparation, and the like.

[0071] In this embodiment, the maintenance terminal provides the confirmation feedback in the form of answering the questions set in advance for some links in the risk step. The answer can be manually selected or input by voice.

[0072] In other embodiments, the confirmation feedback can also be provided in other forms, such as watching a complete video related to the risk step.

[0073] The received confirmation feedback is verified. If the confirmation feedback is valid (the answers to the questions are correct), the blocking is released by calling the unblocking function, and the risk step in the maintenance workflow is continued.

[0074] This embodiment uses the AR smart glasses to quickly and accurately obtain the device maintenance information. The risk step is accurately identified based on the rule evaluation. The blocking process is inserted to pause the workflow. The mechanism of requiring the maintenance terminal to send the confirmation information in multiple forms and verification makes the maintenance personnel fully understand the risk and make preparations before activating the risk step. This embodiment has flexibility in adding the blocking process and the confirmation feedback method, can adapt to different maintenance scene requirements, and effectively improves the maintenance safety and efficiency.

[0075] Embodiment 2: Refer to Figure 2 The difference between this embodiment and embodiment 1 is that before the risk step in the maintenance workflow is continued to be executed, the method further includes:

[0076] S21 data collection, obtain training data, the training data including normal working condition historical data and various historical data safety threshold, the historical data including historical device parameters, historical operation data and historical environment data.

[0077] Device parameters are used to reflect the category and health status of the target device, and the device parameters include device model, temperature, pressure, vibration frequency, current voltage, etc.

[0078] Operation data is used to reflect the operation behavior mode, and the operation data includes load rate, start-stop times, running time, fault code record, etc.

[0079] The environment data includes temperature and humidity, dust concentration, gas composition, light intensity, etc., and the environment data refers to the environment data collected at the same time as the device parameters and operation data.

[0080] The safety threshold label refers to the safety range corresponding to the historical data at different times in the whole life cycle of the target device.

[0081] S22 data processing, the training data is divided into multiple historical segments according to a preset time window (such as 1 hour), and the historical data in the historical segment is recorded as the first data; the historical data of the target device in the preset time window before the current time is obtained, recorded as the second data;

[0082] The KL divergence of each first data and second data in the preset time window is calculated respectively, that is, a certain preset time window is recorded as a target window, and the historical segment in the target window will contain historical device parameters, historical operation data and historical environment data. At this time, the KL divergence of the historical device parameters in the target window and the device parameter part of the second data needs to be calculated, the KL divergence of the historical operation data in the target window and the operation data part of the second data needs to be calculated, and the KL divergence of the historical environment data in the target window and the environment parameter part of the second data needs to be calculated. The calculation formula of KL divergence is mature, which will not be repeated here.

[0083] All the KL divergences in the same preset time window are arranged in ascending order to obtain a KL divergence sequence, and the KL divergence is divided into m quantile intervals based on the KL divergence sequence, and each quantile interval represents different data distribution levels.

[0084] A weight is assigned to each quantile interval, and the sum of the weights of all quantile intervals is equal to 1, the quantile number of each quantile interval is calculated, and the quantile number of the kth quantile interval is calculated by cumulative weight. The calculation model is as follows:

[0085] ;

[0086] Wherein, is the quantile number of the kth quantile interval; i is the serial number of the quantile interval; is the weight of the ith quantile interval.

[0087] The calculation model of the drift index D is as follows:

[0088]

[0089] wherein m is the number of quantile intervals; is the weight of the quantile number of the ith quantile interval; is the quantile number of the ith quantile interval; is the average of the quantile number.

[0090] S23 drift judgment, when the drift index is greater than the preset drift threshold, an alarm signal is sent; otherwise, S24 model training is performed.

[0091] S24 model training, the pre-constructed machine learning model is trained using training data, and during the training process, the historical data is the training data, and the safety threshold corresponding to each kind of historical data is the training label, and the trained machine learning model is obtained.

[0092] The machine learning model can select a multi-output fully connected network model in a neural network or a quantile regression neural network model (QRNN), adopt a multi-task learning mode, and output safety thresholds of different kinds of historical data. Taking the QRNN model as an example, the training process is as follows:

[0093] The input data includes device parameters (such as temperature, vibration frequency), running data (load rate, running time), environmental data (temperature and humidity), and other features that are strongly related to the safety threshold. For continuous features (such as temperature), Z-score standardization processing (mean value is 0, variance is 1) is adopted, and for categorical features (such as device type), One-Hot encoding processing is adopted.

[0094] The quantile numbers of each quantile interval in the historical data are respectively assigned to low / middle / high risk thresholds, and at this time, the training label is the actual value of each quantile number in the historical data.

[0095] The QRNN model includes an input layer, a hidden layer, and an output layer, and the specific structure is as follows:

[0096] The number of neurons in the input layer is equal to the feature dimension;

[0097] The hidden layer includes a fully connected layer and an activation function, the fully connected layer is set to 2-4 layers, each layer has 64-256 neurons (adjusted according to the complexity of the data), and the activation function adopts a ReLU function;

[0098] ​The output layer selects a multi-head output, and three output heads are set in this embodiment, and each output head has one neuron (to predict low / middle / high threshold values respectively).

[0099] The QRNN model selects a quantile regression loss function, and the calculation model is as follows:

[0100] ;

[0101] wherein, is the jth real value; is the jth predicted value; b is a target quantile, for example, b = 0.1 corresponds to a low threshold value, b = 0.4 corresponds to a middle threshold value, and b = 0.9 corresponds to a high threshold value; and N is the total number of real values or predicted values.

[0102] The QRNN model selects an Adam optimizer.

[0103] S25 predicts a safety threshold value, collects real-time data of a target device, the real-time data including real-time device parameters, real-time operation data and real-time environmental data, inputs the real-time data into a trained machine learning model, and outputs safety threshold values of various real-time data.

[0104] S26 analyzes the real-time data, if the real-time data is out of the range of the safety threshold value of the real-time data predicted in the safety threshold value predicted in S25, a blocking process is triggered, and an alarm signal is sent to a maintenance personnel, prompting the maintenance personnel that the real-time data is at risk at this time, otherwise, no processing is performed.

[0105] This embodiment adopts KL divergence to detect data drift, which can reduce the risk of failure due to working condition changes, and further utilizes dynamic threshold to capture device performance degradation, thereby achieving dynamic prediction and risk control of the device safety threshold value.

[0106] Embodiment 3: with reference to Figure 3 The difference between this embodiment and embodiment 1 is that, after the blocking is released in the S12 step of analysis, the method further includes:

[0107] S31 checks a real-time operation picture, acquires a real-time operation picture of a risk step of a maintenance end in real time through AR smart glasses, and calls standard operation pictures and non-standard operation pictures of the risk step in a risk library.

[0108] The standard operation picture refers to a risk step performed by a senior maintenance personnel according to an industry specification or an enterprise standard operation procedure, and is collected by AR smart glasses or a same-parameter camera in multiple angles, for example, the risk step of “power switch closing” needs to collect key frames such as hand position, closing force visualization and instrument values, and at least 500 pictures are collected for each risk step to cover different personnel, different device models and different light scenes.

[0109] Non-standard operation pictures refer to simulating common error operations, such as not powering off before disconnection, wrench size inconsistency, lack of protective equipment, etc. Similarly, key frames of error operations need to be collected from multiple angles, and the number is comparable to that of standard pictures. In addition, error types need to be clearly labeled, such as power-off omission and tool error.

[0110] The standard operation pictures and the non-standard operation pictures are used to train the CNN model to obtain a trained CNN model. The trained CNN model can identify whether a real-time operation picture is a standard operation.

[0111] The real-time operation picture is input into the trained CNN model to obtain a classification result. When the classification result does not meet the expectation, an alert bounding box is pushed to the maintenance end through the AR smart glasses, and the maintenance end is instructed to re-maintain the risk step. When the maintenance end re-maintains the risk step, the method further includes S32 calculating an initial similarity score.

[0112] S32 calculates an initial similarity score. A computer vision algorithm is used to capture 21 hand key points of the maintenance personnel. Based on the hand key points and using SLAM technology, the hand key point coordinates are mapped to a three-dimensional space to generate a continuous hand motion trajectory. The hand motion trajectory is time-aligned with the pre-stored standard operation trajectory in the risk library using DTW, and the dynamic time warping distance is calculated as the initial similarity score.

[0113] S33 initial similarity score determination. Determine whether the initial similarity score is lower than the similarity score threshold. If yes, execute S34 to generate a path. If no, do not process.

[0114] S34 generates a path. The risk step is decomposed to obtain multiple key links in the risk step. Key points of each key link are extracted, including human body key points, target device key points, and target tool key points.

[0115] A path planning algorithm is used to generate a maintenance posture path based on all key points. The process is as follows:

[0116] The maintenance space is modeled as a grid map. Each grid represents a 3D space of 1cm x 1cm x 1cm. Dangerous grids are labeled, such as live areas and areas with device obstacles.

[0117] The starting key point of the key link is taken as the starting point, such as the handle key point when the wrench does not contact the bolt. The target key point is taken as the end point, such as the handle key point when the wrench completely covers the bolt.

[0118] Define a heuristic function and a cost function to search for the shortest global path that meets the constraints, i.e., a sequence of grids.

[0119] The center points of the grids in the grid sequence are taken as control vertices, and a 3rd order B-spline curve is connected to generate a continuous and smooth path (i.e. the maintenance posture path), and it is checked whether the smoothed path meets the ergonomics constraints (such as joint angles), and if not, the control vertices are adjusted for re-interpolation.

[0120] S35 calculates a final similarity score, performs DTW time alignment processing on the hand trajectory of the maintenance personnel when re-operating and the maintenance posture path generated in S34 to obtain a final similarity score, and if the final similarity score is lower than the similarity score threshold, a red warning trajectory is superimposed in the AR interface; if the final similarity score is not lower than the similarity score threshold, a signal of successful maintenance is output.

[0121] When the initial similarity score of the maintenance personnel and the standard operation trajectory is low, the embodiment further analyzes and processes the risk step, generates a maintenance posture path for the step, calculates a final similarity score, and if the final similarity score is still lower than the preset similarity threshold, a red warning trajectory is superimposed in the AR interface, otherwise it is considered that the maintenance is successful. The embodiment realizes the full-process closed-loop control from picture compliance verification to trajectory precision correction of the maintenance operation.

[0122] Embodiment 4: Refer to Figure 4 The difference between the embodiment and embodiment 2 is that after triggering the blocking process in the S12 step analysis, the method further comprises:

[0123] S41 blocking analysis, the duration of the blocking process is counted, when the duration of the blocking process exceeds the preset time threshold, a assistance request package containing the three-dimensional point cloud data of the target device in the risk step, the real-time data of the target device, and the hash value of the historical operation record is automatically generated, and the assistance request package is sent to the remote expert terminal through the 5G network. After receiving the assistance request package, the remote expert terminal establishes a real-time audio and video and data interaction channel, i.e. an AR collaboration channel, with the on-site maintenance terminal through the AR device.

[0124] S42 label arrow, the expert locks the position of the target device corresponding to the risk step by calling the space anchor point through the SLAM technology, and labels a three-dimensional arrow in the AR interface of the maintenance terminal to guide the maintenance personnel to operate.

[0125] Based on the SLAM algorithm, the world coordinate system coordinates and rotation matrix of the position of the corresponding target device are recorded to generate a dynamic guidance sequence containing a timestamp. When the maintenance personnel moves, the SLAM technology is used to match the position of the risk step with the coordinate value of the space anchor point in real time, and the pose of the three-dimensional arrow is dynamically updated through the rigid transformation algorithm.

[0126] S43 conflict detection, when multiple experts intervene at the same time, the three-dimensional arrows marked by each expert are detected for conflicts through a consensus algorithm, including:

[0127] Each three-dimensional arrow marked by an expert is converted into a unified world coordinate system through a SLAM algorithm, and the coordinate format is .

[0128] Based on the three-dimensional coordinates The Euclidean distance between any two three-dimensional arrow position points is calculated, and if the Euclidean distance is less than a preset distance threshold, it is determined that the two three-dimensional arrows corresponding to the Euclidean distance less than the preset distance threshold have a position conflict, an alarm signal is sent to all remote expert terminals, the experts are asked to vote within a certain time to select which three-dimensional arrow to keep, and the majority voting result is executed.

[0129] The embodiment utilizes SLAM technology to accurately position the target device and dynamically guide the maintenance operation, thereby improving the maintenance efficiency and collaboration accuracy.

[0130] Embodiment 5: The embodiment discloses a fault handling auxiliary system based on AR smart glasses, the system comprising: a memory and a processor,

[0131] The memory stores a computer readable storage medium;

[0132] The processor processes the computer program stored on the computer readable storage medium to implement a fault handling auxiliary method based on AR smart glasses.

[0133] The above are preferred embodiments of the present application, and are not intended to limit the protection scope of the present application, therefore: any equivalent changes made according to the structure, shape, principle of the present application shall be covered within the protection scope of the present application.

Claims

1. An AR intelligence glasses-based failure handling assistance method, characterized by, The method comprises the following steps: Scanning the unique ID of the target device by using the AR smart glasses, calling the maintenance workflow of the target device in the pre-constructed risk library according to the analysis result of the unique ID, splitting the maintenance workflow into maintenance steps, marking the maintenance steps with risks as risk steps, and adding a blocking process before the risk steps; If the current step is a risk step, the blocking process is triggered, and after receiving the confirmation feedback of the risk step from the maintenance end, the blocking is released, and the risk step in the maintenance workflow is continued to be executed; Before continuing to execute the risk step in the maintenance workflow, the method further comprises: Obtaining training data, wherein the training data comprises historical data under normal working conditions and safety thresholds of various historical data, the historical data comprises historical device parameters, historical operation data and historical environment data, the historical data in the training data is segmented by using a preset time window to obtain segmented historical data, which is denoted as first data; obtaining the historical data of the target device within a preset time window before the current time, which is denoted as second data; Calculating the KL divergence of each first data and second data within the preset time window, calculating the drift index based on all KL divergences within the preset time window by using a weighted quantile sum algorithm, and issuing an alarm signal when the drift index is greater than a preset drift threshold; otherwise, the pre-constructed machine learning model is trained by using the training data to obtain a trained machine learning model; Collecting real-time data of the target device, wherein the real-time data comprises real-time device parameters, real-time operation data and real-time environment data, inputting the real-time data into the trained machine learning model, and outputting safety thresholds of various real-time data; If the real-time data exceeds the range of the safety thresholds of the real-time data, the blocking process is triggered. 2.The AR smart glasses-based failure handling assistance method of claim 1, wherein, After the blocking is released, the method further comprises: Real-time acquisition of the real-time operation picture of the risk step by the AR smart glasses, calling the standard operation picture and the non-standard operation picture of the risk step in the risk library, training the CNN model by using the standard operation picture and the non-standard picture to obtain a trained CNN model, inputting the real-time operation picture into the trained CNN model to obtain a classification result, and when the classification result does not meet the expectation, pushing a warning bounding box to the maintenance end through the AR smart glasses and instructing the maintenance end to re-maintain the risk step. 3.The AR smart glasses-based failure handling assistance method of claim 2, wherein, When the maintenance end re-maintains the risk step, the method further comprises: Capturing the hand key points of the maintenance personnel by using a computer vision algorithm, reconstructing the hand motion trajectory of the maintenance personnel based on the hand key points and using SLAM technology, performing DTW time alignment on the hand motion trajectory and the pre-stored standard operation trajectory in the risk library to obtain an initial similarity score, and when the initial similarity score is lower than a similarity score threshold, superimposing a red warning trajectory in the AR interface. 4.The AR smart glasses-based failure handling assistance method of claim 3, wherein, Before superimposing the red warning trajectory in the AR interface, the method further comprises: The risk step is decomposed to obtain a plurality of key links in the risk step, key points of each key link are extracted, a path planning algorithm is used, a maintenance posture path is generated based on all the key points, a final similarity score of a hand motion trajectory and the maintenance posture path is calculated, if the final similarity score is lower than a similarity score threshold, a red warning trajectory is superimposed in an AR interface, if the final similarity score is not lower than the similarity score threshold, a signal of successful maintenance is output. 5.The AR intelligent glasses-based failure handling assistance method according to claim 1, characterized in that, After triggering the blocking process, the method further comprises: statistically blocking process duration, when the blocking process duration exceeds a preset time threshold, generating an assistance request package, sending the assistance request package to a remote expert terminal through a 5G network, establishing an AR collaboration channel, the assistance request package comprising: three-dimensional point cloud data of the target device in the risk step, real-time data of the target device, hash value of the historical operation record; The expert calls the spatial anchor point to lock the position of the risk step, and labels a three-dimensional arrow in the AR interface of the maintenance end at the position of the risk step to guide the maintenance personnel to operate. 6.The AR intelligent glasses-based failure handling assistance method according to claim 5, characterized in that, The method further comprises: The spatial coordinates and rotation matrix of the three-dimensional arrow are recorded through the spatial anchor point, and a dynamic guidance sequence containing a time stamp is generated; When the maintenance personnel moves, the SLAM technology is used to match the position of the risk step and the coordinate value of the spatial anchor point in real time, and the pose of the three-dimensional arrow is dynamically updated through the rigid transformation algorithm. 7.The AR intelligence glasses-based failure handling assistance method according to claim 5, characterized in that, When multiple experts intervene at the same time, the three-dimensional arrows labeled by each expert are detected for conflicts through a consensus algorithm, including: The Euclidean distance between the three-dimensional arrows labeled by each expert is calculated, if the Euclidean distance is less than a preset distance threshold, an alarm signal is output.

8. An AR smart glasses-based failure handling assistance system, characterized by, It comprises: a memory and a processor, the memory stores a computer readable storage medium; the processor processes the computer program stored on the computer readable storage medium to realize the method of any one of claims 1-7.

Images