Fault processing auxiliary method and system based on AR intelligent glasses

Abstract

The invention provides a fault processing auxiliary method and system based on AR intelligent glasses, and relates to the technical field of auxiliary fault processing, and the method comprises the steps: employing the AR intelligent glasses to scan a unique ID of a target device, calling a maintenance workflow from a pre-constructed risk library according to an analysis result, and carrying out the maintenance workflow, analyzing maintenance steps in the maintenance workflow, marking risk steps, and adding a blocking process; if the current step is the risk step, blocking is triggered, the blocking is relieved after the overhaul end confirms feedback, and the risk step in the overhaul workflow continues to be executed. According to the method, the blocking process is added before the risk step, and the blocking process is removed after the confirmation feedback of the maintenance end is received, so that the safety of the maintenance process can be effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of assisting in handling faults, in particular to a fault handling assistance method and system based on AR intelligent glasses. BACKGROUND

[0002] In a power system, a distribution network is a power network directly facing users, and its stable operation is crucial to ensuring the normal order of social production and life. The distribution network contains a large number of devices, such as transformers, switch cabinets, cables, etc. These devices will inevitably have various faults during long-term operation, and need to be repaired in a timely manner to restore the normal operation of the devices, reduce power outage time and scope, and improve power supply reliability. Currently, the repair work of distribution network devices mainly relies on manual operation, and repair personnel usually rely on experience and familiarity with the device to gradually carry out work according to the established repair procedures.

[0003] A Chinese invention patent with publication number CN115510253A provides a substation device AR auxiliary maintenance method and terminal based on a three-dimensional knowledge graph. After discovering a fault, the patent determines a target maintenance plan for the user to view. However, for some high-risk maintenance steps, the patent only emphasizes their importance and precautions through text or pictures, and cannot intervene and pause forcibly when the maintenance personnel actually execute the step. The maintenance personnel may skip necessary inspection and confirmation steps and directly proceed to the next operation due to negligence, eagerness to complete the task, etc., thereby burying potential safety hazards. SUMMARY

[0004] In order to improve the safety in the fault repair process, the present application provides a fault handling assistance method and system based on AR intelligent glasses.

[0005] In a first aspect, the present application provides a fault handling assistance method based on AR intelligent glasses, which adopts the following technical solution: A fault handling assistance method based on AR intelligent glasses includes the following steps: Scanning the unique ID of the target device using AR intelligent glasses, retrieving the repair workflow of the target device from the pre-constructed risk library according to the analysis result of the unique ID, splitting the repair workflow into repair steps, marking the repair steps with risks as risk steps, and adding a blocking process before the risk steps; If the current step is a risk step, the blocking process is triggered, and after receiving the confirmation feedback of the risk step from the repair end, the blocking is released, and the risk step in the repair workflow is continued.

[0006] The application adopts the unique ID of the AR intelligent glasses scanning device, analyzes it, obtains an analysis result, retrieves the target device's maintenance workflow in the pre-constructed risk library according to the analysis result, then splits the maintenance workflow into maintenance steps, marks the steps with risks in the maintenance steps as risk steps, adds a blocking process before the risk steps, so that the risk steps cannot be activated before the risk steps are confirmed, and the operation is prevented from being skipped or executed in violation of regulations, the maintenance end is forced to execute the preset maintenance workflow, the randomness of human is reduced, and the safety of the maintenance process is improved.

[0007] Optionally, before continuing to execute the risk steps in the maintenance workflow, the method further comprises: obtaining training data, the training data including historical data under normal working conditions and safety thresholds of various historical data, the historical data including historical device parameters, historical operation data and historical environment data, training the pre-constructed machine learning model by using the training data to obtain a trained machine learning model; collecting real-time data of the target device, the real-time data including real-time device parameters, real-time operation data and real-time environment data, inputting the real-time data into the trained machine learning model, and outputting safety thresholds of various real-time data; if the real-time data exceeds the range of the safety thresholds of the real-time data, triggering the blocking process.

[0008] The application trains the machine learning model by using the historical data and the safety thresholds of various historical data, processes the real-time data by using the trained machine learning model, obtains the safety thresholds of various real-time data, and predicts the safety thresholds based on the trained machine learning model. The safety thresholds are more suitable for actual working conditions than fixed thresholds, improve the accuracy of various safety thresholds, and then determine whether the real-time data is safe. Subsequently, when the real-time data exceeds the safety threshold, the blocking process is triggered. Once the process is blocked, the maintenance personnel must pause the operation and troubleshoot the problem, so that the risk is actively eliminated rather than passively bypassed, and the safety of the maintenance process is further improved.

[0009] Optionally, the method further comprises: segmenting the historical data in the training data by using a preset time window to obtain segmented historical data, denoted as first data; obtaining historical data of the target device in a preset time window before the current time, denoted as second data; calculating the KL divergence of each first data and second data in the preset time window, calculating a drift index based on all the KL divergences in the preset time window by using a weighted quantile summation algorithm, and issuing an alarm signal when the drift index is greater than a preset drift threshold.

[0010] The application quantifies the degree of change of data features by comparing the distribution difference of the training data and the second data, and then aggregates the plurality of KL divergence values to reflect the degree of change of the overall data distribution, and triggers an alarm when the drift index exceeds the preset threshold, prompting the existence of potential risks.

[0011] Optionally, after the blockage is removed, the method further comprises: The real-time operation picture of the risk step by the maintenance end is acquired in real time through the AR smart glasses, the standard operation picture and the non-standard operation picture about the risk step in the risk library are called, the standard operation picture and the non-standard picture are taken as training samples, and the CNN model is trained by using the training samples to obtain the trained CNN model, the real-time operation picture is input into the trained CNN model, and a classification result is obtained, when the classification result does not meet the expectation, an alarm bounding box is pushed to the maintenance end through the AR smart glasses, and the maintenance end is instructed to re-maintain the risk step.

[0012] The operation picture of the risk step by the maintenance end is acquired in real time through the AR smart glasses, and the maintenance action and details are captured in time to avoid the abnormal situation caused by information delay, and the maintenance process is monitored in real time. When the CNN model classification result does not meet the expectation, the application immediately pushes an alarm bounding box to the maintenance end through the AR smart glasses, so that the maintenance personnel can quickly understand the problem, without spending extra time to interpret complex reports or instructions, thereby improving the maintenance efficiency. The alarm bounding box pushed by the AR smart glasses can clearly indicate the problem position and nature, help the maintenance personnel to quickly take correct measures, and try to avoid the occurrence of accidents to ensure the safe performance of the maintenance work.

[0013] The CNN model is trained by calling the standard operation picture and the non-standard operation picture about the risk step in the risk library, which helps to improve the distinguishing ability of the CNN model for the standard operation and the non-standard operation. The real-time operation picture is compared and analyzed with the standard operation picture, and the CNN model is used for classification and judgment, so that whether the maintenance process meets the standard requirement can be strictly controlled. When the classification result does not meet the expectation, the maintenance end is required to re-maintain in time, so that each risk step can be handled correctly, the overall maintenance quality is improved, and the risk of safety accidents caused by improper operation is reduced.

[0014] Optionally, when the maintenance end re-maintains the risk step, the method further comprises: The computer vision algorithm is used to capture the hand key points of the maintenance personnel, the hand key points are used to reconstruct the hand movement trajectory of the maintenance personnel by using the SLAM technology, the hand movement trajectory is time-aligned with the pre-stored standard operation trajectory in the risk library by using the DTW technology, and an initial similarity score is obtained.

[0015] The computer vision algorithm is used to capture the hand key points of the maintenance personnel, the hand key points are used to reconstruct the hand movement trajectory of the maintenance personnel by using the SLAM technology, the hand movement trajectory is time-aligned with the pre-stored standard operation trajectory in the risk library by using the DTW technology, and an initial similarity score is obtained.

[0016] Subsequently, the reconstructed hand movement trajectory is time-aligned with the pre-stored standard operation trajectory in the risk library by using the DTW (dynamic time warping) technology, an initial similarity score is obtained, when the initial similarity score is lower than the pre-set similarity score threshold, it is determined that the operation of the maintenance personnel does not conform to the standard, a red warning trajectory is superimposed in the AR interface, the AR technology can combine virtual information with a real scene, so that the maintenance personnel can intuitively see the difference between the hand movement trajectory and the standard trajectory during operation, the red warning trajectory can quickly attract the attention of the maintenance personnel, timely remind them that the operation is deviated and needs to be adjusted immediately, so that the maintenance personnel can find problems and correct them in time during operation, improve the maintenance efficiency, and reduce the waste of time and resources.

[0017] By strictly monitoring the operation trajectory of the maintenance personnel and timely warning the operation that does not conform to the standard, the application can effectively prevent safety accidents caused by improper operation, and long-term application of the method is helpful for the maintenance personnel to gradually develop a standard safety operation consciousness.

[0018] Optionally, before superimposing the red warning trajectory in the AR interface, the method further comprises: The risk step is decomposed to obtain a plurality of key links in the risk step, key points of each key link are extracted, a path planning algorithm is used, a maintenance posture path is generated based on all the key points, a final similarity score of the hand movement trajectory and the maintenance posture path is calculated, if the final similarity score is lower than the similarity score threshold, a red warning trajectory is superimposed in the AR interface; if the final similarity score is not lower than the similarity score threshold, a signal that the maintenance is successful is output.

[0019] The application obtains a plurality of key links by decomposing the risk step, each key link represents a step that needs to be focused on and accurately executed in the maintenance process, then the key points of each key link are extracted, the key positions and action points that the maintenance personnel need to focus on in each key link are clarified, which helps them to more accurately complete the task of each link and improves the standardization and accuracy of operation.

[0020] Since the installation environments of the target devices are different, the application generates a maintenance posture path based on all key points by using a path planning algorithm, which comprehensively considers the logical relationship and operation sequence between the key links and the reasonable moving trajectory of the hand in space, then the application calculates the final similarity score of the hand movement trajectory and the maintenance posture path, quantifies the difference between the actual operation of the maintenance personnel and the standard path, and then analyzes whether the actual operation of the maintenance personnel is risky in combination with the actual situation. If the final similarity score is lower than the similarity score threshold, a red warning trajectory is superimposed in the AR interface.

[0021] Optionally, after triggering the blocking process, the method further comprises: The duration of the blocking process is counted, and when the duration of the blocking process exceeds a preset time threshold, a request package for assistance is generated, the request package for assistance is sent to a remote expert terminal through a 5G network, an AR collaboration channel is established, and the request package for assistance includes: three-dimensional point cloud data of the target device in the risk step, real-time data of the target device, and a hash value of the historical operation record; The expert calls the spatial anchor point to lock the position of the risk step, and a three-dimensional arrow is marked at the position of the risk step in the AR interface of the maintenance end to guide the maintenance personnel to operate.

[0022] The application can automatically generate an assistance request package when the duration of the blocking process exceeds a preset time threshold by counting the duration of the blocking process, which helps to timely detect long-term stagnation problems in the maintenance process and avoid delays in the overall maintenance schedule due to long blocking time. The assistance request package contains three-dimensional point cloud data of the target device in the risk step, providing spatial information of the risk step for remote experts. Experts can more intuitively understand the specific location and surrounding environment of the risk step based on the three-dimensional point cloud data, thereby more accurately determining the problem and developing more effective solutions. Real-time data of the target device in the assistance request package enables experts to monitor the running state and parameter changes of the device in real time, combined with the situation of the risk step, experts can analyze the correlation between device failure and current operation, and provide more targeted guidance to maintenance personnel to avoid misjudgment and misoperation due to incomplete information. The hash value of the historical operation record in the assistance request package provides data integrity and traceability for the maintenance process. Experts can verify the authenticity and integrity of the historical operation record by verifying the hash value, understand the operation of the maintenance personnel in the previous step, and better analyze the causes of the problem to improve the accuracy of diagnosis.

[0023] Subsequently, the application sends the assistance request package to the remote expert terminal through the 5G network and establishes an AR collaboration channel. With the help of the low-latency AR collaboration channel, experts can participate in the collaboration of the maintenance site in real time, shortening the response and intervention time of experts and improving the efficiency of problem solving.

[0024] Subsequently, the expert calls the spatial anchor to lock the position of the risk step, which can provide a precise reference point for the maintenance personnel in the AR environment. Then, the expert labels a three-dimensional arrow at the position of the risk step in the AR interface of the maintenance terminal, so that the maintenance personnel can operate according to the direction and position of the three-dimensional arrow, reducing the blindness and uncertainty of the operation and improving the accuracy and standardization of the operation.

[0025] Optionally, the method further comprises: Recording the spatial coordinates and rotation matrix of the three-dimensional arrow through the spatial anchor to generate a dynamic guidance sequence containing a timestamp; When the maintenance personnel moves, the SLAM technology is used to match the position of the risk step with the coordinate value of the spatial anchor in real time, and the pose of the three-dimensional arrow is dynamically updated through the rigid transformation algorithm.

[0026] Optionally, when multiple experts intervene at the same time, a consensus algorithm is used to detect conflicts between the three-dimensional arrows labeled by each expert, including: Calculating the Euclidean distance between the three-dimensional arrows labeled by each expert, and if the Euclidean distance is less than a preset distance threshold, an alarm signal is output.

[0027] The application records the spatial coordinates and rotation matrix of the three-dimensional arrow through the spatial anchor point, and generates a dynamic guidance sequence containing a time stamp. The spatial coordinates and rotation matrix can accurately describe the position and posture of the three-dimensional arrow in the three-dimensional space. The dynamic guidance sequence containing the time stamp makes the guidance sequence have order in the time dimension, so that the maintenance personnel can receive the guidance in time sequence, reducing the degree of information confusion and improving the coherence and accuracy of the operation steps.

[0028] When the maintenance personnel move, the SLAM (simultaneous localization and mapping) technology is used to match the position of the risk step with the coordinate value of the spatial anchor point in real time, and the pose of the three-dimensional arrow is dynamically updated through the rigid transformation algorithm. The SLAM technology can perceive the position changes of the maintenance personnel and the risk step in real time, and the rigid transformation algorithm can quickly adjust the position and direction of the three-dimensional arrow according to these changes, so that it always accurately points to the risk step, providing continuous and effective guidance for the maintenance personnel, adapting to the dynamic changes of personnel and equipment in the maintenance process.

[0029] When multiple experts intervene at the same time, the application performs conflict detection on the three-dimensional arrows labeled by each expert through a consensus algorithm, that is, calculates the Euclidean distance between the three-dimensional arrows labeled by each expert. If the Euclidean distance is less than a preset distance threshold, it is determined that there is a conflict. The above scheme can timely find the differences and contradictions between the guidance labeled by different experts, and try to avoid the maintenance personnel from feeling confused due to receiving multiple inconsistent guidance, thereby improving the consistency and accuracy of the guidance information.

[0030] In a second aspect, the application provides a fault handling auxiliary system based on AR smart glasses, which adopts the following technical scheme: A fault handling auxiliary system based on AR smart glasses, comprising a memory and a processor, The memory stores a computer readable storage medium; The processor processes the computer program stored on the computer readable storage medium to implement the method of the first aspect.

[0031] In summary, the application has the following at least one beneficial technical effect: 1. The application scans the unique ID of the device through the AR smart glasses, analyzes it, obtains an analysis result, retrieves the maintenance workflow of the target device in the pre-constructed risk library according to the analysis result, then splits the maintenance workflow into maintenance steps, marks the steps with risks in the maintenance steps as risk steps, and adds a blocking process before the risk steps. The blocking process makes the risk step unable to be activated before the risk step is confirmed, tries to avoid operation skipping or illegal execution, forces the maintenance end to perform according to the preset maintenance workflow, reduces human randomness, and improves the safety of the maintenance process.

[0032] 2. The application uses historical data and various historical data safety thresholds to train the machine learning model, and uses the trained machine learning model to process real-time data to obtain various real-time data safety thresholds. The safety threshold predicted by the trained machine learning model is more in line with the actual working condition than the fixed threshold, which improves the accuracy of various safety thresholds, and then determines whether the real-time data is safe, and then triggers the blocking process when the real-time data exceeds the safety threshold. Once the process is blocked, the maintenance personnel must pause the operation and troubleshoot the problem, so that the risk is actively eliminated rather than passively bypassed, further improving the safety of the maintenance process. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 is a flowchart of embodiment 1 of the application; Figure 2 is a flowchart of embodiment 2 of the application; Figure 3 is a flowchart of embodiment 3 of the application; Figure 4 is a flowchart of embodiment 4 of the application. DETAILED DESCRIPTION

[0034] The following will be combined Figures 1 to 4 Further details of the application will be described below.

[0035] The following explains the related terms in the technical solution of the application: AR smart glasses are a kind of wearable devices with image recognition, data processing and display interaction functions. It can capture the image of the target device through the built-in camera, and use advanced image recognition algorithm to extract the unique identification information on the target device, i.e. unique ID. This unique ID can be in the form of a two-dimensional code, a bar code, an RFID tag or a specific device serial number, etc., which is used to accurately identify and distinguish different target devices. The application uses the form of two-dimensional code, and in other embodiments, other forms of unique ID can also be selected according to the needs.

[0036] The risk library is a pre-established database that stores various information about the target device, including the target device's maintenance workflow.

[0037] The maintenance workflow is a series of ordered maintenance steps and operation specifications for the target device, which describes the entire process from the beginning to the end of the maintenance, covering the specific operation content, required tools, time requirements, etc. of each maintenance step.

[0038] The blocking process is a mechanism for controlling the execution of the maintenance workflow. When the maintenance workflow executes to a risk step, the blocking process is triggered to suspend the continuation of the maintenance workflow until the set conditions are met to unblock. The blocking process can include waiting for the confirmation feedback of the maintenance personnel, obtaining additional approval permission, completing related safety checks, etc.

[0039] Embodiment 1: The embodiment discloses a fault handling auxiliary method based on AR smart glasses, referring to Figure 1 , the method comprises: S11 adding a block and S12 step analysis, scanning the unique ID of the target device by using the AR smart glasses, calling the maintenance workflow from the pre-constructed risk library according to the analysis result, analyzing the maintenance steps in the maintenance workflow and marking the risk steps, and adding the blocking process; if the current is a risk step, the blocking is triggered, and the blocking is unblocked after the confirmation feedback of the maintenance end, and the risk step in the maintenance workflow is continued to be executed, and the execution process of each step of the embodiment is as follows: S11 adds a block, the maintenance personnel wears AR smart glasses, aims the camera of the AR smart glasses at the unique ID mark on the target device, and then the AR smart glasses automatically start the scanning program, quickly identify and read the information in the unique ID, and then analyze the information in the unique ID to obtain the analysis result of the unique ID.

[0040] The analysis result of the unique ID is used as a query condition to query in the pre-constructed risk library, the unique ID is matched with the device ID field in the risk library, the target device related information corresponding to the unique ID is found, and the maintenance workflow of the target device is obtained in the target device related information. Then, according to the pre-set risk assessment rules and standards, the risk of each maintenance step is evaluated, and the maintenance step meeting the risk condition is marked as a risk step.

[0041] The risk assessment can consider multiple factors, such as the complexity of the step, the operation difficulty, the involved dangerous substances or energy, the historical failure record of the device, etc. If any of the above conditions is met or any of the pre-set risk assessment rules is violated, the maintenance step is considered as a risk step.

[0042] In other embodiments, a method of manually pre-labeling each maintenance step as a risk step can also be used to determine the risk step in the maintenance workflow.

[0043] The blocking process is inserted in the execution logic of the risk step, and a blocking function is set before the running node of the risk step. The maintenance workflow is suspended before executing the risk step by calling the blocking function until the unblocking signal is received.

[0044] In other embodiments, the addition of the blocking process before the risk step can also be achieved by modifying the execution state of the workflow, setting a flag, or setting a blocking condition, etc.

[0045] S12 step analysis, according to the mark added in S11, judge whether the current maintenance step is a risk step, if yes, trigger the blocking process, after the blocking process is triggered, it needs to wait for the confirmation feedback of the risk step from the maintenance end, the maintenance end sends the confirmation information through the AR smart glasses or mobile terminal, the confirmation information includes the understanding of the risk step, the safety measures taken, the operation preparation situation, etc.

[0046] In this embodiment, the maintenance end performs confirmation feedback in the form of answering some questions preset in the risk step, the answer mode can be manual selection of answers or voice input of answers.

[0047] In other embodiments, confirmation feedback can also be performed in other forms, such as watching related videos of the complete risk step, etc.

[0048] Verify the received confirmation feedback, if the confirmation feedback is valid feedback (the answers to the questions are all correct, then it is considered as valid feedback), then remove the block by calling the unblock function, and continue to execute the risk step in the maintenance workflow.

[0049] This embodiment uses AR smart glasses to quickly and accurately obtain device maintenance information; accurately identifies risk steps through rule-based evaluation; inserts a blocking process to pause the workflow, and cooperates with the mechanism of requiring the maintenance end to send confirmation information in multiple forms and verification, so that the maintenance personnel fully understand the risk and make preparations before activating the risk step. This embodiment has flexibility in adding a blocking process and confirmation feedback mode, can adapt to different maintenance scene needs, and effectively improves the safety and efficiency of maintenance.

[0050] Embodiment 2: Refer to Figure 2 The difference between this embodiment and embodiment 1 is that before continuing to execute the risk step in the maintenance workflow in the S12 step analysis, the method further comprises: S21 data acquisition, acquire training data, the training data includes historical data under normal working condition and safety threshold of various historical data, the historical data includes historical device parameters, historical operation data and historical environment data.

[0051] The device parameters are used to reflect the category and health status of the target device, and the device parameters include device model, temperature, pressure, vibration frequency, current voltage, etc.

[0052] The operation data is used to embody the operation behavior mode, and includes a load rate, a start-stop frequency, a running time length, a fault code record, and the like.

[0053] The environmental data includes temperature and humidity, dust concentration, gas composition, light intensity, and the like, and refers to the environmental data collected at the same time as the equipment parameter and the operation data.

[0054] The safety threshold label refers to a safety range corresponding to historical data at different time points in the whole life cycle of the target equipment.

[0055] In S22, the training data is divided into a plurality of historical segments according to a preset time window (for example, 1 hour), and the historical data in the historical segments is recorded as first data; the historical data of the target equipment in the preset time window before the current time is obtained and recorded as second data; The KL divergence of each first data and second data in the preset time window is calculated, that is, a certain preset time window is recorded as a target window, the historical segments in the target window include historical equipment parameters, historical operation data and historical environmental data, at this time, the KL divergence of the historical equipment parameters in the target window and the equipment parameter part in the second data is calculated, the KL divergence of the historical operation data in the target window and the operation data part in the second data is calculated, and the KL divergence of the historical environmental data in the target window and the environmental parameter part in the second data is calculated. The calculation formula of the KL divergence is mature and will not be repeated here.

[0056] All the KL divergences in the same preset time window are arranged in ascending order to obtain a KL divergence sequence, and the KL divergences are divided into m quantile intervals based on the KL divergence sequence, and each quantile interval represents a different data distribution level.

[0057] A weight is assigned to each quantile interval, and the sum of the weights of all quantile intervals is equal to 1, the quantile number of each quantile interval is calculated, and the quantile number of the kth quantile interval is calculated by accumulating the weight, and the calculation model is as follows: ; Wherein, is the quantile number of the kth quantile interval; i is the serial number of the quantile interval; is the weight of the i th quantile interval.

[0058] The calculation model of the drift index D is as follows: ; Wherein, m is the number of quantile intervals; is the weight of the i th quantile interval. is the quantile number of the i th quantile interval. The average of the quantiles.

[0059] S23 drift judgment, when the drift index is greater than the preset drift threshold, an alarm signal is sent; otherwise, S24 model training is performed.

[0060] S24 model training, the pre-constructed machine learning model is trained using training data. During the training process, historical data is training data, and the safety threshold corresponding to each historical data is a training label. A trained machine learning model is obtained.

[0061] The machine learning model can select a multi-output fully connected network model in a neural network or a quantile regression neural network model (QRNN). A multi-task learning mode is adopted to output safety thresholds of different types of historical data. Taking the QRNN model as an example, the training process is as follows: The input data includes device parameters (such as temperature, vibration frequency), operation data (load rate, operation time), environmental data (temperature and humidity), and other features that are strongly related to the safety threshold. For continuous features (such as temperature), Z-score standardization processing (mean value of 0, variance of 1) is adopted, and for categorical features (such as device type), One-Hot encoding processing is adopted.

[0062] The quantiles of each quantile interval in the historical data are respectively assigned to the low / middle / high risk threshold. At this time, the training label is the actual value of each quantile in the historical data.

[0063] The QRNN model includes an input layer, a hidden layer, and an output layer. The specific structure is as follows: The number of neurons in the input layer is equal to the feature dimension; The hidden layer includes a fully connected layer and an activation function. The fully connected layer is set to 2-4 layers, with 64-256 neurons in each layer (adjusted according to data complexity). The activation function uses the ReLU function. The output layer selects multi-head output. This embodiment sets 3 output heads, each with 1 neuron (to predict low / middle / high thresholds respectively).

[0064] The QRNN model selects a quantile regression loss function. The calculation model is as follows: ; Wherein, is the jth real value; is the jth predicted value; b is the target quantile, for example, b=0.1 corresponds to the low threshold, b=0.4 corresponds to the middle threshold, and b=0.9 corresponds to the high threshold; N is the total number of real values or predicted values.

[0065] The QRNN model selects the Adam optimizer.

[0066] S25 predicts the safety threshold, collects real-time data of the target device, the real-time data including: real-time device parameters, real-time running data and real-time environmental data, inputs the real-time data into the trained machine learning model, and outputs the safety threshold of various real-time data.

[0067] S26 analyzes the real-time data, if the real-time data exceeds the range of the safety threshold of the real-time data predicted in S25, the blocking process is triggered, and an alarm signal is sent to the maintenance personnel, prompting the maintenance personnel that the real-time data at this time is risky, otherwise, no processing is performed.

[0068] The embodiment adopts KL divergence to detect data drift, which can reduce the risk of failure due to working condition changes. The embodiment also uses dynamic threshold to capture device performance degradation, realizing dynamic prediction and risk control of device safety threshold.

[0069] Embodiment 3: Refer to Figure 3 The difference between this embodiment and embodiment 1 is that after the analysis in S12 is performed, the method further includes: S31 checks the real-time operation picture, acquires the real-time operation picture of the risk step of the maintenance end through the AR smart glasses, and calls the standard operation picture and the non-standard operation picture of the risk step in the risk library.

[0070] The standard operation picture refers to performing the risk step according to the industry standard or enterprise standard operation procedure by a senior maintenance personnel, and collecting key frames such as hand position, closing force visualization and instrument value through AR smart glasses or a same-parameter camera from multiple angles. For example, the risk step of “power switch closing” needs to collect at least 500 pictures to cover different personnel, different device models and different light scenes.

[0071] The non-standard operation picture refers to simulating common error operations, such as not disconnecting the power before disassembling the wire, wrench size inconsistency, lack of protective equipment, etc. The key frames of the error operations also need to be collected from multiple angles, and the number is comparable to that of the standard picture. In addition, the error type needs to be clearly labeled, such as power disconnection omission and tool error.

[0072] The standard operation picture and the non-standard operation picture are used to train the CNN model to obtain the trained CNN model. The trained CNN model can identify whether the real-time operation picture is a standard operation.

[0073] The real-time operation picture is input into the trained CNN model to obtain a classification result. When the classification result does not meet the expectation, an alarm bounding box is pushed to the maintenance end through the AR smart glasses, and the maintenance end is instructed to re-maintain the risk step. When the maintenance end re-maintains the risk step, the method further includes S32 calculating an initial similarity score.

[0074] S32 calculates an initial similarity score, captures 21 hand key points of the maintenance personnel using a computer vision algorithm, maps the hand key points to a three-dimensional space based on the hand key points and using SLAM technology, and generates a continuous hand motion trajectory, performs DTW time alignment on the hand motion trajectory and a standard operation trajectory pre-stored in the risk library, and calculates a dynamic time warping distance as an initial similarity score.

[0075] S33 initial similarity score judgment, judges whether the initial similarity score is lower than the similarity score threshold, if yes, executes S34 to generate a path; if no, does not process.

[0076] S34 generates a path, decomposes the risk steps to obtain a plurality of key links in the risk steps, and extracts key points of each key link, wherein the key points of each key link include: human body key points, target device key points and target tool key points.

[0077] A path planning algorithm is used to generate a maintenance posture path based on all the key points, and the process is as follows: The maintenance space is modeled as a grid map, and each grid represents a 3D space of 1cm x 1cm x 1cm, and dangerous grids are labeled, such as live areas, areas with device obstacles, etc.

[0078] The starting key point of the key link is taken as the starting point, such as the handle key point when the wrench does not contact the bolt, and the target key point is taken as the end point, such as the handle key point when the wrench completely covers the bolt.

[0079] A heuristic function and a cost function are defined to search for the shortest global path that satisfies the constraints, i.e. a grid sequence.

[0080] The center point of the grid in the grid sequence is taken as the control vertex, and a cubic B-spline curve is used to connect it to generate a continuous and smooth path (i.e. a maintenance posture path), and it is checked whether the smoothed path satisfies the ergonomics constraints (such as joint angles), if not, the control vertex is adjusted to reinsert the value.

[0081] S35 calculates a final similarity score, performs DTW time alignment processing on the hand trajectory when the maintenance personnel re-operates and the maintenance posture path generated in S34 to obtain a final similarity score, if the final similarity score is lower than the similarity score threshold, a red warning trajectory is superimposed in the AR interface; if the final similarity score is not lower than the similarity score threshold, a signal of successful maintenance is output.

[0082] The embodiment further analyzes and processes the risk step when the initial similarity score of the maintenance personnel and the standard operation trajectory is low, generates a maintenance posture path for the step, calculates a final similarity score, and if the final similarity score is still lower than the preset similarity threshold, superimposes a red warning trajectory in the AR interface, otherwise, it is considered that the maintenance is successful. The embodiment realizes the full-process closed-loop control from picture compliance verification to trajectory precision correction of the maintenance operation.

[0083] Embodiment 4: Refer to Figure 4 The difference between the embodiment and the embodiment 2 is that, in the step S12 of performing analysis, after triggering the blocking process, the method further comprises: S41 blocking analysis, the duration of the blocking process is counted, when the duration of the blocking process exceeds a preset time threshold, a assistance request package containing the three-dimensional point cloud data of the target device in the risk step, the real-time data of the target device, and the hash value of the historical operation record is automatically generated, and the assistance request package is sent to the remote expert terminal through the 5G network. After receiving the assistance request package, the remote expert terminal establishes a real-time audio and video and data interaction channel, namely an AR collaboration channel, with the on-site maintenance terminal through the AR device.

[0084] S42 label arrow, the expert locks the position of the target device corresponding to the risk step by calling the space anchor point through the SLAM technology, and labels a three-dimensional arrow in the AR interface of the maintenance terminal to guide the maintenance personnel to operate.

[0085] Based on the world coordinate system coordinates and rotation matrix of the position of the corresponding target device recorded by the SLAM algorithm, a dynamic guidance sequence containing a timestamp is generated. When the maintenance personnel moves, the SLAM technology is used to match the position of the risk step with the coordinate value of the space anchor point in real time, and the pose of the three-dimensional arrow is dynamically updated through the rigid transformation algorithm.

[0086] S43 conflict detection, when multiple experts intervene at the same time, the three-dimensional arrows labeled by each expert are detected for conflict through a consensus algorithm, including: The three-dimensional arrows labeled by each expert are converted into a unified world coordinate system through the SLAM algorithm, and the coordinate format is .

[0087] Based on the three-dimensional coordinates The Euclidean distance between any two three-dimensional arrow position points is calculated, if the Euclidean distance is less than a preset distance threshold, it is determined that the two three-dimensional arrows corresponding to the Euclidean distance less than the preset distance threshold have a position conflict, an alarm signal is sent to all remote expert terminals, the experts are asked to vote to retain which three-dimensional arrow within a certain time, and the majority voting result is executed.

[0088] The embodiment utilizes the SLAM technology to accurately position the target device and dynamically guide the maintenance operation, thereby improving the maintenance efficiency and collaboration accuracy.

[0089] Embodiment 5: The embodiment discloses an AR smart glasses-based fault processing auxiliary system, the system comprising a memory and a processor, The memory stores a computer readable storage medium; The processor processes a computer program stored on the computer readable storage medium to implement an AR smart glasses-based fault processing auxiliary method.

[0090] The above are preferred embodiments of the application, not to limit the protection scope of the application, therefore: all equivalent changes made according to the structure, shape, principle of the application should be covered within the protection scope of the application.

Claims

1. A fault handling auxiliary method based on AR smart glasses, characterized in that: include: Use AR smart glasses to scan the unique ID of the target device. Based on the analysis result of the unique ID, retrieve the maintenance workflow of the target device from the pre-built risk library, split the maintenance workflow into maintenance steps, mark the maintenance steps with risks as risk steps, and add blocking processes before the risk steps; If the current step is a risk step, the blocking process is triggered. After receiving confirmation feedback on the risk step from the maintenance end, the blocking is released and the risk step in the maintenance workflow is continued.

2. The fault handling auxiliary method based on AR smart glasses according to claim 1 is characterized in that: Before continuing with the risk step in the troubleshooting workflow, the method further includes: Acquiring training data, the training data including historical data under normal operating conditions and safety thresholds of various historical data, the historical data including historical equipment parameters, historical operating data, and historical environmental data, and training a pre-built machine learning model using the training data to obtain a trained machine learning model; Collect real-time data from target devices, including real-time device parameters, real-time operating data, and real-time environmental data. Input the real-time data into a trained machine learning model, and output safety thresholds for various real-time data. If the real-time data exceeds the safety threshold of the real-time data, the blocking process is triggered.

3. The fault handling auxiliary method based on AR smart glasses according to claim 2 is characterized in that: The method further comprises: Segment the historical data in the training data using a preset time window to obtain the segmented historical data, which is recorded as first data; obtain the historical data of the target device in the preset time window before the current moment, which is recorded as second data; Calculate the KL divergence of each first data and second data within the preset time window, use the weighted quantile sum algorithm, calculate the drift index based on all KL divergences within the preset time window, and issue an alarm signal when the drift index is greater than the preset drift threshold.

4. The fault handling auxiliary method based on AR smart glasses according to claim 1, characterized in that: After unblocking, the method further comprises: The real-time operation screen of the maintenance end for the risk step is obtained through AR smart glasses in real time, and the standard operation screens and non-standard operation screens of the risk steps in the risk library are retrieved. The CNN model is trained using the standard operation screens and non-standard screens to obtain the trained CNN model. The real-time operation screen is input into the trained CNN model to obtain the classification result. When the classification result does not meet the expectation, the AR smart glasses push the warning mark box to the maintenance end, and instruct the maintenance end to re-inspect the risk step.

5. The fault handling auxiliary method based on AR smart glasses according to claim 4 is characterized in that: When the maintenance end re-maintains the risk step, the method further includes: A computer vision algorithm is used to capture the key points of the maintenance personnel's hands. Based on the hand key points and using SLAM technology, the maintenance personnel's hand motion trajectory is reconstructed. The hand motion trajectory is then time-aligned with the standard operation trajectory stored in the risk library through DTW to obtain an initial similarity score. When the initial similarity score is lower than the similarity score threshold, a red warning trajectory is superimposed on the AR interface.

6. The fault handling auxiliary method based on AR smart glasses according to claim 5 is characterized in that: Before superimposing the red warning track on the AR interface, the method further includes: The risk steps are decomposed to obtain multiple key links in the risk steps. The key points of each key link are extracted. A path planning algorithm is used to generate a maintenance posture path based on all key points. The final similarity score between the hand motion trajectory and the maintenance posture path is calculated. If the final similarity score is lower than the similarity score threshold, a red warning trajectory is superimposed on the AR interface; if the final similarity score is not lower than the similarity score threshold, a maintenance success signal is output.

7. The fault handling auxiliary method based on AR smart glasses according to claim 2, characterized in that: After triggering the blocking process, the method further includes: Count the duration of the blocking process. When the blocking process duration exceeds the preset time threshold, generate an assistance request package and send it to the remote expert terminal via the 5G network to establish an AR collaboration channel. The assistance request package includes: 3D point cloud data of the target device in the risk step, real-time data of the target device, and hash values ​​of historical operation records; Experts use spatial anchor points to lock the location of risky steps, and mark three-dimensional arrows at the location of risky steps in the AR interface of the maintenance end to guide maintenance personnel to perform operations.

8. The fault handling auxiliary method based on AR smart glasses according to claim 7 is characterized in that: The method further comprises: The spatial coordinates and rotation matrix of the 3D arrow are recorded through the spatial anchor point to generate a dynamic guidance sequence including a timestamp; When the maintenance personnel move, SLAM technology is used to match the position of the risk step with the coordinate value of the spatial anchor point in real time, and the position of the three-dimensional arrow is dynamically updated through the rigid transformation algorithm.

9. The fault handling auxiliary method based on AR smart glasses according to claim 7, characterized in that: When multiple experts are involved at the same time, a consensus algorithm is used to detect conflicts between the 3D arrows marked by each expert, including: The Euclidean distance between the three-dimensional arrows marked by each expert is calculated. If the Euclidean distance is less than the preset distance threshold, an alarm signal is output.

10. A fault handling auxiliary system based on AR smart glasses, characterized in that: include: memory and processor, The memory stores a computer-readable storage medium; When the processor processes the computer program stored on the computer-readable storage medium, the method according to any one of claims 1 to 9 is implemented.

Images