Intelligent inspection method and system based on AR (Augmented Reality) equipment

Abstract

The invention provides an intelligent inspection method and system based on AR equipment, and relates to the technical field of intelligent inspection, and the method comprises the steps: obtaining real-time operation data, historical operation data, a real-time image and a historical image of target equipment, and adding category labels for the historical operation data and the image; constructing an equipment state diagnosis model containing a BILSTM sub-model and a CNN sub-model by adopting a deep learning algorithm, and training the equipment state diagnosis model by using the labeled historical data and images; and inputting the real-time operation data and the image into the trained model to output a diagnosis result, and if the diagnosis result does not conform to expectation, scanning the block chain NFT by using AR equipment, and calling a standard operation process in the block chain through an intelligent contract for inspection personnel to maintain the equipment. According to the invention, targeted suggestions can be provided for inspection personnel, and the standardization degree of the fault processing flow is improved.

Description

Technical Field

[0001] The present application relates to the technical field of intelligent inspection, and in particular to an intelligent inspection method and system based on AR equipment. Background Art

[0002] With the rapid development of Industry 4.0 and smart manufacturing, the demand for intelligent equipment inspection is growing. Traditional manual inspection methods are inefficient and rely on the inspector's experience, which can easily lead to missed inspections or false positives due to subjective judgment. In recent years, the combination of augmented reality (AR) technology and artificial intelligence (AI) has provided a new solution for intelligent inspection. Through real-time data collection, interactive visualization, and intelligent analysis, it has significantly improved the accuracy and efficiency of inspections.

[0003] The Chinese invention patent application publication number CN113222184A provides an equipment inspection system and method based on augmented reality (AR). The patent uses AR devices to collect equipment identification and upload it to the inspection management platform. The platform retrieves virtual content such as equipment three-dimensional models, inspection instructions, maintenance information, etc., and superimposes them on the real scene in real time to provide visual operation guidance for inspection personnel, support remote expert collaborative guidance, and automatically generate standardized records and associate historical data after the inspection is completed.

[0004] However, the inspection guidance in this patent relies on text, pictures or animations stored in the database, but does not explicitly mention the dynamic association with the SOP, and the guidance information may lack contextual specificity. Summary of the Invention

[0005] In order to provide targeted suggestions to inspection personnel and improve the standardization of troubleshooting processes, this application provides an intelligent inspection method and system based on AR devices.

[0006] In the first aspect, this application provides an intelligent inspection method based on AR devices, which adopts the following technical solutions: The intelligent inspection method based on AR equipment includes the following steps: Obtain real-time operation data, historical operation data, real-time images, and historical images of the target device, and add category labels to the historical operation data and historical images respectively; A deep learning algorithm is used to construct an equipment status diagnosis model, which includes a BILSTM sub-model and a CNN sub-model. The equipment status diagnosis model is trained using historical operating data and historical images with added category labels to obtain a trained equipment status diagnosis model. Real-time operation data and real-time images are input into the trained equipment status diagnosis model to output the diagnosis results. When the diagnosis results do not meet expectations, the AR device is used to scan the preset blockchain NFT, and the standard operating procedures stored in the blockchain are retrieved through the smart contract. The inspection personnel then inspect and repair the target equipment according to the standard operating procedures.

[0007] This application first obtains the real-time operating data, historical operating data, real-time images, and historical images of the target device, and adds category labels to the historical operating data and historical images. Subsequently, this application uses a deep learning algorithm to build a device status diagnosis model. This model includes a BILSTM sub-model and a CNN sub-model. The BILSTM (Bidirectional Long Short-Term Memory Network) sub-model excels at processing sequence data and can effectively capture the time series features and long-term dependencies in historical operating data. It has unique advantages for analyzing the changing trends of the target device's operating status over time. The CNN (Convolutional Neural Network) sub-model can automatically extract features from historical images and identify anomalies or changes in the target device. By combining the two, this application can fully leverage the advantages of different models and comprehensively and accurately diagnose the real-time status of the target device from both the operating data and image perspectives.

[0008] Subsequently, this application uses historical operation data and historical images with added category labels to train the equipment status diagnosis model, so that the equipment status diagnosis model learns the equipment status characteristics corresponding to different categories of data during the training process, thereby improving the equipment status diagnosis model's recognition accuracy of the equipment status. The trained equipment status diagnosis model can better identify various states of the target equipment.

[0009] Subsequently, this application inputs real-time operation data and real-time images into the trained equipment status diagnosis model to obtain diagnosis results. The real-time diagnosis method adopted in this application can promptly detect problems or abnormalities that occur in the target equipment during operation, and provide inspection personnel with instant equipment status information, which helps to take measures as early as possible to avoid further expansion of equipment failures, reduce production interruptions and losses caused by target equipment failures, and improve the reliability and production efficiency of target equipment.

[0010] When the diagnostic results don't meet expectations, AR devices are used to scan preset blockchain NFTs. Smart contracts are then used to retrieve the standard operating procedures stored on the blockchain, providing inspectors with standardized troubleshooting guidance. Blockchain technology enhances the security and immutability of standard operating procedures, enabling inspectors to obtain accurate and reliable operational information. When inspectors perform repairs on target equipment according to standard operating procedures, this improves the standardization and effectiveness of troubleshooting, reduces the risk of problems worsening due to improper operation, and improves inspectors' work efficiency and troubleshooting capabilities.

[0011] This application uses a variety of advanced technical means to achieve automatic monitoring, diagnosis and fault handling guidance of equipment status, reduce manual intervention, improve the efficiency and accuracy of equipment management, and promote the development of equipment inspection towards intelligence and automation.

[0012] Optionally, when the diagnosis result does not meet expectations, the method further includes: Obtain a depth map of the target device, convert the depth map into 3D point cloud data, perform Euclidean clustering on the 3D point cloud data to obtain multiple point cloud segments, and add semantic labels and virtual anchors to each point cloud segmentation block. Based on the diagnosis results, a virtual indicator containing the fault location is generated, a mapping relationship between the fault location and the semantic label is established, and the virtual indicator is associated with the corresponding virtual anchor point according to the mapping relationship, so as to display the virtual indicator in the field of view of the AR device.

[0013] This application first obtains a depth map of the target device. The depth map contains the distance information from each point on the surface of the target device to the AR device. Compared with ordinary two-dimensional images, it can provide three-dimensional spatial structure information of the target device, so that subsequent analysis of the target device is no longer limited to the surface appearance features, but can go deep into the three-dimensional form of the target device, providing a richer data basis for accurately identifying the fault location of the target device. For some target devices with complex internal structures or three-dimensional appearances, the depth map can help clearly present the relative positions and spatial relationships of their various parts.

[0014] Subsequently, this application converts the depth map into three-dimensional point cloud data. The three-dimensional point cloud data represents the geometric shape of the surface of the target device in the form of discrete points. Each point contains its coordinate information in three-dimensional space, which facilitates various geometric operations and analyses, thereby more accurately describing the outline and details of the target device.

[0015] Subsequently, the present application performs Euclidean clustering segmentation on the three-dimensional point cloud data, dividing the point cloud data into multiple point cloud segmentation blocks according to the spatial distance between points, which helps to decompose the target device into different components or areas. Through clustering segmentation, the present application can distinguish the different components of the target device. Even if these components are similar in appearance, each component can be accurately identified based on their distribution and connection relationship in space, thereby facilitating separate fault detection and analysis of each component.

[0016] Subsequently, the present application adds semantic labels to each point cloud segmentation block, giving semantic information to the three-dimensional point cloud data. The semantic labels can correspond the segmented point cloud blocks to the actual components or functional areas of the target device, so that the computer can understand the actual meaning represented by each point cloud segmentation block. Adding virtual anchor points provides a reference point for the positioning and display of virtual indicators. The virtual anchor point is a fixed position pre-set in three-dimensional space and is associated with the point cloud segmentation block. By adding virtual anchor points to the point cloud segmentation blocks, the present application can accurately place the virtual indicator on the corresponding component of the target device, so that the indication information displayed in the field of view of the AR device corresponds to the actual device position.

[0017] This application generates a virtual indicator containing the fault location based on the diagnosis results, which can convert the abstract diagnosis results into intuitive visual information. Subsequently, this application establishes a mapping relationship between the fault location and the semantic label, and associates the virtual indicator with the corresponding virtual anchor point according to the mapping relationship, thereby achieving a precise correspondence between the virtual indicator and the actual equipment fault location. This application uses semantic labels as an intermediate bridge to link the diagnosed fault location with the point cloud segmentation block, and then uses the virtual anchor point to accurately position the virtual indicator on the target device, so that the virtual indicator can be accurately displayed at the actual location where the fault occurs, providing patrol personnel with accurate fault guidance and improving the efficiency and accuracy of fault handling.

[0018] Optionally, before adding semantic labels to each point cloud segment block, the method further includes: The ORB-SLAM3 algorithm is used to extract environmental feature points from the historical image and real-time image at the previous moment through the FAST feature point detector, and an initial weighted co-image map is constructed based on all the environmental feature points. The preset SuperPoint neural network is used to extract key feature points in the real-time image, and non-maximum suppression processing is performed on the key feature points to obtain key feature points whose response values ​​are greater than the preset response value threshold and record them as target feature points; Match the environmental feature points and target feature points in the real-time image. If the match is successful, increase the weight of the corresponding edge in the initial weighted co-graph and output the final weighted co-graph. Based on the final weighted co-visual map, the RANSAC algorithm is used to calculate the initial pose, and the optimal pose is obtained using SVD decomposition. A frustum with a 60° cone angle is constructed with the optical center of the camera of the inspector's current optimal pose as the vertex, and the frustum clipping technology is used to retain the point cloud segments within the frustum space.

[0019] This application adopts the improved ORB-SLAM3 algorithm and uses the FAST feature point detector to extract environmental feature points in the historical image and real-time image of the previous moment. The FAST feature point detector can quickly capture environmental feature points such as corner points in the image, which helps to accurately describe the changes in the environment around the target device at different times.

[0020] Subsequently, this application constructed an initial weighted co-visual map based on all environmental feature points and obtained the correlation between environmental feature points between different image frames. The initial weighted co-visual map further considered the degree of co-visibility between environmental feature points in different images. The larger the weight, the higher the probability that two environmental feature points appear simultaneously in multiple image frames, that is, the stronger their correlation, thereby being able to more accurately describe the spatial and temporal relationship between environmental feature points.

[0021] Subsequently, the present application uses a preset SuperPoint neural network to extract key feature points in the real-time image. Compared with traditional feature point detection methods, the SuperPoint neural network can extract more representative and stable key feature points, which helps to improve the accuracy of feature matching. Afterwards, the present application performs non-maximum suppression processing on the key feature points, obtains key feature points with response values ​​greater than the preset response value threshold, records them as target feature points, and matches the environmental feature points and target feature points in the real-time image. If the match is successful, the weight of the corresponding edge in the initial weighted co-graph is increased, and the final weighted co-graph is output. The initial weighted co-graph is constructed based on the environmental feature points, and its weight reflects the number of matches of the environmental feature points. When the target feature points (i.e., features with higher discrimination) extracted by the SuperPoint neural network are successfully matched with the environmental feature points in the real-time image, the weight of the corresponding edge is increased to obtain the final weighted co-graph. The present application uses the semantic perception of the SuperPoint neural network to compensate for the limitations of traditional features and can improve the reliability of cross-frame matching through the consistency of feature points within the same frame.

[0022] Subsequently, this application uses the RANSAC algorithm to calculate the initial pose based on the final weighted co-graph. In the pose calculation, the RANSAC algorithm can filter out the inliers (i.e., points that conform to the pose model) from the correspondence between the feature points in the final weighted co-graph, thereby calculating a relatively accurate initial pose. Subsequently, this application uses the SVD decomposition algorithm to find the optimal pose. The current optimal pose of the inspector is obtained through the above method. A 60° cone-angle viewing cone is constructed with the optical center of the camera in this pose as the vertex. The cone clipping technique is used to retain the point cloud segments within the cone space and remove the point cloud segments outside the cone space. This solution reduces the amount of data that needs to be processed and improves the efficiency and pertinence of data processing.

[0023] Optionally, before associating the virtual indicator with the corresponding virtual anchor point according to the mapping relationship, the method further includes: Obtain the DBoW3 library of the ORB-SLAM3 algorithm, construct a first vocabulary tree based on environmental feature points and ORB descriptors, construct a second vocabulary tree using key feature points and SuperPoint descriptors, merge the root nodes of the first and second vocabulary trees to obtain a hybrid bag-of-words structure; Based on the hybrid bag-of-words structure, the TF-IDF scores of the environmental feature points and the key feature points in the historical image are calculated. A comprehensive score is calculated based on the TF-IDF scores of the environmental feature points and the TF-IDF scores of the key feature points in the historical image. Historical images with a comprehensive score greater than a preset score threshold are selected as candidate loop images. The similarity between the real-time image and each candidate loop image is calculated separately and recorded as the first data. If there is a first data that is greater than the first preset similarity threshold, the virtual anchor point of the candidate loop image corresponding to the maximum first data is used as the virtual anchor point of the real-time image.

[0024] This application first obtains the DBoW3 library of the ORB-SLAM3 algorithm, constructs a first vocabulary tree based on environmental feature points and ORB descriptors, then uses key feature points and SuperPoint descriptors to construct a second vocabulary tree, and then merges the root nodes of the two to obtain a hybrid bag-of-words structure. Since different descriptors describe features differently, the hybrid bag-of-words structure fuses them together, increasing the dimension and complexity of the features, thereby enabling this application to more comprehensively and accurately represent the feature information in the image when facing complex environments, better distinguish different image areas and objects, and reduce the possibility of feature confusion.

[0025] Subsequently, this application calculates the TF-IDF scores of environmental feature points and key feature points in historical images based on the hybrid bag-of-words structure, and further calculates the comprehensive score. After that, historical images with a comprehensive score greater than a preset score threshold are used as candidate loop images, thereby filtering out those historical images that are dissimilar to the current scene or have a low similarity. During the inspection process, there may be a large number of historical images of the target device, of which only a few are relevant to the current scene. By pre-setting the score threshold, this application can quickly screen out candidate loop images (i.e., images where the current scene is similar to the historical scene).

[0026] Subsequently, the present application calculates the similarity between the real-time image and each candidate loop image respectively, and records it as the first data. If there is a situation where the first data is greater than the first preset similarity threshold, it means that the real-time image and a candidate loop image form a loop, and the virtual anchor point of the candidate loop image corresponding to the maximum first data is used as the virtual anchor point of the real-time image. By calculating the similarity, the present application can accurately find the historical loop image that best matches the real-time image. Since the position of the virtual anchor point has been determined in the historical loop image, the corresponding virtual anchor point is used as the virtual anchor point of the real-time image, which can realize the accurate transfer and positioning of the virtual anchor point between different images, thereby better correcting the accumulated error and improving the accuracy of positioning.

[0027] Optionally, if there is no first data with a similarity greater than a first preset similarity threshold, the method further includes: The similarity between the first vocabulary tree and the second vocabulary tree is calculated by a cross-validation algorithm, recorded as the second data. When the second data exceeds a second preset similarity threshold, the target feature point is projected into the ICP point cloud registration framework to generate matching point pairs with topological constraints. A graph optimization model is constructed, and the LM algorithm is used to optimize the 3D coordinates of the virtual anchor points using the reprojection error of the matching point pairs as the edge constraint.

[0028] The present application calculates the similarity between the first vocabulary tree and the second vocabulary tree through a cross-validation algorithm, and records the similarity between the two as the second data. When the first data fails to reach the similarity threshold, it means that the conventional matching method based on image similarity may not be able to accurately find the loop image. At this time, by cross-validating the similarity of the two vocabulary trees, the present application can dig out potential matching relationships.

[0029] When the second data exceeds the second preset similarity threshold, the present application projects the target feature points into the ICP point cloud registration framework to generate matching point pairs with topological constraints. The ICP algorithm finds the nearest point pairs between the two sets of point clouds and continuously iterates and optimizes the registration parameters to achieve precise alignment of the point clouds. By projecting the target feature points into the ICP framework, the semantic information and topological structure of the feature points can be combined to generate more constrained matching point pairs. These matching point pairs not only consider the spatial distance between points, but also the positional relationship of the feature points in the overall structure, thereby improving the accuracy and robustness of the matching.

[0030] Subsequently, this application constructs a graph optimization model, uses the reprojection error of matching point pairs as edge constraints, and adopts the LM (Levenberg-Marquardt) algorithm to optimize the three-dimensional coordinates of the virtual anchor point. In the graph optimization model, the three-dimensional coordinates of the virtual anchor point are used as optimization variables, and the position of the virtual anchor point is optimized by minimizing the reprojection error. The LM algorithm can effectively reduce the reprojection error, thereby improving the positioning accuracy of the three-dimensional coordinates of the virtual anchor point.

[0031] Optionally, when the diagnosis result does not meet expectations, the method further includes: Decompose the standard operating procedure into multiple operating steps, generate a unique NFT certificate for each operating step, and record the NFT certificate of the i-th operating step as NFT i Voucher, the NFT i The credential includes the step number i, the security level of the i-th operation step, and the time constraint parameters of the i-th operation step; Determine whether the standard operating procedure requires multiple inspectors to operate together. If so, deploy the verification logic of the standard operating procedure in the blockchain; if not, do nothing.

[0032] This application breaks down the standard operating procedure into multiple operating steps and generates a unique NFT certificate for each operating step. Afterwards, it determines whether the standard operating procedure requires multiple inspectors to operate together. When multiple inspectors are required to operate together, the verification logic of the standard operating procedure is deployed in the blockchain. The verification logic can monitor and verify the operating steps of each inspector in real time. Only when all inspectors complete the operation according to the prescribed steps and time requirements, the entire operating process is considered completed, further improving the standardization of the operating process.

[0033] This application deploys the verification logic of standard operating procedures on the blockchain, leveraging the blockchain's immutability and decentralization to improve the fairness and transparency of the verification process. The blockchain's distributed storage and consensus mechanism eliminates the control of the verification logic by a single node or individual, effectively preventing human interference and tampering. Traditional operational process management can potentially involve manual modification of operation records or bypassing verification steps. The application of blockchain technology can prevent this from occurring, enhancing the seriousness and standardization of operational processes.

[0034] Optionally, the verification logic includes: After the inspector responsible for the i-1th operation step completes the i-1th operation step, the NFT is output. i-1 Certificate and generate NFT i Credentials, the inspector responsible for the i-th operation step uses the AR device to scan the preset blockchain NFT to activate the NFT i certificate.

[0035] This application will output NFT only after the inspector responsible for the i-1 operation step completes the step i - 1 Certificate and generate NFT iThe certificate can clearly define the order of the operation steps. Only when the previous step is successfully completed, the certificate of the subsequent step will be generated. This fundamentally ensures that the operation process is carried out in the predetermined order, and tries to avoid safety accidents or diagnostic errors caused by confusion in the operation sequence.

[0036] Since the credential generation for each step depends on the completion of the previous step, if the inspector attempts to skip a step and proceed directly to the subsequent operation, he will not be able to obtain the corresponding NFT credential and thus will not be able to activate the operations of the subsequent steps. The above solution can effectively prevent omissions or skipping during the operation process, so that each step is executed, improving the quality and reliability of the operation.

[0037] Optionally, the verification logic further includes: After the inspector responsible for the i-1th operation step completes the i-1th operation step, the AR device is used to obtain images after the operation is completed, and the operating data after the operation is completed is collected. The images and operating data after the operation are input into the trained equipment status diagnosis model to obtain the diagnosis results after the operation is completed; If the diagnosis result after the operation is completed does not meet expectations, a virtual conference room is built. After the expert wears the VR device, he or she controls and views the target device through the gesture recognition API of the VR controller.

[0038] After the inspection personnel responsible for the i-1th operation step completes the operation, this application uses the AR device to obtain the image after the operation is completed, and collects the operating data after the operation is completed, and inputs the obtained image and operating data into the trained equipment status diagnosis model, and uses the equipment status diagnosis model to output the diagnosis results after the operation is completed. The trained equipment status diagnosis model is constructed based on a large amount of historical data and professional knowledge, and can perform in-depth analysis and comparison of the input information, so as to more accurately determine whether the status of the target equipment is normal, and realize real-time monitoring of the operation process. When the diagnosis result after the operation is completed does not meet expectations, this application builds a virtual conference room so that experts can wear VR equipment and view the status of the target equipment through gesture control. This remote collaboration method breaks the limitations of time and space. Experts do not need to be present on site to understand the actual status of the equipment in real time and provide guidance and suggestions for problem solving.

[0039] Optionally, the method further includes: The diagnostic result after the operation is completed is used as the trigger condition of the smart contract. When the diagnostic result after the operation is completed does not meet the expectations, an abnormal event record is generated and stored in the distributed ledger.

[0040] The diagnostic result after the operation is completed is used as the trigger condition of the smart contract. When the diagnostic result after the operation is completed does not meet expectations, the smart contract will be automatically triggered and the subsequent processing process can be started without human intervention, which improves the timeliness and efficiency of problem handling and minimizes the escalation of problems caused by manual operation delays.

[0041] Secondly, this application provides an intelligent inspection system based on AR devices, which adopts the following technical solutions: The intelligent inspection system based on AR equipment includes: memory and processor, The memory stores a computer-readable storage medium; When the processor processes the computer program stored on the computer-readable storage medium, the method according to the first aspect is implemented.

[0042] In summary, this application includes at least one of the following beneficial technical effects: 1. This application first uses the equipment status diagnosis model to diagnose real-time data and obtain diagnostic results. When the diagnostic results do not meet expectations, an AR device is used to scan the preset blockchain NFT, and the standard operating procedures stored in the blockchain are retrieved through smart contracts, providing standardized troubleshooting guidance for inspection personnel. Blockchain technology improves the security and immutability of standard operating procedures, allowing inspection personnel to obtain accurate and reliable operating information. Inspection personnel inspect and repair target equipment according to standard operating procedures, which can improve the standardization and effectiveness of fault handling, reduce the risk of further deterioration of problems due to improper operation, and also help improve the work efficiency and troubleshooting capabilities of inspection personnel.

[0043] 2. This application generates a virtual indicator containing the fault location based on the diagnosis results, which can convert the abstract diagnosis results into intuitive visual information. Subsequently, this application establishes a mapping relationship between the fault location and the semantic label, and associates the virtual indicator with the corresponding virtual anchor point according to the mapping relationship, thereby achieving a precise correspondence between the virtual indicator and the actual equipment fault location. This application uses semantic labels as an intermediate bridge to link the diagnosed fault location with the point cloud segmentation block, and then uses the virtual anchor point to accurately position the virtual indicator on the target device, so that the virtual indicator can be accurately displayed at the actual location where the fault occurs, providing patrol personnel with accurate fault guidance and improving the efficiency and accuracy of fault handling.

[0044] 3. This application utilizes a variety of technical means to achieve automatic monitoring, diagnosis and fault handling guidance of equipment status, reduce manual intervention, improve the efficiency and accuracy of equipment management, and promote the development of equipment inspection towards intelligence and automation. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a flow chart of Example 1 of the present application; Figure 2 This is a flowchart of Example 2 of the present application. DETAILED DESCRIPTION

[0046] The following combination Figure 1 and Figure 2 This application is described in further detail.

[0047] Example 1: This example discloses an intelligent inspection method based on AR equipment, referring to Figure 1 The method includes: S11 data acquisition, S12 modeling and training, and S13 diagnosis. First, the real-time operation data and historical operation data, real-time images and historical images of the target device are obtained, and category labels are added to the historical operation data and historical images; then, a deep learning algorithm is used to construct a device status diagnosis model containing a BILSTM sub-model and a CNN sub-model, and the model is trained to obtain a trained device status diagnosis model; then, the real-time operation data and real-time images are input into the trained device status diagnosis model to output a diagnosis result. If the diagnosis result does not meet the expectation, the blockchain NFT is scanned with an AR device, and the standard operating procedure in the IPFS distributed storage is retrieved through the smart contract for inspection personnel to repair the equipment. The execution process of each step of this embodiment is as follows: S11 data acquisition obtains real-time operating data of the target device through pre-deployed sensors, obtains real-time images of the target device through the RGB-D camera set in the AR device, and retrieves historical operating data and historical images stored in the blockchain based on the preset blockchain NFT.

[0048] The operating data refers to at least one of temperature, partial discharge, mechanical vibration and gas composition.

[0049] Temperature data includes transformer winding temperature, circuit breaker contact temperature, cable joint temperature, and capacitor case temperature.

[0050] Partial discharge monitoring locations include cable terminals, GIS (gas insulated switchgear), transformer interiors, etc.

[0051] Mechanical vibration monitoring locations include transformer cores, reactors, and pole-mounted switch operating mechanisms. Mechanical vibration data includes vibration frequency (e.g., a loose core can cause 100Hz multiples) and vibration amplitude (RMS values ​​> 5mm / s² may indicate a mechanical failure).

[0052] The monitoring locations of gas composition include SF6 circuit breakers and oil-immersed transformers. The gas components include: SF6 decomposition products (SO2, H2S, reflecting arc or overheating faults), dissolved gases in transformer oil (H2, CH4, C2H2, diagnosing fault types according to the IEC 60599 three-ratio method).

[0053] Category labels are added to historical operation data and historical images respectively, and the category labels include: normal, overload, undervoltage, crack, corrosion, oil leakage, etc.

[0054] S12 modeling and training uses a deep learning algorithm to build a device status diagnosis model, which includes a BILSTM sub-model and a CNN sub-model.

[0055] The BILSTM sub-model includes: input layer, bidirectional LSTM layer (128 neural units) and fully connected layer (64 neural units).

[0056] The CNN sub-model includes: input layer, ResNet-18 backbone network and global average pooling layer.

[0057] The output of the BILSTM sub-model and the output of the CNN sub-model are connected to the attention layer respectively. After the output of the two is processed by the attention layer, the diagnosis results are output through the output layer.

[0058] The equipment status diagnosis model is trained using historical operation data and historical images with added category labels to obtain the trained equipment status diagnosis model. The process is as follows: The historical operation data is sliced ​​according to the preset time window and input into the BILSTM sub-model in the order of collection time. The bidirectional LSTM layer (128 units) extracts the forward and backward time series features of the sliced ​​historical operation data. The fully connected layer (64 units) reduces the dimension of the time series features to obtain the first output data. The historical image is resized to a resolution of 224×224, and the spatial features of the historical image are extracted using the ResNet-18 backbone network. The feature map is output, and the feature map is compressed using a global average pooling layer to obtain the second output data. The first output data and the second output data are input into the attention layer to obtain the weighted target output data, and the target output data is input into the output layer to obtain the classification result.

[0059] Subsequently, basic training is performed using the AdamW optimizer (with an initial learning rate of 0.001) combined with a multi-task loss function (such as FocalLoss and MSE for regression), and the learning rate is dynamically adjusted through Cosine Annealing. Subsequently, Dropout (rate 0.2-0.5), Label Smoothing, and Early Stopping are introduced to prevent overfitting. At the feature layer, the outputs of the CNN sub-model and the BILSTM sub-model are dynamically fused through the attention mechanism until the preset number of iterations is reached or the loss function converges, thereby realizing the training of the device status diagnosis model and obtaining the trained device status diagnosis model.

[0060] S13 diagnosis inputs real-time operation data and real-time images into the trained equipment status diagnosis model and outputs the diagnosis results.

[0061] This implementation encapsulates standard operating procedure (SOP) documents, 3D models, and maintenance videos as NFTs and stores them on the Hyperledger Fabric blockchain. If the diagnostic result is abnormal, a smart contract is automatically invoked to verify the inspector's permissions and return the corresponding NFT URI. If the diagnostic result is normal, no action is taken.

[0062] Inspection personnel wear AR devices to scan the NFT tag on the target device, triggering the smart contract. After the smart contract verifies the device ID and operation permissions, it retrieves the encrypted standard operating procedure file from the IPFS distributed storage.

[0063] The standard operating procedure (SOP) is superimposed on the target equipment in the form of virtual indicators, including automatic highlighting of key parts, real-time floating prompts of digital indicators such as torque parameters, and simultaneous uploading of maintenance process data to the chain for evidence.

[0064] This embodiment diagnoses the real-time operating data and real-time images of the target device, and when the diagnosis results do not meet expectations, calls the standard operating procedure and repairs the target device according to the operating procedure, thereby providing inspection personnel with a standard operating procedure for the current diagnosis results and improving the standardization of the fault handling process.

[0065] Example 2: Reference Figure 2 The difference between this embodiment and embodiment 1 is that, when the diagnosis result does not meet expectations, the method further includes: S21 obtains point cloud segmentation blocks and uses the RGB-D camera or lidar equipped on the AR device to obtain the depth map of the target device.

[0066] According to the pixel coordinates in the depth map And the corresponding depth value d, the calculation model of the three-dimensional coordinates (X, Y, Z) of the three-dimensional point cloud data is as follows: ; ; ; in, is the optical center of the camera, in pixels; is the focal length in pixels.

[0067] The acquired 3D point cloud data is segmented by Euclidean clustering using a DBSCAN variant algorithm based on the Euclidean distance between 3D point cloud data. Specifically, based on the 3D coordinates (X, Y, Z) of the 3D point cloud data, points with a distance less than a distance threshold are grouped into the same cluster to obtain multiple clusters. The 3D point cloud data grouped into one cluster is recorded as a point cloud segmentation block.

[0068] The distance threshold is set according to the size of the target device. For example, if the distance between distribution cabinets is greater than 0.5 meters, the distance threshold is set to 0.4 meters.

[0069] S22 filters the point cloud segmentation blocks, adopts the ORB-SLAM3 algorithm, and uses the FAST feature point detector to extract the detection corner points in the historical image and real-time image of the previous moment, record them as environmental feature points, and construct an initial weighted co-visual map based on all environmental feature points. The nodes of the initial weighted co-visual map are the historical image and real-time image of the previous moment, and the edge weight is the number of matching environmental feature points between the historical image and the real-time image of the previous moment.

[0070] From the above content, we can see that the initial weighted co-graph only contains the connection between the historical image and the real-time image at the previous moment, and the edge weights are calculated using the Brute-Force Matcher algorithm.

[0071] The real-time image is input into the preset SuperPoint neural network, which outputs the key feature point heat map and SuperPoint descriptor. The key feature points are obtained through the heat map non-maximum suppression (NMS), and the key feature points whose response values ​​are greater than the preset response value threshold are recorded as target feature points.

[0072] In this embodiment, the preset response value threshold The calculation model is as follows: ; in, is the standard deviation of the grayscale value of the real-time image; I is the grayscale value in the grayscale image of the real-time image, is the maximum value of the grayscale value.

[0073] In other embodiments, the value of the preset response value threshold may be set as required, such as 0.01.

[0074] The default SuperPoint neural network consists of a shared encoder and two independent decoders: The shared encoder spatially downsamples the input image through convolutional layers, pooling, and nonlinear activation functions to reduce the image dimension. The shared encoder consists of multiple convolutional layers and maximum pooling layers. After three maximum pooling operations, the image size is reduced from H×W to H / 8×W / 8, where H is the height of the image and W is the width of the image.

[0075] After the shared encoder, the SuperPoint neural network is divided into two independent decoder branches, one for target feature point detection and the other for SuperPoint descriptor generation. These two independent decoder branches share the encoder output but learn their own task-specific weights.

[0076] The feature point detector branch is responsible for generating a dense target feature point probability map. The process is as follows: The feature map output by the shared encoder undergoes a series of convolution operations to obtain the probability of each pixel becoming a target feature point. The feature map is processed by Softmax to convert the output of each pixel into a probability distribution, where 65 channels correspond to a local, non-overlapping 8×8 pixel grid area, plus an additional dimension of no interest point. After Softmax processing, the dimension of no interest point is removed, and redundant key feature points are removed through NMS operation to obtain the final key feature point position and confidence. The probability of each pixel being a target feature point is output, and a dense probability map of the target feature point is generated.

[0077] The descriptor generator branch is responsible for extracting features around each detected target feature point and generating a fixed-length descriptor vector. The process is as follows: In order to reduce training memory and computational complexity, the descriptor generator first learns a semi-dense descriptor, performs bilinear interpolation on the semi-dense descriptor to obtain a dense descriptor corresponding to each target feature point, performs L2 normalization on the dense descriptor to ensure that its norm is 1, obtains the final descriptor result, and outputs the descriptor vector for each target feature point.

[0078] This embodiment adopts FLANN's KD tree-based descriptor search method to match the environmental feature points and target feature points in the real-time image. If the match is successful, the weight of the corresponding edge in the initial weighted co-visual map is increased, and the final weighted co-visual map is output.

[0079] In this embodiment, when the environment feature points and the target feature points in the real-time image are successfully matched, the corresponding edge weights in the initial weighted co-graph are updated to 1.

[0080] Three sets of matching point pairs are selected from the final weighted co-visual map, and the camera pose, i.e., the rotation matrix and translation vector, is solved using the RANSAC algorithm. The pose with the largest number of inliers is retained as the initial solution, i.e., the initial pose. A reprojection error function is constructed based on all inliers, and the optimal pose is optimized using the Levenberg-Marquardt algorithm.

[0081] With the optical center of the camera in the current optimal posture of the inspector as the vertex and the principal optical axis as the axis, a viewing cone space with a cone angle of 60° is constructed. For each point cloud segmentation block, the angle between the line from its center point to the optical center of the camera and the principal optical axis is calculated, and the point cloud segmentation blocks with an angle less than 30° of the half cone angle are retained.

[0082] S23 adds semantic labels to the remaining point cloud segments in the point cloud segments filtered by S22, such as incoming switch, busbar, outgoing switch, etc., using machine learning classification or manual labeling.

[0083] Then, a virtual anchor point is added to each point cloud segmentation block, and the position of the virtual anchor point can be the centroid coordinate of the point cloud segmentation block.

[0084] In other embodiments, a local feature descriptor may be used to detect salient points within a segmented block as positions of virtual anchor points.

[0085] S24 establishes a mapping relationship, generates a virtual indicator including the fault location based on the diagnosis result, and establishes a mapping relationship between the fault location and the semantic label.

[0086] S25 determines or optimizes the virtual anchor point, obtains the DBoW3 library of the ORB-SLAM3 algorithm, adopts the K-means clustering method of the DBoW3 library, and constructs a hierarchical vocabulary tree for the ORB descriptor of the training image set, which is recorded as the first vocabulary tree.

[0087] In this embodiment, the first vocabulary tree has 6-10 layers, and the number of branches in each layer is K=10. Each node of the first vocabulary tree is used to store the ORB descriptor cluster center, and the leaf node is used to store the corresponding visual word.

[0088] Similar to the first vocabulary tree, the second vocabulary tree has 6-10 layers, with K=10 branches per layer. A SuperPoint descriptor is randomly selected as the first cluster center. The distances from the remaining SuperPoint descriptors to the first cluster center are calculated, and the probability P of the descriptor being selected as the cluster center is calculated. The SuperPoint descriptor corresponding to the maximum probability P is selected as another cluster center until 10 cluster centers are selected. The calculation model of the probability P is as follows: ; in, is the distance from the ith SuperPoint descriptor to the nearest cluster center.

[0089] Assign the remaining SuperPoint descriptors to the nearest cluster center, recalculate the mean of each cluster as the new cluster center, and output the second vocabulary tree after traversing all SuperPoint descriptors.

[0090] In other embodiments, in order to improve computational efficiency, the SuperPoint descriptor may be subjected to dimensionality reduction processing, and a second vocabulary tree may be constructed based on the SuperPoint descriptor after dimensionality reduction.

[0091] The root nodes of the first vocabulary tree and the second vocabulary tree are merged to obtain a mixed bag-of-words structure.

[0092] Based on the hybrid bag-of-words structure, the TF-IDF scores of the environmental feature points and the TF-IDF scores of the key feature points in the historical images are calculated respectively. A weighted average algorithm is used to calculate the comprehensive score based on the TF-IDF scores of the environmental feature points and the TF-IDF scores of the key feature points in the historical images. Historical images with a comprehensive score greater than the preset score threshold are selected as candidate loop images.

[0093] The similarity between the real-time image and each candidate loop image is calculated separately and recorded as the first data. If there is a first data that is greater than the first preset similarity threshold, the virtual anchor point of the candidate loop image corresponding to the maximum first data is used as the virtual anchor point of the real-time image.

[0094] If there is no first data with a similarity greater than the first preset similarity threshold, it means that the real-time image is significantly different from the candidate loop image and may belong to a new scene. The similarity between the first vocabulary tree and the second vocabulary tree is calculated by the cross-validation algorithm and recorded as the second data. The process is as follows: Traverse each leaf node of the real-time image vocabulary tree (first vocabulary tree) and search for the same node path in the candidate image vocabulary tree (second vocabulary tree).

[0095] For nodes on the overlapping path, weights are assigned according to their depth b , the calculation model of the weight is as follows: ; Where B is the depth of the first vocabulary tree; It is an adjustment parameter with a value range of 0.5-1.

[0096] For all matching point pairs (Lj, Lj') on the overlapping path, extract their TF-IDF values, recorded as TF-IDF(Lj) and TF-IDF(Lj'), and calculate the similarity contribution of the matching point pairs. The calculation model is as follows: ; is the similarity contribution of the matching point pair; min(·) is the minimum operation, the purpose of which is to reduce the interference of high-frequency noise.

[0097] The similarity contributions of all overlapping paths are summed to obtain the similarity between the first vocabulary tree and the second vocabulary. The calculation model of the second data is as follows: ; G is the second data; N is the total number of child nodes of the first vocabulary tree.

[0098] When the second data exceeds the second preset similarity threshold, it indicates that the first vocabulary tree and the second vocabulary tree have sufficient correlation in topological structure. At this time, the target feature points are projected into the ICP point cloud registration framework to generate matching point pairs with topological constraints. The process is as follows: The target feature points are projected into the source point cloud coordinate system of ICP in sequence to generate initial matching point pairs. In the nearest neighbor matching algorithm of the ICP point cloud registration framework, the normal vector direction consistency constraint is added, and the initial matching point pairs with inconsistent normal vectors are proposed to obtain matching point pairs with topological constraints.

[0099] A graph optimization model is constructed, where the nodes of the graph optimization model are the three-dimensional coordinates of virtual anchor points, and the reprojection error of matching point pairs is used as the edge constraint. The calculation model of the reprojection error is as follows: ; Where, e is the reprojection error; is the camera projection function, which is used to map the three-dimensional coordinates to the two-dimensional pixel plane; T1 is the camera pose, including the rotation matrix and translation matrix, that is, the pose transformation matrix of the camera from the world coordinate system to the camera coordinate system; is the three-dimensional coordinate of the virtual anchor point in the k-th matching point pair, that is, the variable to be optimized; is the two-dimensional coordinate of the target feature point actually detected in the actual image by the k-th matching point pair.

[0100] The LM algorithm is used to optimize the three-dimensional coordinates of the virtual anchor point, that is, by adjusting all , minimize the sum of errors of all matching point pairs, so that the optimized three-dimensional coordinates of the virtual anchor point are more consistent with the actual observation.

[0101] S26 shows that the virtual indicator is associated with the corresponding virtual anchor point according to the mapping relationship, so as to display the virtual indicator in the field of view of the AR device.

[0102] By adopting the above solution, this embodiment realizes the function of displaying the virtual indicator in the field of view of the AR device, thereby improving the alignment accuracy between the virtual indicator and the target device.

[0103] Example 3: This example differs from Example 1 in that, when the diagnosis result does not meet expectations, the method further comprises: Logically break down the standard operating procedure (SOP) into multiple independent operating steps (such as equipment inspection, parameter calibration, safety verification, etc.), and each step must meet the requirements of atomicity and verifiability.

[0104] Generate a unique NFT certificate for each operation step on the blockchain through a smart contract, and record the NFT certificate of the i-th operation step as NFT i Voucher, the NFT i The credential includes the step number i, the security level of the i-th operation step, and the time constraint parameter of the i-th operation step.

[0105] Determine whether the standard operating procedure requires multiple inspectors to operate together. If so, deploy the verification logic of the standard operating procedure in the blockchain. If not, no action is taken.

[0106] The verification logic includes: After the inspector responsible for the i-1th operation step completes the i-1th operation step, the NFT is output. i-1 The inspector's mobile terminal (such as AR glasses equipped with blockchain SDK) calls the smart contract to generate NFT after the operation is completed. i Certificates and NFT i The credentials are stored in the consortium chain.

[0107] The inspector responsible for the i-th operation step uses the AR device to scan the preset blockchain NFT to activate the NFT i Get the contract address and token_id of NFTi, call the smart contract to verify the preconditions, and after the verification is passed, the NFT i The status of the credential is updated to activated.

[0108] After the inspector responsible for the i-1th operation step completes the i-1th operation step, the image after the operation is completed is obtained through the AR device, and the operating data after the operation is completed is collected. The image after the operation and the operating data after the operation are input into the trained equipment status diagnosis model to obtain the diagnosis result after the operation is completed. If the diagnosis result after the operation is completed does not meet the expectations, the smart contract triggers the virtual conference room to generate an event, notifying the preset expert list. The expert wears the VR device to enter the virtual conference room and view the status of the target device through the gesture recognition API of the VR controller.

[0109] The diagnostic result after the operation is completed is used as the trigger condition of the smart contract. When the diagnostic result after the operation is completed does not meet expectations, an abnormal event record is generated and stored in the distributed ledger of the alliance chain.

[0110] By adopting the above solution, this embodiment realizes the function of inspecting abnormal distribution network equipment through AR equipment, and realizes the efficiency and intelligence of the inspection process.

[0111] Example 4: This embodiment discloses an intelligent inspection system based on AR equipment, the system comprising: a memory and a processor, The memory stores a computer-readable storage medium; When the processor processes the computer program stored on the computer-readable storage medium, it implements the intelligent inspection method based on AR equipment.

[0112] The above are all preferred embodiments of the present application, and are not intended to limit the scope of protection of the present application. Therefore, any equivalent changes made based on the structure, shape, and principle of the present application should be included in the scope of protection of the present application.

Claims

1. The intelligent inspection method based on AR equipment is characterized by: include: Obtain real-time operation data, historical operation data, real-time images, and historical images of the target device, and add category labels to the historical operation data and historical images respectively; A deep learning algorithm is used to construct an equipment status diagnosis model, which includes a BILSTM sub-model and a CNN sub-model. The equipment status diagnosis model is trained using historical operating data and historical images with added category labels to obtain a trained equipment status diagnosis model. Real-time operation data and real-time images are input into the trained equipment status diagnosis model to output the diagnosis results. When the diagnosis results do not meet expectations, the AR device is used to scan the preset blockchain NFT, and the standard operating procedures stored in the blockchain are retrieved through the smart contract. The inspection personnel then inspect and repair the target equipment according to the standard operating procedures.

2. The intelligent inspection method based on AR equipment according to claim 1 is characterized in that: When the diagnosis result does not meet expectations, the method further includes: Obtain a depth map of the target device, convert the depth map into 3D point cloud data, perform Euclidean clustering on the 3D point cloud data to obtain multiple point cloud segments, and add semantic labels and virtual anchors to each point cloud segmentation block. Based on the diagnosis results, a virtual indicator containing the fault location is generated, a mapping relationship between the fault location and the semantic label is established, and the virtual indicator is associated with the corresponding virtual anchor point according to the mapping relationship, so as to display the virtual indicator in the field of view of the AR device.

3. The intelligent inspection method based on AR equipment according to claim 2 is characterized in that: Before adding semantic labels to each point cloud segment block, the method further includes: The ORB-SLAM3 algorithm is used to extract environmental feature points from the historical image and real-time image at the previous moment through the FAST feature point detector, and an initial weighted co-image map is constructed based on all the environmental feature points. The preset SuperPoint neural network is used to extract key feature points in the real-time image, and non-maximum suppression processing is performed on the key feature points to obtain key feature points whose response values ​​are greater than the preset response value threshold and record them as target feature points; Match the environmental feature points and target feature points in the real-time image. If the match is successful, increase the weight of the corresponding edge in the initial weighted co-graph and output the final weighted co-graph. Based on the final weighted co-visual map, the RANSAC algorithm is used to calculate the initial pose, and the optimal pose is obtained using SVD decomposition. A frustum with a 60° cone angle is constructed with the optical center of the camera of the inspector's current optimal pose as the vertex, and the frustum clipping technology is used to retain the point cloud segments within the frustum space.

4. The intelligent inspection method based on AR equipment according to claim 3 is characterized in that: Before associating the virtual indicator with the corresponding virtual anchor point according to the mapping relationship, the method further includes: Obtain the DBoW3 library of the ORB-SLAM3 algorithm, construct a first vocabulary tree based on environmental feature points and ORB descriptors, construct a second vocabulary tree using key feature points and SuperPoint descriptors, merge the root nodes of the first and second vocabulary trees to obtain a hybrid bag-of-words structure; Based on the hybrid bag-of-words structure, the TF-IDF scores of the environmental feature points and the key feature points in the historical image are calculated. A comprehensive score is calculated based on the TF-IDF scores of the environmental feature points and the TF-IDF scores of the key feature points in the historical image. Historical images with a comprehensive score greater than a preset score threshold are selected as candidate loop images. The similarity between the real-time image and each candidate loop image is calculated separately and recorded as the first data. If there is a first data that is greater than the first preset similarity threshold, the virtual anchor point of the candidate loop image corresponding to the maximum first data is used as the virtual anchor point of the real-time image.

5. The intelligent inspection method based on AR equipment according to claim 4 is characterized in that: If there is no first data with a similarity greater than the first preset similarity threshold, the method further includes: The similarity between the first vocabulary tree and the second vocabulary tree is calculated by a cross-validation algorithm, recorded as the second data. When the second data exceeds a second preset similarity threshold, the target feature point is projected into the ICP point cloud registration framework to generate matching point pairs with topological constraints. A graph optimization model is constructed, and the LM algorithm is used to optimize the 3D coordinates of the virtual anchor points using the reprojection error of the matching point pairs as the edge constraint.

6. The intelligent inspection method based on AR equipment according to any one of claims 1 to 5, characterized in that: When the diagnosis result does not meet expectations, the method further includes: Decompose the standard operating procedure into multiple operating steps, generate a unique NFT certificate for each operating step, and record the NFT certificate of the i-th operating step as NFT i Voucher, the NFT i The credential includes the step number i, the security level of the i-th operation step, and the time constraint parameters of the i-th operation step; Determine whether the standard operating procedure requires multiple inspectors to operate together. If so, deploy the verification logic of the standard operating procedure in the blockchain; if not, do nothing.

7. The intelligent inspection method based on AR equipment according to claim 6, characterized in that: The verification logic includes: After the inspector responsible for the i-1th operation step completes the i-1th operation step, the NFT is output. i-1 Certificate and generate NFT i Credentials, the inspector responsible for the i-th operation step uses the AR device to scan the preset blockchain NFT to activate the NFT i certificate.

8. The intelligent inspection method based on AR equipment according to claim 7, characterized in that: The verification logic also includes: After the inspector responsible for the i-1th operation step completes the i-1th operation step, the AR device is used to obtain images after the operation is completed, and the operating data after the operation is completed is collected. The images and operating data after the operation are input into the trained equipment status diagnosis model to obtain the diagnosis results after the operation is completed; If the diagnosis result after the operation is completed does not meet expectations, a virtual conference room is built. After the expert wears the VR device, he or she controls and views the target device through the gesture recognition API of the VR controller.

9. The intelligent inspection method based on AR equipment according to claim 8, characterized in that: The method further comprises: The diagnostic result after the operation is completed is used as the trigger condition of the smart contract. When the diagnostic result after the operation is completed does not meet the expectations, an abnormal event record is generated and stored in the distributed ledger.

10. The intelligent inspection system based on AR equipment is characterized by: include: memory and processor, The memory stores a computer-readable storage medium; When the processor processes the computer program stored on the computer-readable storage medium, the method according to any one of claims 1 to 9 is implemented.

Images