Smart inspection method and system based on ar device

Abstract

The application provides an AR device-based intelligent inspection method and system, and relates to the technical field of intelligent inspection.The method comprises the following steps: acquiring real-time running data, historical running data, real-time images and historical images of a target device, and adding category labels to the historical running data and images; adopting a deep learning algorithm to construct a device state diagnosis model comprising a BILSTM sub-model and a CNN sub-model, and training the device state diagnosis model by using the labeled historical data and images; inputting the real-time running data and images into the trained model to output a diagnosis result; and if the diagnosis result is not as expected, scanning a blockchain NFT by using an AR device, calling a standard operation process in the blockchain through a smart contract, and providing a maintenance device for an inspector.The application can provide targeted suggestions for the inspector and improve the standardization degree of the fault handling process.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent inspection, in particular to an intelligent inspection method and system based on an AR device. BACKGROUND

[0002] With the rapid development of Industry 4.0 and intelligent manufacturing, the demand for intelligent device inspection is increasing. Traditional manual inspection methods are inefficient and rely on the experience of inspectors, which can easily lead to missed or false inspections due to subjective judgment. In recent years, the combination of augmented reality (AR) technology and artificial intelligence (AI) has provided a new solution for intelligent inspection, significantly improving the accuracy and efficiency of inspection through real-time data collection, visual interaction, and intelligent analysis.

[0003] A Chinese invention patent with publication number CN113222184A provides a device inspection system and method based on augmented reality (AR). The patent collects device identifiers through an AR device and uploads them to an inspection management platform. The platform retrieves virtual content such as device three-dimensional models, inspection guidelines, and maintenance information, and superimposes them in real time into the real scene to provide visual operation guidance for inspectors. It supports remote expert collaborative guidance and automatically generates standardized records and associates historical data after inspection.

[0004] However, the inspection guidance of this patent relies on text, pictures, or animations stored in the database, but does not explicitly mention dynamic association with SOP, and the guidance information may lack contextual relevance. SUMMARY

[0005] To provide targeted recommendations for inspectors and improve the standardization of fault handling processes, the present application provides an intelligent inspection method and system based on an AR device.

[0006] In a first aspect, the present application provides an intelligent inspection method based on an AR device, which adopts the following technical solution:

[0007] The intelligent inspection method based on an AR device includes the following steps:

[0008] Obtain real-time running data, historical running data, real-time images, and historical images of the target device, and add category labels to the historical running data and historical images respectively;

[0009] Use a deep learning algorithm to construct a device state diagnosis model, which includes a BILSTM sub-model and a CNN sub-model. Train the device state diagnosis model using the historical running data and historical images with added category labels to obtain the trained device state diagnosis model.

[0010] The real-time running data and real-time image are input into the trained equipment state diagnosis model, and a diagnosis result is output. When the diagnosis result does not meet the expectation, an AR device scans a preset blockchain NFT, and a standard operation process stored in the blockchain is called through a smart contract. An inspection personnel performs maintenance on the target equipment according to the standard operation process.

[0011] Firstly, the real-time running data, historical running data, real-time image and historical image of the target equipment are acquired, and category labels are added to the historical running data and historical image. Subsequently, a deep learning algorithm is used to construct an equipment state diagnosis model, which includes a BILSTM sub-model and a CNN sub-model. The BILSTM (Bidirectional Long Short-Term Memory) sub-model is good at processing sequence data and can effectively capture the time sequence characteristics and long-term dependencies in the historical running data, which has a unique advantage in analyzing the trend of the running state of the target equipment over time. The CNN (Convolutional Neural Network) sub-model can automatically extract features from the historical image and identify abnormalities or changes in the target equipment. By combining the two, the advantages of different models can be fully utilized to comprehensively and accurately diagnose the real-time state of the target equipment from both running data and image perspectives.

[0012] Subsequently, the historical running data and historical image with added category labels are used to train the equipment state diagnosis model, so that the equipment state diagnosis model learns the equipment state features corresponding to different categories of data during the training process, thereby improving the recognition accuracy of the equipment state diagnosis model for the equipment state. The trained equipment state diagnosis model can better identify various states of the target equipment.

[0013] Subsequently, the real-time running data and real-time image are input into the trained equipment state diagnosis model to obtain a diagnosis result. The real-time diagnosis method used by the present application can timely discover problems or abnormalities in the running process of the target equipment, providing immediate equipment state information for the inspection personnel, which helps to take measures as early as possible to minimize the further expansion of equipment failure, reduce production interruptions and losses caused by target equipment failure, and improve the reliability and production efficiency of the target equipment.

[0014] When the diagnosis result does not meet the expectation, an AR device scans a preset blockchain NFT, and a standard operation process stored in the blockchain is called through a smart contract, providing standardized fault handling guidance for the inspection personnel. The blockchain technology improves the security and tamper resistance of the standard operation process, enabling the inspection personnel to obtain accurate and reliable operation information. The inspection personnel performs maintenance on the target equipment according to the standard operation process, which can improve the standardization and effectiveness of fault handling, reduce the risk of further deterioration of problems caused by improper operation, and also help to improve the work efficiency and fault handling ability of the inspection personnel.

[0015] The application realizes automatic monitoring, diagnosis and fault handling guidance of the equipment state by adopting various advanced technical means, reduces manual intervention, improves the efficiency and accuracy of equipment management, and promotes the development of equipment inspection towards intelligence and automation.

[0016] Optionally, when the diagnosis result does not meet the expectation, the method further comprises:

[0017] Obtaining a depth map of the target equipment, converting the depth map into three-dimensional point cloud data, performing Euclidean clustering segmentation on the three-dimensional point cloud data to obtain a plurality of point cloud segmentation blocks, adding semantic labels and virtual anchor points to each point cloud segmentation block respectively;

[0018] Generating a virtual indicator containing the fault position based on the diagnosis result, establishing a mapping relationship between the fault position and the semantic label, and associating the virtual indicator with the corresponding virtual anchor point according to the mapping relationship, for displaying the virtual indicator in the field of view angle of the AR device.

[0019] The application first obtains a depth map of the target equipment, which contains the distance information from each point on the surface of the target equipment to the AR device. Compared with ordinary two-dimensional images, it can provide three-dimensional spatial structure information of the target equipment, so that the subsequent analysis of the target equipment is no longer limited to the surface appearance features, but can go deep into the three-dimensional form of the target equipment, providing a richer data basis for accurately identifying the fault position of the target equipment. For some target equipment with complex internal structure or three-dimensional appearance, the depth map can help clearly present the relative position and spatial relationship of each part.

[0020] Subsequently, the application converts the depth map into three-dimensional point cloud data, which represents the geometric shape of the surface of the target equipment in the form of discrete points. Each point contains its coordinate information in three-dimensional space, which is convenient for various geometric operations and analysis, so as to more accurately describe the outline and details of the target equipment.

[0021] Subsequently, the application performs Euclidean clustering segmentation on the three-dimensional point cloud data, divides the point cloud data into a plurality of point cloud segmentation blocks according to the spatial distance between points, which is helpful to decompose the target equipment into different components or regions. Through clustering segmentation, the application can distinguish different parts of the target equipment. Even if these parts are similar in appearance, according to their distribution and connection relationship in space, each part can be accurately identified, so as to help to perform individual fault detection and analysis on each part.

[0022] Subsequently, the application adds semantic labels to each point cloud segmentation block, giving the three-dimensional point cloud data semantic information. The semantic labels can correspond the segmented point cloud blocks to the actual components or functional areas of the target device, so that the computer can understand the actual meaning represented by each point cloud segmentation block. Adding virtual anchor points provides reference points for the positioning and display of virtual indicators. Virtual anchor points are fixed positions set in advance in three-dimensional space and are associated with point cloud segmentation blocks. By adding virtual anchor points to point cloud segmentation blocks, the application can accurately place virtual indicators on the corresponding components of the target device, so that the displayed indication information in the field of view of the AR device corresponds to the actual device position and the corresponding components.

[0023] The application generates virtual indicators containing fault locations based on diagnostic results, which can convert abstract diagnostic results into intuitive visual information. Subsequently, the application establishes a mapping relationship between fault locations and semantic labels and associates virtual indicators with corresponding virtual anchor points based on the mapping relationship, achieving accurate correspondence between virtual indicators and actual device fault locations. The application uses semantic labels as an intermediate bridge to link the diagnosed fault locations to point cloud segmentation blocks, and then uses virtual anchor points to accurately position virtual indicators on the target device, so that virtual indicators can be accurately displayed at the actual location of the fault, providing precise fault guidance for inspection personnel and improving the efficiency and accuracy of fault handling.

[0024] Optionally, before adding semantic labels to each point cloud segmentation block respectively, the method further comprises:

[0025] Using the ORB-SLAM3 algorithm, environmental feature points in the historical image and the real-time image of the previous moment are extracted respectively by a FAST feature point detector, and an initial weighted co-view is constructed based on all the environmental feature points;

[0026] A preset SuperPoint neural network is used to extract key feature points in the real-time image, and non-maximum suppression processing is performed on the key feature points to obtain key feature points with a response value greater than a preset response value threshold, which are denoted as target feature points;

[0027] The environmental feature points and the target feature points in the real-time image are matched, if the matching is successful, the weight of the corresponding edge in the initial weighted co-view is increased, and a final weighted co-view is output;

[0028] Based on the final weighted co-view, an initial pose is calculated using the RANSAC algorithm, and an optimal pose is calculated using SVD decomposition. A view cone space with a 60° cone angle is constructed with the camera optical center of the current optimal pose of the inspection personnel as the vertex, and a view cone clipping technique is used to retain the point cloud segmentation blocks within the view cone space.

[0029] The improved ORB-SLAM3 algorithm is adopted to extract the environmental feature points in the historical image and the real-time image at the previous moment respectively through the FAST feature point detector, and the FAST feature point detector can quickly capture the environmental feature points such as corners in the image, which helps to accurately describe the change of the environment around the target device at different times.

[0030] Subsequently, the initial weighted co-view is constructed based on all the environmental feature points, and the correlation between the environmental feature points in different image frames is obtained, and the initial weighted co-view further considers the co-view degree between the environmental feature points in different images, and the greater the weight, the higher the probability of the two environmental feature points appearing in multiple image frames, that is, the stronger their correlation, so as to more accurately describe the spatial and temporal relationship between the environmental feature points.

[0031] Subsequently, the preset SuperPoint neural network is used to extract the key feature points in the real-time image, compared with the traditional feature point detection method, the SuperPoint neural network can extract more representative and stable key feature points, which helps to improve the accuracy of feature matching. Then, the non-maximum suppression processing is performed on the key feature points, and the key feature points with a response value greater than a preset response value threshold are obtained, which are denoted as target feature points, and the environmental feature points and the target feature points in the real-time image are matched. If the matching is successful, the weight of the corresponding edge in the initial weighted co-view is increased, and the final weighted co-view is output. The initial weighted co-view is constructed based on the environmental feature points, and the weight reflects the matching number of the environmental feature points. When the target feature points (i.e. features with higher distinguishability) extracted by the SuperPoint neural network are successfully matched with the environmental feature points in the real-time image, the weight of the corresponding edge is increased to obtain the final weighted co-view. The semantic perception of the SuperPoint neural network compensates for the limitations of traditional features, and the consistency of the feature points in the same frame can improve the reliability of cross-frame matching.

[0032] Subsequently, the initial pose is calculated based on the final weighted co-view using the RANSAC algorithm. In the pose calculation, the RANSAC algorithm can filter out the inliers (i.e. points that meet the pose model) from the corresponding relationship of the feature points in the final weighted co-view, so as to calculate a more accurate initial pose. Then, the optimal pose is obtained by using the SVD decomposition algorithm. The current optimal pose of the inspection personnel is obtained by the above-mentioned manner, and the camera optical center under the pose is taken as the vertex to construct a 60° cone angle view cone space. The view cone clipping technology is used to retain the point cloud segmentation block in the view cone space and remove the point cloud segmentation block outside the view cone space. The above-mentioned scheme reduces the amount of data to be processed, improves the efficiency and pertinence of data processing.

[0033] Optionally, before associating the virtual indicator with the corresponding virtual anchor point according to the mapping relationship, the method further comprises:

[0034] The DBoW3 library of the ORB-SLAM3 algorithm is acquired, a first vocabulary tree is constructed based on the environmental feature points and the ORB descriptor, a second vocabulary tree is constructed using the key feature points and the SuperPoint descriptor, and the root nodes of the first vocabulary tree and the second vocabulary tree are merged to obtain a hybrid bag-of-words structure;

[0035] Based on the hybrid bag-of-words structure, the TF-IDF scores of the environmental feature points and the TF-IDF scores of the key feature points in the historical images are calculated, the comprehensive scores are calculated based on the TF-IDF scores of the environmental feature points and the TF-IDF scores of the key feature points in the historical images, and the historical images with the comprehensive scores greater than a preset score threshold are taken as candidate loopback images.

[0036] The similarity of the real-time image and each candidate loopback image is calculated respectively, denoted as first data, and if there is first data greater than a first preset similarity threshold, the virtual anchor point of the candidate loopback image corresponding to the maximum first data is taken as the virtual anchor point of the real-time image.

[0037] The DBoW3 library of the ORB-SLAM3 algorithm is acquired, a first vocabulary tree is constructed based on the environmental feature points and the ORB descriptor, a second vocabulary tree is constructed using the key feature points and the SuperPoint descriptor, and the root nodes of the first vocabulary tree and the second vocabulary tree are merged to obtain a hybrid bag-of-words structure, since different descriptors describe features in different ways, the hybrid bag-of-words structure fuses them together, increasing the dimension and complexity of the features, and further enabling the application to more comprehensively and accurately represent feature information in images, better distinguish different image regions and objects, and reduce the possibility of feature confusion.

[0038] Subsequently, the TF-IDF scores of the environmental feature points and the TF-IDF scores of the key feature points in the historical images are calculated based on the hybrid bag-of-words structure, and the comprehensive scores are further calculated, and then the historical images with the comprehensive scores greater than a preset score threshold are taken as candidate loopback images, thereby filtering out historical images that are not similar or have low similarity to the current scene. During the inspection process, there may be a large number of historical images about the target device, only a small number of which are related to the current scene. By pre-setting the score threshold, the application can quickly screen out candidate loopback images (i.e., images similar to the current scene and historical scenes).

[0039] Subsequently, the application calculates the similarity of the real-time image and each candidate loopback image respectively, denoted as first data, if there is a case where the first data is greater than a first preset similarity threshold, it indicates that the real-time image and a certain candidate loopback image constitute a loopback, the virtual anchor point of the candidate loopback image corresponding to the maximum first data is taken as the virtual anchor point of the real-time image. By calculating the similarity, the application can accurately find the historical loopback image that best matches the real-time image. Since the position of the virtual anchor point in the historical loopback image has been determined, taking the corresponding virtual anchor point as the virtual anchor point of the real-time image can realize accurate transmission and positioning of the virtual anchor point between different images, thereby better correcting the cumulative error and improving the accuracy of positioning.

[0040] Optionally, if there is no first data greater than the first preset similarity threshold, the method further comprises:

[0041] The similarity of the first vocabulary tree and the second vocabulary tree is calculated by a cross-validation algorithm, denoted as second data, when the second data exceeds a second preset similarity threshold, the target feature points are projected into an ICP point cloud registration framework to generate matching point pairs with topological constraints;

[0042] An optimization model of a graph is constructed, taking the re-projection error of the matching point pairs as an edge constraint, and using the LM algorithm to optimize the three-dimensional coordinates of the virtual anchor point.

[0043] The application calculates the similarity of the first vocabulary tree and the second vocabulary tree by a cross-validation algorithm, and the similarity of the two is denoted as second data. When the first data fails to reach the similarity threshold, it indicates that the conventional matching method based on image similarity may not be able to accurately find the loopback image. At this time, by cross-validating the similarity of the two vocabulary trees, the application can excavate potential matching relationships.

[0044] When the second data exceeds the second preset similarity threshold, the application projects the target feature points into an ICP point cloud registration framework to generate matching point pairs with topological constraints. The ICP algorithm finds the nearest point pair between two sets of point clouds and continuously iteratively optimizes the registration parameters to realize accurate alignment of the point clouds. By projecting the target feature points into the ICP framework, more constrained matching point pairs can be generated in combination with the semantic information and topological structure of the feature points. These matching point pairs not only consider the spatial distance between points, but also consider the positional relationship of the feature points in the overall structure, thereby improving the accuracy and robustness of matching.

[0045] Subsequently, the application constructs a graph optimization model to match the re-projection error of the point pairs as a side constraint, and uses the LM (Levenberg-Marquardt) algorithm to optimize the three-dimensional coordinates of the virtual anchor points. In the graph optimization model, the three-dimensional coordinates of the virtual anchor points are taken as optimization variables, and the position of the virtual anchor points is optimized by minimizing the re-projection error. The LM algorithm can effectively reduce the re-projection error, thereby improving the positioning accuracy of the three-dimensional coordinates of the virtual anchor points.

[0046] Optionally, when the diagnosis result does not meet the expectation, the method further comprises:

[0047] The standard operation process is decomposed into a plurality of operation steps, a unique NFT certificate is generated for each operation step, and the NFT certificate of the i-th operation step is denoted as NFT i certificate, the NFT i certificate includes a step number i, a security level of the i-th operation step, and a time constraint parameter of the i-th operation step.

[0048] It is judged whether the standard operation process needs to be operated by multiple inspection personnel, and if so, the verification logic of the standard operation process is deployed in the blockchain; if not, no processing is performed.

[0049] The standard operation process is decomposed into a plurality of operation steps, and a unique NFT certificate is generated for each operation step. Then, it is judged whether the standard operation process needs to be operated by multiple inspection personnel. When the standard operation process needs to be operated by multiple inspection personnel, the verification logic of the standard operation process is deployed in the blockchain. The verification logic can monitor and verify the operation steps of each inspection personnel in real time. Only when all inspection personnel complete the operation according to the specified steps and time requirements, the entire operation process is completed, thereby further improving the standardization degree of the operation process.

[0050] The verification logic of the standard operation process is deployed in the blockchain, which can utilize the tamper-proof and decentralized characteristics of the blockchain to improve the fairness and transparency of the verification process. The distributed storage and consensus mechanism of the blockchain makes the verification logic not controlled by a single node or individual, effectively preventing human interference and tampering. In traditional operation process management, there may be cases of human modification of operation records or bypassing of verification links. However, the application of blockchain technology can prevent such phenomena from occurring, thereby improving the seriousness and standardization of the operation process.

[0051] Optionally, the verification logic comprises:

[0052] After the inspection personnel responsible for the i-1-th operation step completes the i-1-th operation step, the NFT i-1 certificate is outputted, and the NFT iThe credential, the inspection personnel responsible for the i-th operation step scans the preset blockchain NFT using the AR device to activate the NFT i The credential.

[0053] The present application will output the NFT after the inspection personnel responsible for the i-1-th operation step completes the step i - 1 The credential, and generates the NFT i The credential, thereby clearly defining the sequence of operation steps. Only when the previous step is successfully completed will the credential of the subsequent step be generated, thereby fundamentally ensuring that the operation process proceeds in the predetermined order and minimizing the risk of safety accidents or diagnostic errors due to disordered operation sequences.

[0054] Since the generation of the credential for each step depends on the completion of the previous step, if the inspection personnel attempts to skip a certain step and proceed with the subsequent operation, the corresponding NFT credential cannot be obtained, thereby preventing the activation of the subsequent operation step. The above-mentioned scheme can effectively prevent the omission or skipping of operations during the operation process, ensuring that each step is executed and improving the quality and reliability of the operation.

[0055] Optionally, the verification logic further comprises:

[0056] After the inspection personnel responsible for the i-1-th operation step completes the i-1-th operation step, the AR device is used to obtain an image after the operation is completed, and operation data after the operation is completed is collected. The image after the operation is completed and the operation data after the operation is completed are input into the trained device state diagnosis model to obtain a diagnosis result after the operation is completed.

[0057] If the diagnosis result after the operation is completed does not meet the expectations, a virtual conference room is constructed, and the expert wears a VR device to control the target device through gesture recognition API of the VR controller.

[0058] After the inspection personnel responsible for the i-1-th operation step completes the operation, the present application obtains an image after the operation is completed through an AR device, and collects operation data after the operation is completed. The obtained image and operation data are input into the trained device state diagnosis model, and the device state diagnosis model outputs a diagnosis result after the operation is completed. The trained device state diagnosis model is constructed based on a large amount of historical data and professional knowledge, and can perform deep analysis and comparison on the input information, thereby more accurately judging whether the state of the target device is normal, and realizing real-time monitoring of the operation process. When the diagnosis result after the operation is completed does not meet the expectations, the present application constructs a virtual conference room, and the expert wears a VR device to control the target device through gesture recognition API of the VR controller. This remote collaboration method breaks the time and space constraints, and the expert can understand the actual state of the device in real time without being present on site, thereby providing guidance and suggestions for problem solving.

[0059] Optionally, the method further comprises:

[0060] The diagnosis result after the operation is completed is taken as a smart contract trigger condition, and when the diagnosis result after the operation is completed does not meet the expectation, an abnormal event record is generated and stored in the distributed ledger.

[0061] The diagnosis result after the operation is taken as a smart contract trigger condition, and when the diagnosis result after the operation does not meet the expectation, the smart contract is automatically triggered, and the subsequent processing process can be started without manual intervention, improving the timeliness and efficiency of problem processing and minimizing the problem expansion caused by manual operation delay.

[0062] In a second aspect, the present application provides an intelligent inspection system based on an AR device, which adopts the following technical solution:

[0063] The intelligent inspection system based on the AR device comprises a memory and a processor,

[0064] The memory stores a computer readable storage medium;

[0065] When the processor processes the computer program stored on the computer readable storage medium, the method as described in the first aspect is implemented.

[0066] In summary, the present application includes at least one of the following beneficial technical effects:

[0067] 1. The present application first adopts a device state diagnosis model to diagnose real-time data and obtains a diagnosis result, when the diagnosis result does not meet the expectation, the AR device scans the preset blockchain NFT, and the standard operation process stored in the blockchain is called through the smart contract, providing the inspection personnel with a standardized fault processing guide, the blockchain technology improves the security and non-tamperability of the standard operation process, so that the inspection personnel can obtain accurate and reliable operation information. The inspection personnel can improve the standardization and effectiveness of fault processing by repairing the target device according to the standard operation process, reduce the risk of further deterioration of the problem caused by improper operation, and also help to improve the work efficiency and fault processing ability of the inspection personnel.

[0068] 2.The application generates a virtual indicator containing a fault location based on the diagnosis result, which can convert abstract diagnosis results into intuitive visual information. Subsequently, the application establishes a mapping relationship between the fault location and the semantic label, and associates the virtual indicator with the corresponding virtual anchor point according to the mapping relationship, realizing the accurate correspondence between the virtual indicator and the actual device fault location. The application uses semantic labels as an intermediate bridge to link the diagnosed fault location with point cloud segmentation blocks, and then uses virtual anchors to accurately position the virtual indicator on the target device, so that the virtual indicator can be accurately displayed at the actual location of the fault, providing precise fault guidance for inspection personnel and improving the efficiency and accuracy of fault handling.

[0069] 3.The application uses multiple technical means to realize automatic monitoring, diagnosis and fault handling guidance of the device state, reduces manual intervention, improves the efficiency and accuracy of device management, and promotes the development of device inspection towards intelligence and automation. BRIEF DESCRIPTION OF DRAWINGS

[0070] Figure 1 is a flowchart of embodiment 1 of the application;

[0071] Figure 2 is a flowchart of embodiment 2 of the application. DETAILED DESCRIPTION

[0072] The application will be further described below in conjunction with Figure 1 and Figure 2 .

[0073] Embodiment 1: This embodiment discloses an intelligent inspection method based on an AR device, referring to Figure 1 , the method comprises: S11 data acquisition, S12 modeling and training, and S13 diagnosis. First, the real-time running data and the historical running data, the real-time image and the historical image of the target device are obtained, and the historical running data and the historical image are labeled with category labels. Then, a device state diagnosis model containing a BILSTM sub-model and a CNN sub-model is constructed by using a deep learning algorithm, and the device state diagnosis model is trained to obtain a trained device state diagnosis model. Then, the real-time running data and the real-time image are input into the trained device state diagnosis model to output a diagnosis result. If the diagnosis result is not as expected, the AR device scans the blockchain NFT, and the standard operation process in the IPFS distributed storage is called through the smart contract, for the inspection personnel to overhaul the device. The execution process of each step of this embodiment is as follows:

[0074] S11 data acquisition, the real-time running data of the target device is obtained through the pre-deployed sensor, the real-time image of the target device is obtained through the RGB-D camera set in the AR device, and the historical running data and the historical image stored in the blockchain are called based on the preset blockchain NFT.

[0075] The operation data refers to at least one of temperature, partial discharge, mechanical vibration and gas composition.

[0076] The temperature data includes transformer winding temperature, circuit breaker contact temperature, cable joint temperature and capacitor shell temperature.

[0077] The monitoring position of the partial discharge includes cable terminal, GIS (gas insulated switchgear), transformer interior, etc.

[0078] The monitoring position of the mechanical vibration includes transformer core, reactor, pole-mounted switch operating mechanism, etc. The mechanical vibration data includes vibration frequency (such as 100Hz multiple frequency caused by core looseness) and vibration amplitude (RMS value > 5mm / s² may indicate mechanical failure).

[0079] The monitoring position of the gas composition includes SF6 circuit breaker and oil-immersed transformer. The gas composition includes SF6 decomposition products (SO2, H2S, reflecting arc or overheating fault), dissolved gas in transformer oil (H2, CH4, C2H2, diagnosing fault type according to IEC 60599 three-ratio method).

[0080] Class labels are added to the historical operation data and historical images respectively, including normal, overload, undervoltage, crack, corrosion, oil leakage, etc.

[0081] S12 modeling and training, using a deep learning algorithm to construct a device state diagnosis model, including a BILSTM sub-model and a CNN sub-model.

[0082] The BILSTM sub-model includes an input layer, a bidirectional LSTM layer (128 neural units) and a fully connected layer (64 neural units).

[0083] The CNN sub-model includes an input layer, a ResNet-18 backbone network and a global average pooling layer.

[0084] The output of the BILSTM sub-model and the output of the CNN sub-model are respectively connected with an attention layer. After processing by the attention layer, the outputs of the two are output by an output layer to output a diagnosis result.

[0085] The historical operation data and historical images with added class labels are used to train the device state diagnosis model to obtain a trained device state diagnosis model, the process being as follows:

[0086] The historical running data is sliced according to a preset time window, and the sliced historical running data is input into the BILSTM submodel according to the collection time sequence. A bidirectional LSTM layer (128 units) extracts the forward and backward time sequence features of the sliced historical running data, and a full connection layer (64 units) reduces the dimension of the time sequence features to obtain first output data.

[0087] The historical image is adjusted to a resolution of 224x224, the spatial features of the historical image are extracted by using a ResNet-18 backbone network, a feature map is output, and the feature map is compressed by using a global average pooling layer to obtain second output data.

[0088] The first output data and the second output data are input into an attention layer to obtain weighted target output data, and the target output data is input into an output layer to obtain a classification result.

[0089] Subsequently, a basic training is performed by using an AdamW optimizer (with an initial learning rate of 0.001) in combination with a multi-task loss function (such as FocalLoss and MSE for regression), and a learning rate is dynamically adjusted by using a Cosine Annealing. Subsequently, a Dropout (with a rate of 0.2-0.5), a Label Smoothing and an Early Stopping are introduced to prevent overfitting. The outputs of the CNN submodel and the BILSTM submodel are dynamically fused by using an attention mechanism at a feature layer until a preset iteration number or a loss function converges, so as to realize training of the equipment state diagnosis model and obtain a trained equipment state diagnosis model.

[0090] S13 diagnosis, inputting the real-time running data and the real-time image into the trained equipment state diagnosis model to output a diagnosis result.

[0091] In this embodiment, a standard operation procedure (SOP) document, a 3D model and a maintenance video are packaged into an NFT and stored in a Hyperledger Fabric blockchain. When the diagnosis result is abnormal, a smart contract is automatically called to verify the inspection personnel's authority and return the URI of the corresponding NFT. When the diagnosis result is normal, no processing is performed.

[0092] The inspection personnel scans the NFT label on the target device by wearing an AR device, triggers a smart contract, and the smart contract verifies the device ID and operation authority, and then calls the encrypted standard operation procedure file from the IPFS distributed storage.

[0093] The standard operation procedure (SOP) is superimposed on the target device in the form of a virtual indicator, specifically including: automatic highlighting of key parts, real-time floating prompt of digital indicators such as torque parameters, and synchronization of maintenance process data to the chain for storage.

[0094] The embodiment can provide a standard operation process for the current diagnostic result for the inspection personnel, and improve the standardization degree of the fault processing process.

[0095] Embodiment 2: Refer to Figure 2 The embodiment is different from the embodiment 1 in that when the diagnostic result is not as expected, the method further comprises:

[0096] S21 obtains a point cloud segmentation block, and obtains a depth map of the target device by using an RGB-D camera or a laser radar arranged on an AR device.

[0097] According to the pixel coordinates (u, v) in the depth map and the corresponding depth value d, the calculation model of the three-dimensional coordinates (X, Y, Z) of the three-dimensional point cloud data is as follows:

[0098] ;

[0099] ;

[0100] ;

[0101] wherein, is the camera optical center, and the unit is pixel; is the focal length, and the unit is pixel.

[0102] The obtained three-dimensional point cloud data is subjected to Euclidean clustering segmentation, and the DBSCAN variant algorithm of the Euclidean distance between the three-dimensional point cloud data is used, specifically: according to the three-dimensional coordinates (X, Y, Z) of the three-dimensional point cloud data, the points with a distance less than a distance threshold between two points are classified into the same cluster, a plurality of clusters are obtained, and the three-dimensional point cloud data classified into a cluster is recorded as a point cloud segmentation block.

[0103] wherein, the distance threshold is set according to the size of the target device, for example, the distance between the power distribution cabinets is greater than 0.5 meters, and the distance threshold is set to 0.4 meters.

[0104] S22 screens the point cloud segmentation block, adopts the ORB-SLAM3 algorithm, detects the detection corners in the historical image and the real-time image of the last moment by using the FAST feature point detector, records them as environmental feature points, constructs an initial weighted co-view based on all the environmental feature points, the nodes of the initial weighted co-view are the historical image and the real-time image of the last moment, and the edge weight is the number of matched environmental feature points between the historical image and the real-time image of the last moment.

[0105] From the above, the initial weighted consensus view only contains the connection of the historical image and the real-time image at the last moment, and the edge weight is counted by the Brute-Force Matcher algorithm.

[0106] The real-time image is input into the preset SuperPoint neural network, and the preset SuperPoint neural network outputs a key feature point heat map and a SuperPoint descriptor. The key feature points are obtained by non-maximum suppression (NMS) of the heat map. The key feature points with a response value greater than a preset response value threshold are recorded as target feature points.

[0107] In the embodiment, the preset response value threshold The calculation model is as follows:

[0108] ;

[0109] wherein, is the standard deviation of the gray value of the real-time image; I is the gray value in the gray image of the real-time image, is the maximum value in the gray value.

[0110] In other embodiments, the value of the preset response value threshold can be set according to requirements, such as 0.01.

[0111] The preset SuperPoint neural network includes a shared encoder and two independent decoders:

[0112] The shared encoder performs spatial down-sampling on the input image through a convolution layer, pooling and a nonlinear activation function, and reduces the image dimension. The shared encoder is composed of multiple convolution layers and maximum pooling layers. After three maximum pooling operations, the image size is reduced from HxW to H / 8xW / 8, H is the height of the image, and W is the width of the image.

[0113] After the shared encoder, the SuperPoint neural network is divided into two independent decoder branches, which are respectively used for target feature point detection and SuperPoint descriptor generation. The two independent decoder branches share the output of the encoder, but learn respective task-specific weights.

[0114] The feature point detector branch is responsible for generating a dense target feature point probability map, and the process is as follows:

[0115] The feature map shared by the encoder output is subjected to a series of convolution operations to obtain the probability of each pixel being a target feature point, and the feature map is subjected to Softmax processing to convert the output of each pixel into a probability distribution, wherein 65 channels correspond to a local, non-overlapping 8x8 pixel grid area, and an additional non-interest point dimension is added. After Softmax processing, the non-interest point dimension is removed, redundant key feature points are removed through NMS operation, and the final key feature point position and confidence are obtained. The probability that each pixel is a target feature point is output, and a dense probability map of target feature points is generated.

[0116] The descriptor generator branch is responsible for extracting features around each detected target feature point and generating a fixed-length descriptor vector, the process being as follows:

[0117] In order to reduce the training memory and calculation amount, the descriptor generator first learns to obtain a semi-dense descriptor, performs bilinear interpolation on the semi-dense descriptor to obtain a dense descriptor corresponding to each target feature point, performs L2 normalization processing on the dense descriptor to ensure that its norm is 1, and obtains the final descriptor result, outputting the descriptor vector of each target feature point.

[0118] The embodiment adopts the FLANN-based KD tree descriptor search method to match the environmental feature points and target feature points in the real-time image. If the matching is successful, the weight of the corresponding edge in the initial weighted consensus view is increased, and the final weighted consensus view is output.

[0119] In the embodiment, when the environmental feature points and target feature points in the real-time image are successfully matched, the weight of the corresponding edge in the initial weighted consensus view is updated to 1.

[0120] Three groups of matching point pairs are selected from the final weighted consensus view, and the camera pose, i.e. the rotation matrix and the translation vector, is solved by the RANSAC algorithm. The pose with the largest number of inliers is retained as the initial solution, i.e. the initial pose. A re-projection error function is constructed based on all inliers, and the optimal pose is solved by the Levenberg-Marquardt algorithm.

[0121] A cone space with a 60° cone angle is constructed with the camera optical center of the current optimal pose of the inspection personnel as the vertex and the principal axis as the axis. For each point cloud segmentation block, the included angle between the connecting line of the center point of the point cloud segmentation block to the camera optical center and the principal axis is calculated, and the point cloud segmentation block with an included angle less than half the cone angle 30° is retained.

[0122] S23 adds, respectively adds semantic labels such as incoming line switch, bus, outgoing line switch, etc. to the remaining point cloud segmentation blocks in the point cloud segmentation blocks screened by S22 in a manner of machine learning classification or manual annotation.

[0123] After that, a virtual anchor point is added to each point cloud segmentation block, and the position of the virtual anchor point can be the centroid coordinate in the point cloud segmentation block.

[0124] In other embodiments, the local feature descriptor can also be used to detect the salient points in the segmentation block as the position of the virtual anchor point.

[0125] S24 establishes a mapping relationship, generates a virtual indicator containing a fault site based on the diagnostic result, and establishes a mapping relationship between the fault site and the semantic label.

[0126] S25 determines or optimizes the virtual anchor point, obtains the DBoW3 library of the ORB-SLAM3 algorithm, adopts the K-means clustering method of the DBoW3 library, constructs a hierarchical vocabulary tree for the ORB descriptor of the training image set, and is recorded as a first vocabulary tree.

[0127] In the embodiment, the number of layers of the first vocabulary tree is 6-10, the number of branches K=10 at each layer, and each node of the first vocabulary tree is used to store the ORB descriptor clustering center. The leaf node is used to store the corresponding visual word.

[0128] Similar to the first vocabulary tree, the number of layers of the second vocabulary tree is 6-10, the number of branches K=10 at each layer, a SuperPoint descriptor is randomly selected as the first clustering center, the distances of the remaining SuperPoint descriptors to the first clustering center are calculated, and the probability P of being selected as a clustering center is calculated. The SuperPoint descriptor corresponding to the maximum probability P is taken as another clustering center, and the process is repeated until 10 clustering centers are selected. The calculation model of the probability P is as follows:

[0129] ;

[0130] wherein, is the distance of the ith SuperPoint descriptor to the nearest clustering center.

[0131] The remaining SuperPoint descriptors are assigned to the nearest clustering center, the mean of each cluster is recalculated as a new clustering center, and the process is repeated until all SuperPoint descriptors are traversed. Finally, the second vocabulary tree is output.

[0132] In other embodiments, in order to improve the calculation efficiency, the SuperPoint descriptor can also be processed by dimension reduction, and the second vocabulary tree is constructed based on the dimension-reduced SuperPoint descriptor.

[0133] The root nodes of the first vocabulary tree and the second vocabulary tree are merged to obtain a hybrid bag-of-words structure.

[0134] Based on the mixed bag-of-words structure, the TF-IDF scores of the environmental feature points and the TF-IDF scores of the key feature points in the historical image are calculated respectively, and the weighted average algorithm is used to calculate the comprehensive score based on the TF-IDF scores of the environmental feature points and the TF-IDF scores of the key feature points in the historical image, and the historical image with a comprehensive score greater than a preset score threshold is taken as a candidate loopback image.

[0135] The similarity of the real-time image and each candidate loopback image is calculated respectively, denoted as first data, and if there is first data greater than a first preset similarity threshold, the virtual anchor point of the candidate loopback image corresponding to the maximum first data is taken as the virtual anchor point of the real-time image.

[0136] If there is no first data greater than the first preset similarity threshold, it means that the real-time image and the candidate loopback image are quite different, and may belong to a new scene. The similarity of the first vocabulary tree and the second vocabulary tree is calculated by a cross-validation algorithm, denoted as second data, and the process is as follows:

[0137] Each leaf node of the real-time image vocabulary tree (the first vocabulary tree) is traversed, and the same node path is searched in the candidate image vocabulary tree (the second vocabulary tree).

[0138] For the nodes on the coincident path, a weight is assigned according to the depth b of the node , and the calculation model of the weight is as follows:

[0139] ;

[0140] Wherein, B is the depth of the first vocabulary tree; is an adjustment parameter, and the value range is 0.5-1.

[0141] The TF-IDF values of all matching point pairs (Lj, Lj') on the coincident path are extracted respectively, denoted as TF-IDF(Lj) and TF-IDF(Lj'), and the similarity contribution of the matching point pair is calculated, and the calculation model is as follows:

[0142] ;

[0143] is the similarity contribution of the matching point pair; min(·) is a minimum value operation, and the purpose is to reduce the interference of high-frequency noise.

[0144] The similarity contributions of all coincident paths are summed up to obtain the similarity of the first vocabulary tree and the second vocabulary, that is, the calculation model of the second data is as follows:

[0145] ;

[0146] G is the second data; N is the total number of leaf nodes of the first vocabulary tree.

[0147] When the second data exceeds the second preset similarity threshold, it indicates that the first vocabulary tree and the second vocabulary tree have sufficient relevance in the topological structure, at this time the target feature point is projected into the ICP point cloud registration framework to generate a matching point pair with topological constraints, and the process is as follows:

[0148] The target feature point is projected into the source point cloud coordinate system of ICP in turn to generate an initial matching point pair, in the nearest neighbor matching algorithm of the ICP point cloud registration framework, the consistency constraint of the normal vector direction is increased, the initial matching point pair with inconsistent normal vectors is proposed, and the matching point pair with topological constraints is obtained.

[0149] A graph optimization model is constructed, the nodes of the graph optimization model are three-dimensional coordinates of virtual anchor points, and the reprojection error of the matching point pair is used as an edge constraint, and the calculation model of the reprojection error is as follows:

[0150] ;

[0151] Wherein, e is the reprojection error; is a camera projection function for mapping three-dimensional coordinates to a two-dimensional pixel plane; T1 is a camera pose, including a rotation matrix and a translation matrix, that is, a pose transformation matrix of the camera from the world coordinate system to the camera coordinate system; is the three-dimensional coordinate of the virtual anchor point in the kth matching point pair, that is, the variable to be optimized; is the two-dimensional coordinate of the target feature point actually detected in the actual image in the kth matching point pair.

[0152] The LM algorithm is used to optimize the three-dimensional coordinates of the virtual anchor point, that is, by adjusting all , the total error sum of all matching point pairs is minimized, so that the optimized three-dimensional coordinates of the virtual anchor point are more consistent with the actual observation.

[0153] S26 shows that the virtual indicator is associated with the corresponding virtual anchor point according to the mapping relationship, and is used to display the virtual indicator in the field of view angle of the AR device.

[0154] By using the above scheme, the embodiment realizes the function of displaying the virtual indicator in the field of view angle of the AR device, and improves the alignment accuracy of the virtual indicator and the target device.

[0155] Embodiment 3: The difference between this embodiment and embodiment 1 is that when the diagnostic result does not meet the expectation, the method further comprises:

[0156] The standard operation procedure (SOP) is logically disassembled into multiple independent operation steps (such as equipment inspection, parameter calibration, safety verification, etc.), and each step needs to meet atomicity and verifiability.

[0157] A unique NFT credential is generated for each operation step on the blockchain through the smart contract, and the NFT credential of the i-th operation step is denoted as NFT i credential, the NFT i credential includes the step number i, the security level of the i-th operation step, and the time constraint parameter of the i-th operation step.

[0158] If the standard operation flow requires multiple inspection personnel to operate together, the verification logic of the standard operation flow is deployed in the blockchain. If not, no processing is performed.

[0159] The verification logic includes:

[0160] After the inspection personnel responsible for the i-1-th operation step completes the i-1-th operation step, the NFT i-1 credential is output, and the NFT i credential is generated by the mobile terminal (such as AR glasses equipped with a blockchain SDK) of the inspection personnel after the operation is completed, and the NFT i credential is stored in the alliance chain.

[0161] The inspection personnel responsible for the i-th operation step scans the preset blockchain NFT using an AR device to activate the NFT i credential, obtains the contract address and token_id of NFTi, calls the smart contract to verify the precondition, and after the verification is passed, updates the state of the NFT i credential to activated.

[0162] After the inspection personnel responsible for the i-1-th operation step completes the i-1-th operation step, the image after the operation is completed is obtained through the AR device, and the running data after the operation is completed is collected, the image after the operation is completed and the running data after the operation are input into the trained device state diagnosis model, and the diagnosis result after the operation is completed is obtained. If the diagnosis result after the operation is completed does not meet the expectation, the smart contract triggers a virtual conference room to generate an event, notifies a preset expert list, and the expert wears a VR device to enter the virtual conference room and views the target device through the gesture recognition API of the VR controller.

[0163] The diagnosis result after the operation is completed is used as a smart contract triggering condition, and when the diagnosis result after the operation is completed does not meet the expectation, an abnormal event record is generated and stored in the distributed ledger of the alliance chain.

[0164] By using the above scheme, the embodiment realizes the function of inspecting the abnormal power distribution network equipment through the AR device, and realizes the efficiency and intelligence of the inspection process.

[0165] Embodiment 4: This embodiment discloses an AR device-based intelligent inspection system, the system comprising: a memory and a processor,

[0166] The memory stores a computer readable storage medium;

[0167] The processor processes the computer program stored on the computer readable storage medium to implement the AR device-based intelligent inspection method.

[0168] The above are preferred embodiments of the present application, not to limit the protection scope of the present application, therefore: all equivalent changes made according to the structure, shape, principle of the present application should be covered within the protection scope of the present application.

Claims

1. An intelligent inspection method based on AR devices, characterized in that, include: Acquire real-time operating data, historical operating data, real-time images, and historical images of the target device, and add category labels to the historical operating data and historical images respectively; A deep learning algorithm is used to construct an equipment status diagnosis model, which includes a BILSTM sub-model and a CNN sub-model. The equipment status diagnosis model is trained using historical operating data and historical images with added category labels to obtain the trained equipment status diagnosis model. Real-time operating data and real-time images are input into the trained equipment status diagnosis model, and the diagnosis results are output. When the diagnosis results do not meet expectations, AR devices are used to scan the preset blockchain NFTs, and standard operating procedures stored in the blockchain are retrieved through smart contracts. Inspection personnel then perform maintenance on the target equipment according to the standard operating procedures. When the diagnostic results do not meet expectations, the method further includes: Acquire the depth map of the target device, convert the depth map into 3D point cloud data, perform Euclidean clustering on the 3D point cloud data to obtain multiple point cloud segmentation blocks, and add semantic labels and virtual anchor points to each point cloud segmentation block. Based on the diagnostic results, a virtual indicator containing the fault location is generated. A mapping relationship between the fault location and semantic tags is established. The virtual indicator is associated with the corresponding virtual anchor point according to the mapping relationship. The virtual indicator is used to display the virtual indicator in the field of view of the AR device. The standard operating procedure is to overlay the virtual indicator on the target device. The standard operating procedure is broken down into multiple steps, and a unique NFT certificate is generated for each step. The NFT certificate of the i-th step is denoted as NFT. i Certificate, the NFT i The certificate includes step number i, the security level of the i-th operation step, and the time constraint parameter of the i-th operation step; Determine whether the standard operating procedure requires multiple inspectors to operate together. If so, deploy the verification logic of the standard operating procedure in the blockchain; otherwise, do not take any action. The verification logic includes: after the inspector responsible for the (i-1)th operation step completes the (i-1)th operation step, outputting the NFT. i-1 Credentials and generate NFTs i The credential, the inspection personnel responsible for the i-th operation step, use AR equipment to scan the preset blockchain NFT to activate the NFT. i certificate.

2. The intelligent inspection method based on AR devices according to claim 1, characterized in that, Before adding semantic labels to each point cloud segment, the method further includes: The ORB-SLAM3 algorithm is used to extract environmental feature points from the historical image and the real-time image of the previous time step using the FAST feature point detector, and an initial weighted co-view is constructed based on all environmental feature points. A preset SuperPoint neural network is used to extract key feature points in real-time images. Non-maximum suppression is applied to the key feature points to obtain key feature points with response values ​​greater than a preset response value threshold, which are then recorded as target feature points. Match environmental feature points and target feature points in the real-time image. If the match is successful, increase the weight of the corresponding edge in the initial weighted co-view and output the final weighted co-view. The initial pose is calculated using the RANSAC algorithm based on the final weighted common view, and the optimal pose is obtained by using SVD decomposition. A 60° cone-shaped view frustum space is constructed with the camera optical center of the current optimal pose of the inspection personnel as the vertex. The point cloud segmentation blocks in the view frustum space are preserved by using the view frustum clipping technique.

3. The intelligent inspection method based on AR devices according to claim 2, characterized in that, Before associating the virtual indicator with the corresponding virtual anchor point according to the mapping relationship, the method further includes: Obtain the DBoW3 library for the ORB-SLAM3 algorithm, construct the first vocabulary tree based on environmental feature points and ORB descriptors, construct the second vocabulary tree using key feature points and SuperPoint descriptors, merge the root nodes of the first and second vocabulary trees to obtain a hybrid bag-of-words structure; Based on the hybrid bag-of-words structure, the TF-IDF scores of environmental feature points and key feature points in historical images are calculated. A comprehensive score is calculated based on the TF-IDF scores of environmental feature points and key feature points in historical images. Historical images with a comprehensive score greater than a preset score threshold are selected as candidate loop closure images. The similarity between the real-time image and each candidate loop closure image is calculated and denoted as the first data. If there is a first data that is greater than the first preset similarity threshold, the virtual anchor point of the candidate loop closure image corresponding to the first data with the maximum value is used as the virtual anchor point of the real-time image.

4. The intelligent inspection method based on AR devices according to claim 3, characterized in that, If no first data point with a similarity greater than the first preset threshold exists, the method further includes: The similarity between the first vocabulary tree and the second vocabulary tree is calculated using a cross-validation algorithm and recorded as the second data. When the second data exceeds the second preset similarity threshold, the target feature points are projected into the ICP point cloud registration framework to generate matching point pairs with topological constraints. A graph optimization model is constructed, with the reprojection error of matching point pairs as the edge constraint, and the LM algorithm is used to optimize the three-dimensional coordinates of virtual anchor points.

5. The intelligent inspection method based on AR devices according to claim 1, characterized in that, The verification logic also includes: After the inspector responsible for the (i-1)th operation step completes the (i-1)th operation step, he uses the AR device to acquire the image after the operation is completed and collects the operation data after the operation is completed. He then inputs the image and operation data after the operation is completed into the trained equipment status diagnosis model to obtain the diagnosis result after the operation is completed. If the diagnostic results after the operation do not meet expectations, a virtual conference room is constructed, and experts wearing VR devices control and view the target device through the gesture recognition API of the VR controller.

6. The intelligent inspection method based on AR devices according to claim 5, characterized in that, The method further includes: The diagnostic results after the operation are completed are used as the trigger condition for the smart contract. When the diagnostic results after the operation are completed do not meet expectations, an abnormal event record is generated and stored in the distributed ledger.

7. An intelligent inspection system based on AR devices, characterized in that, include: Memory and processor The memory contains a computer-readable storage medium; When the processor processes a computer program stored on the computer-readable storage medium, it implements the method as described in any one of claims 1-6.

Images