Power grid inspection remote assistance system and method based on AR glasses

Abstract

The invention discloses a power grid inspection remote assistance system and method based on AR glasses, and the system comprises an AR glasses end, an expert webpage end and a background server, and achieves the remote assistance through the data interaction of three parties. The AR glasses end collects an operation scene image, recognizes a voice / gesture instruction and loads an equipment 3D model; the expert end supports target frame selection, whiteboard drawing, picture sharing and 3D model interaction; the background server is mainly responsible for data transfer and target tracking algorithm calculation; target tracking adopts an ORB feature detection-optical flow method tracking strategy, target initial positioning is realized through ORB features, then optical flow method real-time tracking is switched, and the tracking efficiency and stability are improved. According to the invention, stable attachment and tracking of the expert labeling information in the dynamic visual field of the inspection personnel are realized, and the precision and operation efficiency of remote guidance are remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of power system equipment operation and maintenance technology, and particularly relates to a remote assistance system and method for power grid inspection based on AR glasses. Background Technology

[0002] As a critical national energy infrastructure, the safe and stable operation of the power grid is directly related to the normal order of the national economy and social life. With the continuous advancement of intelligent technologies, the daily inspection and emergency maintenance of power grid equipment increasingly rely on information technology and remote methods. Remote assistance technology, with its advantages of overcoming geographical limitations and rapidly responding to on-site needs, has become an important direction for improving the efficiency of power grid operation and maintenance. It is gradually being applied to inspection operations in substations, transmission lines, and other scenarios, driving the transformation of traditional operation and maintenance models towards digitalization and collaboration.

[0003] In existing technical solutions, remote assistance systems mostly employ a combination of real-time video calls and basic annotation functions to provide remote support. Inspection personnel are typically equipped with handheld terminals or head-mounted camera devices to transmit real-time images of the equipment to experts, who then communicate via voice guidance supplemented by screen sharing. Some systems are attempting to incorporate augmented reality technology, overlaying two-dimensional annotations such as arrows, circles, and text onto the video feed to assist experts in indicating specific equipment locations.

[0004] However, the most significant drawback of existing technology lies in the lack of stable tracking capability of labeled information in dynamic operation scenarios. Because robust binding to the equipment target is not achieved, once the video image shifts due to personnel movement, equipment rotation, or changes in viewing angle, the labeled information cannot move synchronously with the target, resulting in inaccurate positioning and making it difficult for on-site personnel to accurately understand the expert's intent. This deficiency severely restricts the accuracy and operational efficiency of remote guidance, especially in operations involving complex equipment structures or requiring multi-angle viewing; frequent label drift significantly increases the risk of misoperation and communication costs. Summary of the Invention

[0005] Purpose of the invention: The purpose of this invention is to provide a remote assistance system and method for power grid inspection based on AR glasses, which can achieve stable tracking of labeled information in dynamic scenarios, thereby significantly improving the accuracy and operational efficiency of remote guidance.

[0006] Technical solution: The remote assistance system for power grid inspection described in this invention includes:

[0007] The AR glasses are equipped with an image acquisition module, a voice recognition module, a gesture recognition module, and a 3D model loading and display module, which are used to acquire images of the work scene, recognize user input commands, and load and display 3D models of the equipment to be observed.

[0008] The expert web interface is equipped with a target selection module, a whiteboard drawing module, an image sharing module, and a 3D model viewing and interaction module, which are used by experts to select targets, draw auxiliary patterns, share reference images, and view and interact with 3D models.

[0009] The backend server is configured with a signaling service module, a video stream service module, a file service module, an algorithm service module, and a forwarding service module. The signaling service module and the video stream service module jointly implement the video chat function, the file service module is used to receive and send files, the algorithm service module is used to calculate the target recognition algorithm, and the forwarding service module is used to relay messages that need to be sent to each other by the expert web terminal and the AR glasses terminal.

[0010] Both the AR glasses and the expert webpage establish a communication connection with the backend server via the network.

[0011] This invention's system uses AR glasses to collect real-time data on on-site operations and accurately identify user commands. Combined with 3D model loading and display capabilities, it enables on-site inspectors to intuitively obtain equipment structural information. The expert web interface utilizes target selection, whiteboard drawing, and image sharing modules, allowing remote experts to clearly mark operating areas, create guidance diagrams, and share reference materials. Simultaneously, the 3D model viewing and interaction module provides in-depth analysis of equipment details. The backend server ensures real-time audio and video communication through signaling and video streaming services, enables rapid data transmission through file services, improves target recognition accuracy through algorithm services, and ensures reliable synchronization of commands and data between the two ends through forwarding services. The entire system works collaboratively, achieving stable tracking and virtual-real fusion of marked information in dynamic inspection scenarios, significantly improving the accuracy and operational efficiency of remote guidance.

[0012] Preferably, the target selection module on the expert webpage is used to respond to the expert's selection operation and generate a target selection instruction;

[0013] The algorithm service module of the backend server is configured to receive the bounding box selection instruction and generate target bounding box position data by using a target tracking method that combines ORB feature detection and optical flow tracking.

[0014] The AR glasses are configured to receive the target frame position data and overlay a tracking frame onto the corresponding target on the display screen.

[0015] By initiating selection commands via an expert's web interface, the backend utilizes ORB features and optical flow for robust visual computation. Ultimately, the tracking boxes are overlaid in real time on the AR glasses, enabling the expert-annotated boxes to be continuously and stably locked onto target objects in dynamic videos. This effectively overcomes the annotation drift problem caused by personnel movement or changes in perspective, thus accurately and persistently anchoring the expert's remote guidance intent to the real scene, significantly improving the efficiency of on-site personnel in identifying and operating key targets.

[0016] Preferably, the voice recognition module and gesture recognition module of the AR glasses are used to receive user commands to control the display state of the 3D model, the display state including zooming, rotation and viewpoint switching;

[0017] The forwarding service module of the backend server is used to synchronize the user instructions to the expert web page so that the 3D model displayed on the expert web page and the AR glasses can be kept in sync.

[0018] The 3D model viewing and interaction module on the expert webpage is used to respond to the expert's operation to generate highlight instructions for specific parts of the 3D model.

[0019] The highlighting command is forwarded by the backend server to the AR glasses, and the 3D model loading and display module of the AR glasses performs the highlighting display of the corresponding part.

[0020] Through voice and gesture recognition on the AR glasses, on-site personnel can easily manipulate the 3D model. Simultaneously, a synchronization mechanism on the backend server ensures that the model display on the expert's web interface and the AR glasses is consistent in real time. Experts can proactively issue highlight commands to remotely guide on-site personnel to focus on key parts of the equipment. This two-way interactive mechanism effectively bridges the cognitive gap between remote and on-site personnel, enabling both parties to conduct precise reference and discussion based on a shared, collaboratively operable virtual model, greatly improving the collaborative efficiency and accuracy of remote guidance.

[0021] The remote assistance method for power grid inspection described in this invention includes the following steps:

[0022] Receive the target selection instruction sent by the expert's web interface, and extract the target area information in the instruction as the initial target template;

[0023] Receive real-time image frames of the work scene transmitted from the AR glasses, perform feature detection and matching on the current image frame, and determine that the target detection is successful if the matching similarity reaches the first preset threshold.

[0024] After successful target detection, switch to optical flow method to perform inter-frame target tracking, update the position coordinates of the target box according to the displacement vector of key points, and generate real-time tracking box data;

[0025] The key point matching rate of optical flow tracking is continuously monitored. If the matching rate is lower than the second preset threshold, the tracking is determined to be unsuccessful, and feature detection and matching are re-executed to achieve target relocation.

[0026] This remote assistance method achieves reliable target detection through initial target template extraction and feature matching. After successful detection, it switches to an efficient optical flow method for continuous inter-frame tracking, thereby generating stable real-time tracking boxes. Simultaneously, the system continuously monitors tracking quality. If occlusion or rapid movement leads to insufficient keypoint matching, it automatically re-triggers target detection to resume tracking. This dynamic switching mechanism effectively ensures continuous and stable tracking of the target by the bounding box in complex dynamic scenes, significantly improving the consistency and accuracy of remote guidance.

[0027] Preferably, the feature detection and matching performed on the current image frame includes:

[0028] A Gaussian pyramid is constructed for the image frame, and a corner detection algorithm is used to extract candidate key points of the target at each level of the pyramid.

[0029] Calculate binary descriptors for the extracted candidate keypoints to generate feature vectors with rotation and scale invariance;

[0030] The feature vector of the current image frame is matched with the feature vector of the initial target template, and the feature matching similarity is calculated.

[0031] This feature detection and matching method achieves multi-scale analysis by constructing a Gaussian pyramid and extracts robust features with rotation and scale invariance using corner detection and binary descriptors. This enables the system to achieve high-precision feature matching between the current image and the initial target template even under complex conditions such as changes in field of view and distance, laying a solid foundation for the reliable identification and stable tracking of subsequent targets.

[0032] Preferably, the corner detection algorithm selects pixels with significant grayscale changes as key points by comparing the grayscale differences between candidate pixels and multiple surrounding pixels.

[0033] This corner detection algorithm accurately identifies corner features with rich texture and unique structural information in an image by keenly capturing the significant differences in grayscale values ​​between candidate pixels and their surrounding areas. This method ensures that the extracted key points have high uniqueness and stability, providing a crucial guarantee for subsequent generation of reliable feature descriptors and high-accuracy feature matching, thereby enhancing the robustness of the entire system in target recognition and tracking in complex scenes.

[0034] Preferably, the optical flow method employs a pyramid-based hierarchical iterative calculation approach.

[0035] The optical flow method employs a pyramid-based hierarchical iterative calculation approach, which uses a multi-layered calculation strategy from coarse to fine. It first estimates the macroscopic motion of the target at a large scale, and then gradually refines it to a small scale to correct local displacement. This effectively addresses situations where the target undergoes significant motion or displacement in the image, significantly improving the robustness and computational efficiency of inter-frame target tracking, and ensuring the stability and smooth update of the tracking box position in dynamic scenes.

[0036] Preferably, before the step of receiving the real-time image frame of the work scene transmitted from the AR glasses, the method further includes:

[0037] The AR glasses perform video compression processing on the collected work scene images before transmission; the backend server decompresses the received video stream before forwarding it to the expert webpage.

[0038] By compressing and transmitting the original video on the AR glasses and decompressing and forwarding it on the backend server, the network bandwidth consumption and transmission latency of the real-time video stream are significantly reduced. This ensures the smoothness and real-time performance of the video under limited network conditions, providing a stable and timely video source for remote diagnosis and accurate annotation by experts, and ensuring the continuity and efficiency of the entire remote assistance process.

[0039] Preferably, the 3D model loading and display module of the AR glasses supports two loading modes: local caching and server download. In the local caching mode, the 3D model is pre-stored on the AR glasses. In the server download mode, the AR glasses send a model request to the backend server to obtain the model file.

[0040] By supporting both local caching and server download modes for 3D models, the system can flexibly retrieve the latest models from the server when network conditions are good, and quickly load locally pre-stored models when network conditions are poor or when extreme response speed is required. This ensures the stable and efficient presentation of 3D visualization guidance content in various complex on-site operating environments, effectively improving the environmental adaptability and operational smoothness of the remote assistance system.

[0041] Preferably, the feature detection and matching uses ORB feature vectors, and the optical flow method uses the Lucas-Kanade optical flow algorithm.

[0042] By combining computationally efficient ORB features for target recognition and relocation, and employing the classic Lucas-Kanade optical flow algorithm for fast inter-frame tracking, the algorithm ensures real-time performance while maintaining accuracy and stability during the tracking process. This results in continuous, smooth, and timely visual tracking of target bounding boxes in complex field environments.

[0043] Beneficial Effects: Compared with existing technologies, this invention has the following significant advantages: 1. By performing target recognition calculations through the "algorithm service module" of the backend server in the system architecture, and sending the results to the AR glasses for overlay display via the "forwarding service module," the expert annotation information is stably attached and tracked in the dynamic field of vision of the inspection personnel, significantly improving the accuracy and operational efficiency of remote guidance; 2. By utilizing the "target selection module," "whiteboard drawing module," and "image sharing module" on the expert webpage, combined with the real-time screen display on the AR glasses, the expert's annotations, drawings, and other guidance information can be directly and accurately overlaid on the real working scene of the inspection personnel, effectively solving the problem of... 1. It overcomes the problems of vague descriptions and misunderstandings in traditional voice or video communication; 2. Through the "3D model loading and display module" on the AR glasses and the "3D model viewing and interaction module" on the expert's webpage, both parties can jointly load, view, and operate the 3D model of the same device, providing a visual collaborative platform that transcends two-dimensional images for understanding complex device structures and clarifying fault mechanisms; 3. By integrating functions such as "video chat," "file sharing," "command recognition," and "data forwarding," the system constructs a seamless collaborative remote assistance environment, enabling experts to fully perceive the on-site situation and respond quickly, significantly improving the overall collaborative efficiency from problem diagnosis to resolution. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the system structure of the present invention;

[0045] Figure 2 This is a schematic diagram of the method flow of the present invention. Detailed Implementation

[0046] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0047] This invention provides a remote assistance system for power grid inspection based on AR glasses, such as... Figure 1 As shown, it includes: AR glasses, expert web page, and backend server.

[0048] The AR glasses are equipped with an image acquisition module, a voice recognition module, a gesture recognition module, and a 3D model loading and display module. The image acquisition module captures images of the work scene and generates a video image in front of the glasses. The voice recognition module converts the user's voice into specific commands. The gesture recognition module converts the user's gestures into specific commands. The 3D model loading and display module loads and displays the 3D model of the device to be observed and allows operation of the model according to commands. Specifically, the AR glasses load the device's 3D model through the 3D model loading and display module. The user inputs voice commands through the voice recognition module or triggers gesture commands through the gesture recognition module to control the scaling, rotation, or viewpoint switching of the 3D model. The command data is synchronized to the expert webpage via the backend server to ensure consistency between the 3D model on the expert's end and the glasses. The expert clicks on a specific part of the 3D model through the interactive module on the expert's end, generating a highlight command for that part. This command is forwarded to the AR glasses via the backend server, and the AR glasses highlight the corresponding part of the 3D model.

[0049] The expert web interface includes a target selection module, a whiteboard drawing module, an image sharing module, and a 3D model viewing and interaction module. The target selection module allows experts to select targets and submit the selection information to the backend server for calculation. The whiteboard drawing module allows experts to draw auxiliary patterns and share them with the glasses. The image sharing module can send images to the glasses for sharing reference images. The 3D model viewing and interaction module allows direct viewing of the 3D model from the glasses' perspective, highlighting triggered areas by clicking, and synchronizing the highlighted area information to the glasses. Specifically, experts select targets to be tracked in the image using the target selection module on the expert web interface. The selection command is forwarded by the backend server to the algorithm calculation module. The algorithm calculation module uses the "ORB feature detection-optical flow tracking" target tracking method to generate target bounding box position data, which is then sent to the AR glasses. The AR glasses then overlay the tracking bounding box onto the corresponding target on the displayed screen.

[0050] The backend server is configured with signaling service modules, video streaming service modules, file service modules, algorithm service modules, and forwarding service modules. The signaling service module and video streaming service module work together to enable video chat functionality in the form of multi-person video conferencing. The file service module is used to receive and send files, including 3D model files, image files, and inspection video recordings. The algorithm service module is used to calculate target recognition algorithms and push the results to the AR glasses in real time. The forwarding service module relays messages that need to be sent between the expert web page and the AR glasses, including whiteboard drawings, images, and operations on 3D models.

[0051] Both the AR glasses and the expert webpage establish communication connections with the backend server via the network, forming a three-way data interaction link of "AR glasses - backend server - expert".

[0052] This invention also provides a remote assistance method for power grid inspection based on AR glasses. This method combines ORB and optical flow methods to reduce resource consumption and improve server response speed. Figure 2 As shown, the method includes the following steps.

[0053] Step 1: Obtaining the initial target template

[0054] Experts, through the target selection module on the expert web interface, use a mouse or touch screen to select key equipment components (such as cable joints, insulators, etc.) as targets to be tracked in the received real-time image of the work scene, generating a selection command containing the coordinates and dimensions of the target area. The algorithm service module then extracts the target area information from the command and stores the image of that area as an initial target template, providing a benchmark for subsequent feature matching.

[0055] Step 2: ORB Feature Detection and Matching

[0056] Before performing this step, the image acquisition module on the AR glasses acquires real-time images of the work scene and then performs H.265 video compression processing through the built-in encoding unit. This compression format can significantly reduce the amount of data while ensuring image clarity. The compressed video stream is transmitted over the network to the video stream service module of the backend server, where it is decompressed and then synchronized to the display interface on the expert webpage through the forwarding service module.

[0057] After the decompressed real-time image frames enter the algorithm service module, the ORB feature detection process is started:

[0058] Gaussian Pyramid Construction and Keypoint Extraction: The algorithm service module first constructs a Gaussian pyramid for the current image frame, generating images of different levels through multi-scale scaling to achieve scale invariance detection. At each level of the pyramid, the FAST corner detection algorithm is used to extract candidate keypoints, with the algorithm threshold set to 15 (within the optimal range of 10-20). Specifically, by comparing the grayscale difference between the candidate pixel and its 16 neighboring pixels, pixels with grayscale changes exceeding the threshold are selected as keypoints, ensuring that the extracted feature points have significant discriminative power.

[0059] BRIEF descriptor generation: For the selected candidate keypoints, the algorithm service module calculates their BRIEF descriptors: By randomly selecting several pairs of pixels in the neighborhood of the keypoint and comparing their grayscale values, a binary feature vector is generated. The feature vector is then given rotation invariance by combining the orientation information of the keypoint, and finally, an ORB feature vector with both rotation and scale invariance is obtained.

[0060] Feature matching and detection determination: The ORB feature vector of the current image frame is brute-force matched with the ORB feature vector of the initial target template stored in S1, and the feature matching similarity between the two is calculated. In this embodiment, the preset similarity threshold is 90%. If the matching result reaches or exceeds this threshold, the target detection is determined to be successful, and the subsequent tracking process is entered; if the threshold is not reached, the detection step of S2 is continued for the next frame image.

[0061] Step 3: Inter-frame tracking using Lucas-Kanade optical flow method

[0062] After successful target detection, the system automatically switches to optical flow tracking mode to achieve real-time tracking with low resource consumption.

[0063] Seed point selection and displacement calculation (S31): From the ORB feature points successfully matched in S2, key points with uniform distribution and high response values ​​are selected as optical flow tracking seed points. The algorithm service module uses the Lucas-Kanade optical flow algorithm to calculate the displacement vector of these seed points between two adjacent frames. To balance tracking accuracy and computational efficiency, the algorithm adopts a pyramid hierarchical iterative calculation method, setting the number of iterations to 4 and the pyramid level to 3. The displacement calculation results are optimized through multi-scale iteration to reduce tracking errors caused by image blurring or excessively fast movement.

[0064] Target bounding box position update: Based on the average displacement vector of all seed points, the algorithm service module updates the position coordinates and size of the target bounding box in real time, generates tracking box information containing real-time coordinate data, and transmits it to the AR glasses via a data relay link. The AR glasses display unit overlays the red tracking box on the real-time operation screen to ensure that the inspection personnel can intuitively see the key areas marked by the experts.

[0065] Step 4: Tracking Status Monitoring and Relocation

[0066] The algorithm service module continuously calculates the matching rate of seed points during optical flow tracking (i.e., the proportion of successfully matched seed points in adjacent frames to the total number of seed points), with a preset matching rate threshold of 55%. If the matching rate is ≥55%, the optical flow tracking mode is maintained and the tracking box is continuously updated; if the matching rate is <55%, it is determined that the tracking has failed (mostly caused by severe screen shaking, target occlusion, or rapid movement of inspection personnel), and the system immediately returns to step S2 to re-execute ORB feature detection and matching, thereby achieving rapid relocation of the target and ensuring tracking continuity.

[0067] In addition to the core modules mentioned above, the AR glasses also integrate a dual-mode 3D model loading module, further enhancing the system's usability.

[0068] 3D Model Loading and Multi-Terminal Synchronous Interaction: The 3D model loading and display module supports two modes: local caching and server download. When the inspection area is a frequently inspected area, the local caching mode can be used: 3D models of common equipment in the area (such as 10kV cable branch boxes, surge arresters, etc.) are pre-stored in the local storage unit of the AR glasses. After the inspection personnel recognize the target, they can instantly call it through voice commands (such as "load model") or gesture commands (such as pinching two fingers to trigger loading). When it is necessary to view the local uncached equipment model, switch to server download mode: The AR glasses send a model request to the file service module of the backend server. The file service module retrieves the corresponding model file and transmits it to the AR glasses through a high-speed network. After loading is completed, it is synchronized to the display interface.

[0069] During the model interaction, inspectors can input commands such as "zoom in on the model" or "rotate 90 degrees" through the voice recognition module, or make swiping or zooming gestures through the gesture recognition module. The command data is forwarded to the expert web page via the backend server, ensuring that the 3D model displayed on the expert's end is completely consistent with the operation state on the glasses end. When the expert clicks on a specific part of the model (such as a cable terminal) through the 3D model viewing interaction module, a highlight command for that part is generated. This command is then transmitted to the AR glasses end, and the corresponding part of the model is immediately displayed in bright blue flashing, accurately guiding the inspectors to focus on the core area.

[0070] Example of overall collaborative workflow of the system (taking cable fault inspection in remote areas as an example)

[0071] When inspectors discovered an abnormal discharge phenomenon in a cable corridor in a remote mountainous area, they immediately put on AR glasses and initiated remote assistance via voice command. The AR glasses then sent an assistance request to the web page of a pre-set expert through the backend server.

[0072] After receiving the request, the expert's web interface receives the H.265 compressed video transmitted from the AR glasses via the video streaming service module (the discharge scene is clearly displayed after decompression).

[0073] Experts used the target selection module to select cable connectors around the discharge point. The selection command triggered the background algorithm service module to perform ORB feature detection and optical flow tracking. The tracking box was immediately overlaid on the AR glasses screen. Because the lighting was dim, the experts uploaded a standard structural diagram of the same type of cable connector through the image sharing module. The image was then displayed synchronously on the AR glasses screen.

[0074] To further analyze the fault mechanism, on-site personnel accessed the 3D model viewing interaction module, triggering the server to send a 3D model of the cable connector of this model to the experts. The inspectors rotated the model to a cross-sectional view using gestures, and the experts clicked on the insulation layer inside the model and triggered a highlight. At the same time, they drew a schematic diagram of insulation layer damage repair using the whiteboard drawing module. The schematic diagram was directly superimposed on the real-time screen of the AR glasses and the 3D model.

[0075] Inspection personnel conducted tests based on highlighted instructions and graphic prompts. After confirming the insulation layer was damaged, the results were fed back via voice, and experts provided remote guidance on the repair steps. The entire process did not require the presence of experts and the fault diagnosis and repair plan were determined in just 30 minutes, significantly reducing the risk of power grid outages.

Claims

1. A remote assistance system for power grid inspection based on AR glasses, characterized in that, include: The AR glasses are equipped with an image acquisition module, a voice recognition module, a gesture recognition module, and a 3D model loading and display module, which are used to acquire images of the work scene, recognize user input commands, and load and display 3D models of the equipment to be observed. The expert web interface is equipped with a target selection module, a whiteboard drawing module, an image sharing module, and a 3D model viewing and interaction module, which are used by experts to select targets, draw auxiliary patterns, share reference images, and view and interact with 3D models. The backend server is configured with a signaling service module, a video stream service module, a file service module, an algorithm service module, and a forwarding service module. The signaling service module and the video stream service module jointly implement the video chat function, the file service module is used to receive and send files, the algorithm service module is used to calculate the target recognition algorithm, and the forwarding service module is used to relay messages that need to be sent to each other by the expert web terminal and the AR glasses terminal. Both the AR glasses and the expert webpage establish a communication connection with the backend server via the network.

2. The system according to claim 1, characterized in that, The target selection module on the expert webpage is used to respond to the expert's selection operation and generate target selection instructions; The algorithm service module of the backend server is configured to receive the bounding box selection instruction and generate target bounding box position data by using a target tracking method that combines ORB feature detection and optical flow tracking. The AR glasses are configured to receive the target frame position data and overlay a tracking frame onto the corresponding target on the display screen.

3. The system according to claim 1, characterized in that, The voice recognition module and gesture recognition module of the AR glasses are used to receive user commands to control the display state of the 3D model, including zooming, rotation and viewpoint switching. The forwarding service module of the backend server is used to synchronize the user instructions to the expert web page so that the 3D model displayed on the expert web page and the AR glasses can be kept in sync. The 3D model viewing and interaction module on the expert webpage is used to respond to the expert's operation to generate highlight instructions for specific parts of the 3D model. The highlighting command is forwarded by the backend server to the AR glasses, and the 3D model loading and display module of the AR glasses performs the highlighting display of the corresponding part.

4. A remote assistance method for power grid inspection based on AR glasses, characterized in that, Includes the following steps: Receive the target selection instruction sent by the expert's web interface, and extract the target area information in the instruction as the initial target template; Receive real-time image frames of the work scene transmitted from the AR glasses, perform feature detection and matching on the current image frame, and determine that the target detection is successful if the matching similarity reaches the first preset threshold. After successful target detection, switch to optical flow method to perform inter-frame target tracking, update the position coordinates of the target box according to the displacement vector of key points, and generate real-time tracking box data; The key point matching rate of optical flow tracking is continuously monitored. If the matching rate is lower than the second preset threshold, the tracking is determined to be unsuccessful, and feature detection and matching are re-executed to achieve target relocation.

5. The method according to claim 4, characterized in that, The feature detection and matching performed on the current image frame includes: A Gaussian pyramid is constructed for the image frame, and a corner detection algorithm is used to extract candidate key points of the target at each level of the pyramid. Calculate binary descriptors for the extracted candidate keypoints to generate feature vectors with rotation and scale invariance; The feature vector of the current image frame is matched with the feature vector of the initial target template, and the feature matching similarity is calculated.

6. The method according to claim 5, characterized in that, The corner detection algorithm selects pixels with significant grayscale changes as key points by comparing the grayscale differences between candidate pixels and multiple surrounding pixels.

7. The method according to claim 4, characterized in that, The optical flow method employs a pyramid-based hierarchical iterative calculation approach.

8. The method according to claim 4, characterized in that, Before the step of receiving the real-time image frame of the work scene transmitted from the AR glasses, the method further includes: The AR glasses perform video compression processing on the collected work scene images before transmission; the backend server decompresses the received video stream before forwarding it to the expert webpage.

9. The method according to claim 4, characterized in that, The 3D model loading and display module of the AR glasses supports two loading modes: local caching and server download. In the local caching mode, the 3D model is pre-stored on the AR glasses. In the server download mode, the AR glasses send a model request to the backend server to obtain the model file.

10. The method according to claim 4, characterized in that, The feature detection and matching uses ORB feature vectors, and the optical flow method uses the Lucas-Kanade optical flow algorithm.

Images