Electric power equipment remote cooperation maintenance method and system based on augmented reality driving

Abstract

The invention provides a remote cooperative maintenance method and system for power equipment based on augmented reality driving, the method is applied to a maintenance system, the maintenance system comprises an on-site maintenance terminal and an expert remote terminal, and the on-site maintenance terminal is provided with augmented reality glasses. The method comprises the following steps: collecting picture information of power equipment through augmented reality glasses at a field maintenance end, and transmitting the picture information to an expert remote end; receiving annotation information of an expert on the picture information through the expert remote end, and feeding back the annotation information to the field maintenance end; the received annotation information is displayed to the corresponding position in the collected picture information through the field maintenance terminal, remote cooperation of field maintenance personnel and experts is achieved, the experts can conduct annotation guidance on field equipment at the remote terminal, the maintenance personnel can see the annotation information in real time, and therefore the maintenance efficiency and accuracy are improved, and the maintenance cost is reduced. And the safety risk caused by misoperation is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of power equipment maintenance, and in particular to a method and system for remote collaborative maintenance of power equipment based on augmented reality. Background Art

[0002] The stable operation of power equipment is crucial to ensuring power supply. During the maintenance of power equipment, on-site maintenance personnel may encounter complex fault conditions, which can be difficult to solve quickly and accurately due to their limited experience and knowledge.

[0003] During the maintenance of power equipment, traditional remote collaboration methods mainly rely on telephones and video conferencing. These methods cannot intuitively display the actual situation of the equipment, and it is difficult for experts to accurately guide on-site maintenance personnel to perform operations. The collaboration between experts and maintenance personnel is inefficient, resulting in extended maintenance time and even possible safety accidents due to improper operation. Summary of the Invention

[0004] The present invention provides a remote collaborative maintenance method and system for electric power equipment based on augmented reality driving, which is used to solve the technical problem that the existing technology cannot achieve efficient collaborative maintenance during the maintenance process of electric power equipment.

[0005] In one aspect, the present invention provides a method for remote collaborative maintenance of power equipment based on augmented reality. The method is applied to a maintenance system, wherein the maintenance system includes an on-site maintenance terminal and an expert remote terminal, wherein the on-site maintenance terminal is provided with augmented reality glasses. The method includes: Collecting image information of the power equipment through the augmented reality glasses at the on-site maintenance terminal, and transmitting the image information to the expert remote terminal; receiving, via the expert remote terminal, the expert's annotation information on the image information, and feeding back the annotation information to the on-site maintenance terminal; The received annotation information is displayed at a corresponding position in the collected image information through the on-site maintenance terminal.

[0006] According to the present invention, a remote collaborative maintenance method for power equipment based on augmented reality is provided, wherein the augmented reality glasses at the on-site maintenance end collect image information of the power equipment and transmit the image information to the expert remote end, including: Collecting image information of the power equipment through the augmented reality glasses at the on-site maintenance terminal; Dividing the image information into key information and non-key information; When the network transmission does not meet the preset transmission requirements, the key information is preferentially transmitted to the expert remote terminal, and then the non-key information is transmitted to the expert remote terminal; When the network transmission meets the preset transmission requirements, the critical information and the non-critical information are simultaneously transmitted to the expert remote terminal.

[0007] According to the present invention, a remote collaborative maintenance method for power equipment based on augmented reality driving is provided, wherein the image information is divided into key information and non-key information, including: Using edge computing to identify abnormal areas in the image information and characteristic points of the power equipment as key information; wherein the abnormal areas include but are not limited to abnormal temperature areas, damaged areas, and areas indicating abnormal equipment operating status; and the characteristic points include but are not limited to outlines, connection points, and identifications of power equipment components; Information other than the key information in the screen information is regarded as non-key information.

[0008] According to the present invention, a remote collaborative maintenance method for power equipment based on augmented reality driving is provided, wherein the expert remote terminal receives the expert's annotation information on the image information and feeds the annotation information back to the on-site maintenance terminal, including: After receiving the key information at the expert remote terminal, identifying and displaying a failure mode of the key information; Receiving annotation information made by an expert on the screen information using an augmented reality-assisted annotation tool; wherein the annotation information includes but is not limited to indication of an inspection location, operating steps, precautions, and fault diagnosis suggestions; Determine a corresponding transmission priority according to the fault type of the marked information, and when network transmission does not meet the preset transmission requirements, transmit the marked information to the on-site maintenance terminal based on the transmission priority; The timestamp, expert identity, and annotation content of the annotation information are recorded.

[0009] According to the present invention, a remote collaborative maintenance method for power equipment based on augmented reality driving is provided, which identifies and displays the failure mode of the key information, including: Based on a preset failure mode database, identifying the failure mode of the key information; Generating a fault diagnosis report according to the fault mode, and displaying the fault diagnosis report on a display interface of the expert remote terminal; Receiving annotation information performed by an expert on the image information using an augmented reality-assisted annotation tool, including: providing an augmented reality-assisted annotation tool at the expert's remote terminal; Receiving annotation information performed by an expert on the image information using the augmented reality auxiliary annotation tool; Dynamically adjust the annotation position, size and transparency of the received annotation information to align the annotation information with the screen information.

[0010] According to the present invention, a remote collaborative maintenance method for power equipment based on augmented reality driving is provided, wherein the on-site maintenance terminal displays the received annotation information at a corresponding position in the collected image information, including: After receiving the annotation information, the on-site maintenance terminal uses a time-space synchronization algorithm to match the annotation information with the collected image information to ensure that the annotation information is consistent with the actual position of the device when displayed; The on-site maintenance terminal obtains the maintenance personnel's perspective and operation progress, and dynamically updates the content and orientation of the annotation information based on the perspective and the operation progress, so that the annotation information is always consistent with the maintenance personnel's current operation and perspective; Based on the lighting conditions of the maintenance personnel's field of view, the brightness and contrast of the annotation information are automatically adjusted when the annotation information is displayed.

[0011] According to the present invention, a remote collaborative maintenance method for power equipment based on augmented reality driving is provided, in which the maintenance personnel's perspective and operation progress are obtained through the on-site maintenance terminal, and the content and orientation of the annotation information are dynamically updated based on the perspective and operation progress, so that the annotation information is always consistent with the maintenance personnel's current operation and perspective, including: Collecting head posture data using a gyroscope and an accelerometer built into the augmented reality glasses, wherein the head posture data includes pitch angle, yaw angle, and roll angle; Collecting gaze direction data through an eye tracking sensor built into the augmented reality glasses, wherein the gaze direction data includes a gaze direction vector and a focal distance; Determining the current viewing angle information of the maintenance personnel according to the head posture data and the sight direction data, wherein the viewing angle information includes viewing direction, viewing angle, and viewing range; Adjust the display orientation of the annotation information according to the current viewing angle information to ensure that the annotation information is always within the field of view of the maintenance personnel and is consistent with the actual position and angle of the equipment; Combined with the maintenance personnel's operation progress, the display content of the annotation information is dynamically updated to provide guidance information corresponding to the current operation stage.

[0012] According to the present invention, a remote collaborative maintenance method for power equipment based on augmented reality driving is provided, wherein the image information of the power equipment is collected by the augmented reality glasses at the on-site maintenance end, and further comprises: Collecting three-dimensional spatial data of power equipment through the augmented reality glasses at the on-site maintenance end; Match and fuse the collected 3D spatial data with the pre-built 3D model of power equipment; On the display interface of the augmented reality glasses at the on-site maintenance end, the matched and fused three-dimensional model and the collected image information are superimposed and displayed.

[0013] According to the present invention, a remote collaborative maintenance method for power equipment based on augmented reality is provided, which matches and fuses collected three-dimensional spatial data with a pre-built three-dimensional model of the power equipment, including: Extract feature points from the collected three-dimensional spatial data; Using a feature matching algorithm, the extracted feature points are matched with corresponding feature points in a pre-built three-dimensional model of the power equipment to determine the spatial correspondence between the three-dimensional spatial data and the three-dimensional model of the power equipment; According to the matching spatial correspondence, the three-dimensional spatial data is spatially aligned with the three-dimensional model of the power equipment to maintain consistency in position and posture.

[0014] On the other hand, the present invention also provides an augmented reality-driven remote collaborative maintenance system for power equipment, comprising: an on-site maintenance terminal and an expert remote terminal, wherein the on-site maintenance terminal is provided with augmented reality glasses; The augmented reality glasses at the on-site maintenance end collect image information of the power equipment and transmit the image information to the expert remote end; The expert remote terminal receives the expert's annotation information on the image information and feeds the annotation information back to the on-site maintenance terminal; The on-site maintenance terminal displays the received annotation information at a corresponding position in the collected image information.

[0015] The augmented reality-driven remote collaborative maintenance method and system for power equipment provided by the present invention collects image information of the power equipment through augmented reality glasses at the on-site maintenance end, transmits the image information to the expert remote end, receives the expert's annotation information on the image information through the expert remote end, and feeds back the annotation information to the on-site maintenance end, thereby realizing remote collaboration between on-site maintenance personnel and experts. Experts can provide annotation guidance for on-site equipment at the remote end, and maintenance personnel can see the annotation information in real time, thereby improving maintenance efficiency and accuracy and reducing safety risks caused by improper operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 11 is a flow chart of a method for remote collaborative maintenance of power equipment based on augmented reality driving provided by an embodiment of the present invention; Figure 2 1 is a schematic structural diagram of an augmented reality-driven remote collaborative maintenance system for power equipment provided by an embodiment of the present invention; Figure 3 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0018] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0019] Figure 1 This is a flow chart illustrating a method for remote collaborative maintenance of power equipment based on augmented reality, as provided in an embodiment of the present invention. This method is applied to a maintenance system that can include an on-site maintenance terminal and an expert remote terminal, with the on-site maintenance terminal equipped with augmented reality glasses. The on-site maintenance terminal and the expert remote terminal can communicate remotely.

[0020] See also Figure 1 , the remote collaborative maintenance method of power equipment based on augmented reality driving may include the following steps.

[0021] Step 101: Collect image information of power equipment through augmented reality glasses at the on-site maintenance end, and transmit the image information to the expert remote end.

[0022] For example, after arriving at the power equipment maintenance site wearing augmented reality (AR) glasses, field maintenance personnel activate the glasses' video capture function. The glasses have a built-in high-definition camera that can capture images and video information of power equipment in real time. The camera's high resolution, high frame rate, and excellent low-light performance ensure clear and accurate equipment information can be captured in a variety of environmental conditions. The captured image information is transmitted in real time to the remote expert via a high-speed and stable communication network (such as 5G). To reduce the amount of video data, advanced video encoding algorithms (such as H.265) are used to efficiently encode the captured video, achieving low-latency video transmission.

[0023] Step 102: Receive the expert's annotation information on the screen information through the expert remote terminal, and feed back the annotation information to the on-site maintenance terminal.

[0024] For example, a remote expert uses dedicated software to receive images of power equipment transmitted from an on-site maintenance site. Experts can view these real-time images on a large screen or specialized equipment. Experts use annotation software to mark the video footage. The annotation software supports a variety of annotation tools, such as brushes, arrows, and text annotations. Based on the live video feed, experts can mark directly on the video, indicating key areas of concern, operating procedures, and precautions for maintenance personnel. Once the annotation is complete, the remote expert transmits the information back to the on-site maintenance site in real time via the communication network. This information is sent as data, ensuring it reaches the on-site maintenance site quickly and accurately.

[0025] Step 103: Display the received annotation information at a corresponding position in the collected image information through the on-site maintenance terminal.

[0026] For example, AR glasses at the on-site maintenance site receive annotation information transmitted from the expert's remote site. The AR glasses fuse the annotation information with the real-time video footage and display it. For example, if the expert marks a part for disassembly in the video, this annotation is accurately displayed at the actual location of the part in the AR glasses' field of view.

[0027] In this embodiment, the augmented reality glasses at the on-site maintenance end collect the screen information of the power equipment and transmit the screen information to the expert remote end. The expert remote end receives the expert's annotation information on the screen information and feeds the annotation information back to the on-site maintenance end, thereby realizing remote collaboration between on-site maintenance personnel and experts. Experts can provide annotation guidance for on-site equipment at the remote end, and maintenance personnel can see the annotation information in real time, thereby improving maintenance efficiency and accuracy and reducing safety risks caused by improper operation.

[0028] In one embodiment of this specification, the augmented reality glasses at the on-site maintenance end collect image information of the power equipment and transmit the image information to the expert remote end, including: Collect visual information of power equipment through augmented reality glasses at the on-site maintenance end; Divide the screen information into key information and non-key information; When network transmission does not meet the preset transmission requirements, key information is transmitted to the expert remote terminal first, and then non-key information is transmitted to the expert remote terminal; When the network transmission meets the preset transmission requirements, critical information and non-critical information are transmitted to the expert remote end at the same time.

[0029] In this embodiment, during the image information transmission process, information is divided into critical and non-critical information based on network conditions, with critical information being transmitted first. This ensures that even in unstable network environments, experts can quickly obtain key device information, further improving the efficiency and reliability of remote collaboration. The preset transmission requirements can be set according to the situation and are not specifically limited here.

[0030] In one embodiment of this specification, the screen information is divided into key information and non-key information, including: Use edge computing to identify abnormal areas in image information and characteristic points of power equipment as key information; abnormal areas include but are not limited to abnormal temperature areas, damaged areas, and areas indicating abnormal equipment operating status; characteristic points include but are not limited to the outlines, connection points, and logos of power equipment components; Information other than the key information in the screen information is regarded as non-key information.

[0031] In this embodiment, for example, if a screen displays a running transformer, using edge computing technology, the system can identify areas of abnormal transformer surface temperature (for example, a localized area with significantly higher temperatures than other areas, possibly due to a localized overheating fault). This abnormal temperature area is identified as key information. The system can also identify features such as the transformer's outline, connection points (such as high-voltage and low-voltage terminals), and identification (such as the number on the equipment nameplate). These features are also key information because they can help experts quickly locate important parts of the equipment. Equipment operating status abnormality indication areas are specific areas on power equipment that can visually reflect whether its operating status is normal. These areas typically include indicator lights, instrument readings, display screens, or other visual elements that display the equipment's operating parameters or status.

[0032] This embodiment uses edge computing technology to accurately identify abnormal areas and feature points in the image as key information, enabling the system to more efficiently screen out content that is of great guiding significance for maintenance, providing a more accurate basis for subsequent labeling and guidance, and improving the pertinence and accuracy of maintenance.

[0033] In one embodiment of the present specification, the expert remote terminal receives the expert's annotation information on the screen information and feeds the annotation information back to the on-site maintenance terminal, including: After receiving the key information at the remote end of the expert, identify and display the failure mode of the key information; Receive annotation information from experts using augmented reality-assisted annotation tools on the screen information; the annotation information includes but is not limited to indications of maintenance locations, operating steps, precautions, and fault diagnosis suggestions; Determine the corresponding transmission priority based on the fault type of the marked information. When the network transmission does not meet the preset transmission requirements, the marked information is transmitted to the on-site maintenance terminal based on the transmission priority; Record the timestamp, expert identity, and annotation content of the annotation information.

[0034] In this embodiment, for example, suppose that the image information captured by the AR glasses at the on-site maintenance end contains a key piece of information: a temperature abnormality (such as local overheating) in a transformer component. After the expert remote end receives this key information, the system will match it based on a preset fault mode database. For example, the database records that "temperature abnormalities are usually related to component overload or poor heat dissipation." The system identifies the fault mode of this key information as "poor heat dissipation." The system generates a fault diagnosis report based on the identified fault mode and displays it on the display interface of the expert remote end. The report content may include: Fault location: transformer component X; Fault mode: poor heat dissipation; and a recommendation to check whether the heat dissipation system is blocked.

[0035] After reviewing the fault diagnosis report remotely, the expert used augmented reality-assisted annotation tools to annotate the image. Using the brush tool, the expert drew a red arrow on the component with the abnormal temperature and added a caption: "Please check if the radiator is blocked here." The annotation included the location to be inspected (the component pointed by the red arrow), the operating steps (check the radiator), and precautions (make sure the device is powered off before operating).

[0036] After the expert completes the annotation, the system needs to transmit the annotation information back to the on-site maintenance team. The system prioritizes transmission based on the fault type (e.g., poor heat dissipation) of the annotation. For example, a heat dissipation issue, which could cause equipment overheating and damage, is marked as high priority. If network conditions are unstable, the system prioritizes transmission of high-priority annotations (such as red arrows and key text annotations), ensuring that on-site maintenance personnel receive the most important guidance quickly.

[0037] In this embodiment, the experts use augmented reality-assisted annotation tools to perform annotations on the remote end. The system determines the transmission priority of the annotation information based on the fault type, ensuring that important annotation information is transmitted first. At the same time, the detailed information of the annotation information is recorded to facilitate subsequent tracing and analysis, further optimizing the annotation guidance function of remote collaboration.

[0038] In one embodiment of this specification, identifying and displaying failure modes of key information includes: Based on the preset failure mode database, identify the failure mode of key information; Generate a fault diagnosis report based on the fault mode and display the fault diagnosis report on the display interface of the expert remote terminal; Receive annotation information from experts using augmented reality-assisted annotation tools on the screen information, including: Provide augmented reality-assisted annotation tools on the expert remote end; Receive annotation information made by experts on the screen information using augmented reality assisted annotation tools; Dynamically adjust the annotation position, size and transparency of the received annotation information to align the annotation information with the screen information.

[0039] In this embodiment, based on a preset fault mode database, the fault mode of key information is quickly identified and a diagnostic report is generated to provide decision support for experts. At the same time, the display effect of the annotation information is dynamically adjusted to ensure that the annotation information is accurately aligned with the screen information, thereby improving the efficiency of fault diagnosis and the accuracy of annotation.

[0040] In one embodiment of the present specification, displaying the received annotation information at a corresponding position in the collected image information by the on-site maintenance terminal includes: After receiving the annotation information at the on-site maintenance end, the annotation information is matched with the collected image information using a time-space synchronization algorithm to ensure that the annotation information is consistent with the actual position of the equipment when displayed; The maintenance personnel's perspective and operation progress are obtained through the on-site maintenance terminal, and the content and orientation of the annotation information are dynamically updated based on the perspective and operation progress, so that the annotation information is always consistent with the maintenance personnel's current operation and perspective; Based on the lighting conditions of the maintenance personnel's field of view, the brightness and contrast of the annotation information are automatically adjusted when the annotation information is displayed.

[0041] In this embodiment, through the spatiotemporal synchronization algorithm and dynamic update mechanism, the annotation information can adjust the display content and orientation in real time according to the maintenance personnel's perspective and operation progress, and optimize the display effect in combination with lighting conditions to ensure that the annotation information is always clearly visible and closely synchronized with the maintenance personnel's operations, thereby enhancing the interactivity and guidance of the maintenance process.

[0042] In one embodiment of this specification, the maintenance personnel's perspective and operation progress are obtained through the on-site maintenance terminal, and the content and orientation of the annotation information are dynamically updated based on the perspective and operation progress, so that the annotation information is always consistent with the maintenance personnel's current operation and perspective, including: The head posture data is collected through the gyroscope and accelerometer built into the augmented reality glasses, where the head posture data includes pitch angle, yaw angle and roll angle; Collecting gaze direction data through an eye tracking sensor built into the augmented reality glasses, wherein the gaze direction data includes a gaze direction vector and a focal distance; Determine the current viewing angle information of the maintenance personnel based on the head posture data and the sight direction data, wherein the viewing angle information includes the viewing direction, viewing angle, and viewing range; Adjust the display orientation of the annotation information based on the current viewing angle to ensure that the annotation information is always within the field of view of the maintenance personnel and consistent with the actual position and angle of the equipment; Combined with the maintenance personnel's operation progress, the display content of the annotation information is dynamically updated to provide guidance information corresponding to the current operation stage.

[0043] In this embodiment, with the help of multiple sensors built into the augmented reality glasses, the maintenance personnel's head posture and line of sight direction data are accurately obtained, so as to accurately determine their perspective information, and dynamically adjust the display orientation and content of the annotation information accordingly, so as to achieve real-time matching of the annotation information with the maintenance personnel's current operation and perspective, further improving the convenience and accuracy of maintenance.

[0044] In one embodiment of the present specification, collecting image information of power equipment through augmented reality glasses at the on-site maintenance terminal also includes: Collect three-dimensional spatial data of power equipment through augmented reality glasses at the on-site maintenance end; Match and fuse the collected 3D spatial data with the pre-built 3D model of power equipment; On the display interface of the augmented reality glasses at the on-site maintenance end, the matched and fused three-dimensional model is superimposed and displayed with the collected image information.

[0045] In this embodiment, three-dimensional spatial data of the power equipment is collected and matched and fused with the three-dimensional model, and the fused three-dimensional model is superimposed and displayed on the collected image information, providing maintenance personnel with more intuitive equipment structure and component location information, further enhancing the maintenance personnel's visual perception and understanding of the equipment, and improving maintenance efficiency and accuracy.

[0046] In one embodiment of the present specification, matching and fusing the collected three-dimensional spatial data with a pre-built three-dimensional model of the power equipment includes: Extracting feature points from the collected three-dimensional spatial data; wherein the feature points include but are not limited to the outlines, connection points, identification marks and boundaries of abnormal areas of equipment components; Using a feature matching algorithm, the extracted feature points are matched with corresponding feature points in a pre-built three-dimensional model of the power equipment to determine the spatial correspondence between the three-dimensional spatial data and the three-dimensional model of the power equipment; According to the matching spatial correspondence, the three-dimensional spatial data is spatially aligned with the three-dimensional model of the power equipment to maintain consistency in position and posture.

[0047] In this embodiment, accurate alignment of data and model is achieved through feature point extraction and matching algorithms, ensuring the consistency of position and posture between the three-dimensional model and the actual equipment, providing maintenance personnel with more accurate visual assistance, and further improving the system's augmented reality effect and maintenance reliability.

[0048] In some other embodiments of this specification, the method for remote collaborative maintenance of power equipment based on augmented reality driving further includes: At the remote end, experts use machine learning algorithms to predict potential fault points based on historical maintenance data and real-time image information, and automatically generate pre-annotated information; The remote expert further optimizes the annotation content based on the pre-annotated information and combines expert experience to form the final annotation information; At the on-site maintenance end, when the maintenance personnel's operation progress deviates from the preset standard operating procedures, the augmented reality glasses actively remind the maintenance personnel through visual, voice or vibration feedback, and adjust the display position and content of the annotation information in real time to guide the maintenance personnel back to the correct operating procedures; The on-site maintenance end dynamically updates the accuracy and priority of pre-annotation information based on the maintenance personnel's operational feedback to optimize subsequent fault prediction and annotation generation.

[0049] In some other embodiments of this specification, a machine learning algorithm is used to predict potential fault points and automatically generate pre-labeled information, including: Collect and organize historical maintenance data, including fault type, fault location, maintenance operation records, equipment operating parameters and corresponding annotation information; Build a machine learning model, using historical maintenance data as training samples to train the model to identify potential failure modes and key features; At the remote end, the expert receives real-time image information and equipment operating parameters transmitted from the on-site maintenance end and inputs them into the machine learning model; The machine learning model predicts potential fault points based on real-time input data and generates pre-annotated information, including the predicted fault location, possible fault type, and preliminary repair recommendations. The expert remote end verifies and adjusts the pre-annotated information, and supplements the detailed annotation content based on the expert experience to form the final annotation information; The generation process of pre-labeled information and the expert's adjustments are recorded to optimize the machine learning model and improve the accuracy and practicality of subsequent predictions.

[0050] In some other embodiments of this specification, the method for remote collaborative maintenance of power equipment based on augmented reality driving further includes: At the remote end, virtual reality (VR) equipment is used to build a virtual maintenance scene that is consistent with the on-site maintenance environment. Experts can rehearse the maintenance operation process in the virtual scene and generate operation guidance templates; The expert remote terminal synchronizes the operation guidance template to the augmented reality glasses at the on-site maintenance terminal. During the actual operation, the augmented reality glasses display the operation guidance template in the virtual scene in real time and dynamically adjust the display content according to the actual situation on site. At the on-site maintenance end, augmented reality glasses use eye tracking and gesture recognition technology to capture the maintenance personnel's operating intentions and questions in real time, and feed this information back to the remote expert end; Based on the feedback information, the remote expert provides targeted real-time interactive guidance to the maintenance personnel through augmented reality glasses, including voice answers, dynamic annotation adjustments or virtual operation demonstrations.

[0051] In some other embodiments of this specification, the steps of constructing a virtual maintenance scene and generating an operation guidance template using a virtual reality device include: At the remote end, virtual reality equipment is used to create a virtual maintenance scene that is consistent with the on-site maintenance environment, including equipment models, maintenance tools, operation interfaces and other elements; Experts conduct operation rehearsals in a virtual maintenance scenario, recording the operation path, key operation steps, and precautions; Automatically generate an operation guidance template based on the expert's operation preview process. The operation guidance template includes visual annotations, voice prompts, and gesture operation instructions for each operation step. Synchronize the operation guide template with the augmented reality glasses at the on-site maintenance end. When the maintenance personnel are actually operating, the augmented reality glasses will display the corresponding annotation information and operation prompts according to the operation guide template; At the on-site maintenance end, augmented reality glasses use eye tracking and gesture recognition technology to monitor the maintenance personnel's operating status in real time. When operational deviations are detected, the displayed operation guidance template content is automatically adjusted to guide the maintenance personnel to correct the operation; The expert remote terminal updates the operation guidance template in real time based on the operation status and problems fed back by the on-site maintenance terminal, ensuring that it is consistent with the actual situation on site and providing accurate interactive guidance for maintenance personnel.

[0052] Based on the same general inventive concept, the present invention also protects a remote collaborative maintenance system for power equipment driven by augmented reality, such as Figure 2 As shown, Figure 2This is a schematic diagram of the structure of the augmented reality-driven remote collaborative maintenance system for electric power equipment provided by an embodiment of the present invention. The augmented reality-driven remote collaborative maintenance system for electric power equipment provided by the present invention is described below. The augmented reality-driven remote collaborative maintenance system for electric power equipment described below and the augmented reality-driven remote collaborative maintenance method for electric power equipment described above can be used in conjunction with each other.

[0053] The power equipment remote collaborative maintenance system driven by augmented reality includes an on-site maintenance terminal 201 and an expert remote terminal 202 , and the on-site maintenance terminal 201 is provided with augmented reality glasses 203 .

[0054] The augmented reality glasses 203 at the on-site maintenance terminal 201 collect the image information of the power equipment and transmit the image information to the expert remote terminal 202; The expert remote terminal 202 receives the expert's annotation information on the screen information and feeds the annotation information back to the on-site maintenance terminal 201; The on-site maintenance terminal 201 displays the received annotation information at a corresponding position in the collected image information.

[0055] Figure 3 It is a structural diagram of an electronic device provided by an embodiment of the present invention.

[0056] like Figure 3 As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communication interface 320, and the memory 330 communicate with each other via the communication bus 340. The processor 310 may call the logic instructions in the memory 330 to execute the remote collaborative maintenance method for power equipment based on augmented reality.

[0057] Furthermore, the logic instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0058] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the augmented reality-driven remote collaborative maintenance method for power equipment provided by the above methods.

[0059] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the augmented reality-driven remote collaborative maintenance method for power equipment provided by the above methods.

[0060] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0061] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0062] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A remote collaborative maintenance method for power equipment based on augmented reality, characterized in that: The method is applied to a maintenance system, the maintenance system comprising an on-site maintenance terminal and an expert remote terminal, the on-site maintenance terminal being provided with augmented reality glasses, and the method comprising: Collecting image information of the power equipment through the augmented reality glasses at the on-site maintenance terminal, and transmitting the image information to the expert remote terminal; receiving, via the expert remote terminal, the expert's annotation information on the image information, and feeding back the annotation information to the on-site maintenance terminal; The received annotation information is displayed at a corresponding position in the collected image information through the on-site maintenance terminal.

2. The method for remote collaborative maintenance of power equipment based on augmented reality driving according to claim 1 is characterized in that: The method of collecting image information of the power equipment through the augmented reality glasses at the on-site maintenance terminal and transmitting the image information to the expert remote terminal includes: Collecting image information of the power equipment through the augmented reality glasses at the on-site maintenance terminal; Dividing the image information into key information and non-key information; When the network transmission does not meet the preset transmission requirements, the key information is preferentially transmitted to the expert remote terminal, and then the non-key information is transmitted to the expert remote terminal; When the network transmission meets the preset transmission requirements, the critical information and the non-critical information are simultaneously transmitted to the expert remote terminal.

3. The method for remote collaborative maintenance of power equipment based on augmented reality driving according to claim 2 is characterized in that: The screen information is divided into key information and non-key information, including: Using edge computing to identify abnormal areas in the image information and characteristic points of the power equipment as key information; wherein the abnormal areas include but are not limited to abnormal temperature areas, damaged areas, and areas indicating abnormal equipment operating status; and the characteristic points include but are not limited to outlines, connection points, and identifications of power equipment components; Information other than the key information in the screen information is regarded as non-key information.

4. The method for remote collaborative maintenance of power equipment based on augmented reality driving according to claim 2, characterized in that: Receiving, through the expert remote terminal, the expert's annotation information on the image information, and feeding back the annotation information to the on-site maintenance terminal, including: After receiving the key information at the expert remote terminal, identifying and displaying a failure mode of the key information; Receiving annotation information made by an expert on the screen information using an augmented reality-assisted annotation tool; wherein the annotation information includes but is not limited to indication of an inspection location, operating steps, precautions, and fault diagnosis suggestions; Determine a corresponding transmission priority according to the fault type of the marked information, and when network transmission does not meet the preset transmission requirements, transmit the marked information to the on-site maintenance terminal based on the transmission priority; The timestamp, expert identity, and annotation content of the annotation information are recorded.

5. The method for remote collaborative maintenance of power equipment based on augmented reality driving according to claim 4 is characterized in that: Identify and display failure modes with critical information, including: Based on a preset failure mode database, identifying the failure mode of the key information; Generating a fault diagnosis report according to the fault mode, and displaying the fault diagnosis report on a display interface of the expert remote terminal; Receiving annotation information performed by an expert on the image information using an augmented reality-assisted annotation tool, including: providing an augmented reality-assisted annotation tool at the expert's remote terminal; Receiving annotation information performed by an expert on the image information using the augmented reality auxiliary annotation tool; Dynamically adjust the annotation position, size and transparency of the received annotation information to align the annotation information with the screen information.

6. The method for remote collaborative maintenance of power equipment based on augmented reality driving according to claim 5 is characterized in that: Displaying the received annotation information at a corresponding position in the collected image information by the on-site maintenance terminal includes: After receiving the annotation information, the on-site maintenance terminal uses a time-space synchronization algorithm to match the annotation information with the collected image information to ensure that the annotation information is consistent with the actual position of the device when displayed; The on-site maintenance terminal obtains the maintenance personnel's perspective and operation progress, and dynamically updates the content and orientation of the annotation information based on the perspective and the operation progress, so that the annotation information is always consistent with the maintenance personnel's current operation and perspective; Based on the lighting conditions of the maintenance personnel's field of view, the brightness and contrast of the annotation information are automatically adjusted when the annotation information is displayed.

7. The method for remote collaborative maintenance of power equipment based on augmented reality driving according to claim 6 is characterized in that: The field maintenance terminal obtains the maintenance personnel's perspective and operation progress, and dynamically updates the content and orientation of the annotation information based on the perspective and the operation progress, so that the annotation information is always consistent with the maintenance personnel's current operation and perspective, including: Collecting head posture data using a gyroscope and an accelerometer built into the augmented reality glasses, wherein the head posture data includes pitch angle, yaw angle, and roll angle; Collecting gaze direction data through an eye tracking sensor built into the augmented reality glasses, wherein the gaze direction data includes a gaze direction vector and a focal distance; Determining the current viewing angle information of the maintenance personnel according to the head posture data and the sight direction data, wherein the viewing angle information includes viewing direction, viewing angle, and viewing range; Adjust the display orientation of the annotation information according to the current viewing angle information to ensure that the annotation information is always within the field of view of the maintenance personnel and is consistent with the actual position and angle of the equipment; Combined with the maintenance personnel's operation progress, the display content of the annotation information is dynamically updated to provide guidance information corresponding to the current operation stage.

8. The method for remote collaborative maintenance of power equipment based on augmented reality driving according to claim 1, characterized in that: The image information of the power equipment is collected by the augmented reality glasses at the on-site maintenance end, and further includes: Collecting three-dimensional spatial data of power equipment through the augmented reality glasses at the on-site maintenance end; Match and fuse the collected 3D spatial data with the pre-built 3D model of power equipment; On the display interface of the augmented reality glasses at the on-site maintenance end, the matched and fused three-dimensional model and the collected image information are superimposed and displayed.

9. The method for remote collaborative maintenance of power equipment based on augmented reality driving according to claim 8, characterized in that: Match and fuse the collected 3D spatial data with the pre-built 3D model of the power equipment, including: Extract feature points from the collected three-dimensional spatial data; Using a feature matching algorithm, the extracted feature points are matched with corresponding feature points in a pre-built three-dimensional model of the power equipment to determine the spatial correspondence between the three-dimensional spatial data and the three-dimensional model of the power equipment; According to the matching spatial correspondence, the three-dimensional spatial data is spatially aligned with the three-dimensional model of the power equipment to maintain consistency in position and posture.

10. A remote collaborative maintenance system for power equipment based on augmented reality, characterized in that: include: An on-site maintenance terminal and an expert remote terminal, wherein the on-site maintenance terminal is provided with augmented reality glasses; The augmented reality glasses at the on-site maintenance end collect image information of the power equipment and transmit the image information to the expert remote end; The expert remote terminal receives the expert's annotation information on the image information and feeds the annotation information back to the on-site maintenance terminal; The on-site maintenance terminal displays the received annotation information at a corresponding position in the collected image information.

Images