GIS three-dimensional model and equipment interaction method and system based on mixed reality

By using a mixed reality technology GIS three-dimensional model and equipment interaction method in the maintenance of GIS equipment of pumped storage power stations, the complexity of data integration and real-time update is solved, the maintenance efficiency and safety are improved, and the operation difficulty is reduced, and the operation is reduced. Remote guidance and collaboration improve user experience.

CN119088213APending Publication Date: 2024-12-06BEIJING ZHIZHENYUN INTELLIGENT TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411096318.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-09
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

In the maintenance of GIS equipment in pumped storage power stations, the existing technology has problems such as data integration complexity, real-time update accuracy challenges, user acceptance and training needs, which has affected the widespread application of mixed reality technology in the industrial field.

Method used

Using a GIS three-dimensional model based on mixed reality interaction method, the three-dimensional model is obtained through MR equipment, the location matching between physical equipment and virtual equipment is simulated, and the user operations are captured using gesture recognition technology, data display and security warning are provided, and maintenance is constructed. Knowledge graph that supports remote guidance and collaboration.

Benefits of technology

It improves maintenance efficiency and accuracy, reduces the cognitive burden of operators, enhances safety and real-time, supports remote guidance, improves user experience and universality, and optimizes the equipment maintenance process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119088213A_ABST
    Figure CN119088213A_ABST
Patent Text Reader

Abstract

The invention discloses a GIS three-dimensional model and device interaction method and system based on mixed reality, and the method comprises the steps: obtaining a three-dimensional model of a specified GIS device, and carrying out the overlapping matching of a physical device corresponding to the specified GIS device and a virtual device position in the three-dimensional model corresponding to the specified GIS device; simulation is carried out on the maintenance process of the specified GIS equipment through the MR equipment, and maintenance simulation animation demonstration is carried out on virtual equipment in a virtual environment in the simulation process; the method comprises the following steps: selecting GIS equipment staring by a user through a staring function of MR equipment, capturing a hand image of the user, identifying a hand position, a gesture and an action of the user according to the captured hand image of the user, and identifying a gesture operation instruction of the user by utilizing the hand position, the gesture and the action of the user; and triggering a data display event of the GIS device staring by the user according to the user gesture operation instruction. The maintenance efficiency and accuracy of the gas insulated switchgear in the pumped storage power station are improved by using the mixed reality and gesture recognition technology.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of device interaction, and in particular relates to a method and system for interaction between a GIS three-dimensional model and a device based on mixed reality. Background Art

[0002] At present, mixed reality (MR) technology has shown a lot of potential in the application of 3D models and equipment interaction of pumped storage power station GIS equipment. However, there are still some technical defects and challenges:

[0003] First, the complexity of technical integration: Integrating multi-source data (such as BIM, GIS, and IoT data) into a unified MR platform is a complex project that requires solving technical issues such as data format compatibility and real-time updates. Second, precision and accuracy: Although laser scanning and photogrammetry technologies can provide high-precision 3D models, there are still challenges in the accuracy of real-time data updates and equipment status feedback, especially in high-frequency real-time monitoring scenarios. Third, user acceptance and training needs: The use of new technologies requires user adaptation and training, especially for non-technical personnel. There may be a learning curve, which may affect the widespread application and effectiveness of the technology. These defects and challenges show that although mixed reality has significant potential in the application of complex equipment such as pumped-storage power stations, many technical and operational obstacles still need to be overcome during implementation to achieve its widespread application and sustainable development in the industrial field. Summary of the invention

[0004] To this end, the present invention provides a GIS three-dimensional model and equipment interaction method and system based on mixed reality, so as to improve the maintenance efficiency and accuracy of gas insulated switchgear (GIS) in pumped storage power stations by utilizing mixed reality (MR) and gesture recognition technology.

[0005] In order to achieve the above object, the present invention provides the following technical solution: A method for interacting a GIS three-dimensional model with a device based on mixed reality, comprising:

[0006] Obtaining a three-dimensional model of a specified GIS device, and aligning a physical device corresponding to the specified GIS device with a virtual device position in the three-dimensional model corresponding to the specified GIS device;

[0007] The maintenance process of the designated GIS equipment is simulated by MR equipment. The maintenance simulation animation of the virtual equipment is demonstrated in a virtual environment. The demonstration objects of the maintenance simulation animation include the maintenance environment, maintenance process, distribution of on-site dangerous points, and concrete operation standards.

[0008] Selecting a GIS device that the user is gazing at through the gaze function of the MR device, capturing a hand image of the user, recognizing a hand position, posture, and motion of the user according to the captured hand image of the user, and recognizing a gesture operation instruction of the user using the hand position, posture, and motion of the user;

[0009] A data display event of the GIS device that the user is gazing at is triggered according to the user's gesture operation instruction, and the display object of the data display event includes the device name number, manufacturer, instruction manual, and operation and maintenance manual corresponding to the GIS device that the user is gazing at.

[0010] As a preferred solution for the interaction method between GIS 3D model and equipment based on mixed reality, it also includes:

[0011] Construct a maintenance knowledge graph, and retrieve the maintenance history and equipment information of the specified GIS equipment through the maintenance knowledge graph; the maintenance history includes maintenance time, maintenance personnel, and maintenance subject information;

[0012] The retrieval method of the maintenance knowledge graph includes keyword retrieval or voice retrieval.

[0013] As a preferred solution for the interaction method between GIS 3D model and equipment based on mixed reality, it also includes:

[0014] Acquire image information of the maintenance operation site through an external camera, capture abnormal behavior of maintenance personnel through the image information of the maintenance operation site, and issue a safety warning when the maintenance personnel are caught having preset abnormal behavior;

[0015] The preset abnormal behaviors include whether the person is wearing a helmet, whether the person has fallen to the ground, whether the person is smoking, whether the person is wearing gloves, and whether the person has exceeded the safety area.

[0016] As a preferred solution for the GIS three-dimensional model and device interaction method based on mixed reality, in the process of identifying the user's hand position, posture and action according to the captured user hand image:

[0017] Determine whether the thumb tips of the two hands of the user are in contact, and determine whether the index finger tips of the two hands of the user are in contact;

[0018] If the thumb tips of the user's two hands are in contact, and the index finger tips of the user's two hands are in contact, obtain the current time;

[0019] Determine whether the difference between the current time and the last gesture detection time is greater than a preset time threshold. If the difference between the current time and the last gesture detection time is greater than the preset time threshold, update the last gesture detection time and trigger the function of the last gesture.

[0020] As a preferred solution for the GIS three-dimensional model and device interaction method based on mixed reality, in the process of identifying the user's hand position, posture and action according to the captured user hand image:

[0021] Determine whether the pinkies of the two hands of the user are in contact, obtain the distance between the pinkies of the two hands, and if the distance between the pinkies of the two hands is less than the preset fingertip contact distance threshold, obtain the current time;

[0022] Determine whether the difference between the current time and the last gesture detection time is greater than a preset time threshold. If the difference between the current time and the last gesture detection time is greater than the preset time threshold, update the last gesture detection time and trigger the function of the last gesture.

[0023] The present invention also provides a GIS three-dimensional model and equipment interaction system based on mixed reality, comprising:

[0024] A physical-virtual device position matching module is used to obtain a three-dimensional model of a specified GIS device, and to match the physical device corresponding to the specified GIS device with the virtual device position in the three-dimensional model corresponding to the specified GIS device;

[0025] The maintenance process simulation module is used to simulate the maintenance process of the specified GIS equipment through the MR device. The simulation process performs maintenance simulation animation demonstration on the virtual equipment in the virtual environment. The demonstration objects of the maintenance simulation animation include the maintenance environment, maintenance process, on-site dangerous point distribution, and concrete operation standards.

[0026] A gesture operation instruction recognition module is used to select a GIS device that the user is gazing at through the gaze function of the MR device, capture the user's hand image, recognize the user's hand position, posture and movement based on the captured user's hand image, and recognize the user's gesture operation instruction using the user's hand position, posture and movement;

[0027] The data display module is used to trigger a data display event of the GIS device that the user is gazing at according to the user's gesture operation instruction. The display object of the data display event includes the device name number, manufacturer, instruction manual, and operation and maintenance manual corresponding to the GIS device that the user is gazing at.

[0028] As the preferred solution for the GIS 3D model and equipment interaction system based on mixed reality, it also includes:

[0029] A maintenance knowledge graph module is used to construct a maintenance knowledge graph, and retrieve the maintenance history and equipment information of the specified GIS equipment through the maintenance knowledge graph; the maintenance history includes maintenance time, maintenance personnel, and maintenance subject information;

[0030] The retrieval method of the maintenance knowledge graph includes keyword retrieval or voice retrieval.

[0031] As the preferred solution for the GIS 3D model and equipment interaction system based on mixed reality, it also includes:

[0032] The abnormal behavior warning module is used to obtain image information of the maintenance operation site through an external camera, capture abnormal behavior of maintenance personnel through the image information of the maintenance operation site, and issue a safety warning when the maintenance personnel are caught with preset abnormal behavior;

[0033] The preset abnormal behaviors include whether the person is wearing a helmet, whether the person has fallen to the ground, whether the person is smoking, whether the person is wearing gloves, and whether the person has exceeded the safety area.

[0034] As a preferred solution for the GIS three-dimensional model and device interaction system based on mixed reality, in the gesture operation instruction recognition module:

[0035] Determine whether the thumb tips of the two hands of the user are in contact, and determine whether the index finger tips of the two hands of the user are in contact;

[0036] If the tips of the thumbs of the user's two hands are in contact, and the tips of the index fingers of the user's two hands are in contact, obtain the current time;

[0037] Determine whether the difference between the current time and the last gesture detection time is greater than a preset time threshold. If the difference between the current time and the last gesture detection time is greater than the preset time threshold, update the last gesture detection time and trigger the function of the last gesture.

[0038] As a preferred solution for the GIS three-dimensional model and device interaction system based on mixed reality, in the gesture operation instruction recognition module:

[0039] Determine whether the pinkies of the two hands of the user are in contact, obtain the distance between the pinkies of the two hands, and if the distance between the pinkies of the two hands is less than the preset fingertip contact distance threshold, obtain the current time;

[0040] Determine whether the difference between the current time and the last gesture detection time is greater than a preset time threshold. If the difference between the current time and the last gesture detection time is greater than the preset time threshold, update the last gesture detection time and trigger the function of the last gesture.

[0041] The beneficial effects of the present invention are as follows:

[0042] (1) Improve operational efficiency and accuracy: Mixed reality technology can overlay virtual information onto real scenes and achieve intuitive operation and information interaction through gesture recognition. Operators can obtain equipment status, start maintenance procedures, or record key data with simple gestures, greatly improving operational efficiency and accuracy.

[0043] (2) Reduce the cognitive burden on operators: Traditional equipment maintenance may require cumbersome operating steps and a large number of equipment manual references. Through mixed reality gesture recognition technology, operators do not need to frequently switch their sight lines or consult manuals. They can complete complex operations directly on site with simple gestures, thereby reducing cognitive burden.

[0044] (3) Enhanced safety and real-time performance: The gesture recognition system can obtain equipment status data in real time and display it in real time with virtual information. Operators can immediately know the operating status and possible risks of the equipment. This real-time feedback mechanism helps to detect potential safety hazards in advance and ensure operational safety.

[0045] (4) Support for remote guidance and collaboration: Mixed reality technology combined with gesture recognition enables remote experts to guide on-site operators to perform equipment maintenance through real-time video streaming and gesture interaction. This remote collaboration capability greatly enhances the work efficiency and response capabilities of the pumped storage power station equipment maintenance team.

[0046] (5) Improve user experience and universality of operation: Compared with traditional operation interfaces and command input methods, gesture recognition combined with mixed reality technology is not restricted by language or cultural differences, making operation more intuitive and universal. This improvement in user experience helps operators adapt to new technologies and workflows more quickly.

[0047] It can be seen that the gesture recognition technology based on mixed reality in the auxiliary maintenance application of GIS equipment in pumped storage power stations not only improves the operational efficiency and safety, but also optimizes the management and control of the equipment maintenance process. It is an innovative technical solution with significant application prospects and technical effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the drawings required for the implementation methods or the prior art descriptions are briefly introduced below. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0049] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0050] Figure 1A flow chart of a method for interacting a GIS three-dimensional model with a device based on mixed reality provided in an embodiment of the present invention;

[0051] Figure 2 An application interface 1 of a GIS three-dimensional model and device interaction method based on mixed reality provided in an embodiment of the present invention;

[0052] Figure 3 The second application interface of the method for interacting with a GIS three-dimensional model and a device based on mixed reality provided by an embodiment of the present invention;

[0053] Figure 4 The third application interface of the method for interacting with a GIS three-dimensional model and a device based on mixed reality provided by an embodiment of the present invention;

[0054] Figure 5 The fourth application interface of the method for interacting with a GIS three-dimensional model and a device based on mixed reality provided by an embodiment of the present invention;

[0055] Figure 6 A simulation preview interface for a GIS three-dimensional model and device interaction method based on mixed reality provided in an embodiment of the present invention;

[0056] Figure 7 A schematic diagram of the architecture of a GIS three-dimensional model and device interaction device based on mixed reality provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] As we all know, at present, laser scanning and high-resolution photogrammetry technology can be used to obtain detailed 3D data of pumped storage power stations and generate accurate 3D models. 3D visualization technology uses game engines such as Unity or Unreal Engine to build and render high-quality 3D models to provide an immersive visual experience. Head-mounted displays: Devices such as Microsoft's HoloLens and Magic Leap allow users to view and operate virtual 3D models in real environments. Through cameras and sensors (such as Leap Motion and Kinect), user gestures are recognized to interact naturally with 3D models. Voice control: Voice command control of devices and models is achieved using voice recognition technology (such as Microsoft's Azure Cognitive Services). Touch screen interaction: Direct operation and control on tablets and touch screens.

[0059] As we all know, the Internet of Things (IoT) integration technology integrates IoT sensor data with the GIS system to achieve real-time monitoring and data feedback of pumped storage power station equipment. Digital twin technology creates a digital twin of the power station to reflect the status of the actual equipment in real time, perform status prediction and fault diagnosis.

[0060] The definitions of the terms and abbreviations involved in the following embodiments are as follows:

[0061] Mixed Reality (MR): Mixed reality is a combination of augmented reality (AR) and virtual reality (VR), which allows users to interact with digital objects in both the real and virtual worlds.

[0062] Pumped Storage Hydropower Plant: A pumped storage hydropower plant is a facility that uses electricity during off-peak hours to pump water to a higher location for storage, and then releases the water to generate electricity during peak hours.

[0063] GIS (Gas-Insulated Switchgear): Gas-insulated switchgear is a switchgear that uses gas as the insulating medium and is widely used in high-voltage power systems.

[0064] 3D Model: A 3D model is a mathematical representation of the appearance and structure of an object in three-dimensional space.

[0065] Device Interaction Technology: refers to the technical means for information exchange and interaction between different devices, which may include physical connection, network communication and other methods.

[0066] On the basis of the above-mentioned related technologies, in order to improve the maintenance efficiency and accuracy of gas insulated switchgear (GIS) in pumped storage power stations. By using MR head-mounted devices and gesture recognition systems, maintenance personnel can perform equipment inspection, fault diagnosis and maintenance operations in an environment where virtual and reality are integrated. The embodiment of the present invention provides a GIS three-dimensional model and equipment interaction method and system based on mixed reality. The following is the specific content of the embodiment of the present invention.

[0067] Example 1

[0068] See also Figure 1 Embodiment 1 of the present invention provides a method for interacting a GIS three-dimensional model with a device based on mixed reality, comprising the following steps:

[0069] S1. Obtain a three-dimensional model of a specified GIS device, and match the physical device corresponding to the specified GIS device with the virtual device position in the three-dimensional model corresponding to the specified GIS device;

[0070] S2. The maintenance process of the designated GIS equipment is simulated by MR equipment. The maintenance simulation animation of the virtual equipment is demonstrated in a virtual environment. The demonstration objects of the maintenance simulation animation include the maintenance environment, maintenance process, distribution of on-site dangerous points, and concrete operation standards.

[0071] S3, selecting a GIS device that the user is gazing at through the gaze function of the MR device, capturing a hand image of the user, identifying a hand position, posture and motion of the user according to the captured hand image of the user, and identifying a gesture operation instruction of the user using the hand position, posture and motion of the user;

[0072] S4. Trigger a data display event of the GIS device that the user is gazing at according to the user's gesture operation instruction, wherein the display object of the data display event includes the device name number, manufacturer, instruction manual, and operation and maintenance manual corresponding to the GIS device that the user is gazing at.

[0073] In this embodiment, in step S1, the GIS device model is displayed in three dimensions, including text introduction and voice broadcast, and the position of the GIS device can be fine-tuned to completely overlap the virtual device with the physical device. The spatial positioning function of the MR device is used to completely overlap the virtual scene with the physical device.

[0074] In this embodiment, in step S2, a highly simulated simulation of the maintenance process of the GIS equipment in the pumped-storage power station is implemented through the MR device, and an animation demonstration of the three-dimensional model of the GIS equipment is performed in a virtual environment, so that the maintenance environment, maintenance procedures, distribution of on-site dangerous points, concrete operating standards, etc. are familiarized in advance, and the equipment structure, process principles and maintenance procedures are deeply understood from multiple angles.

[0075] In this embodiment, in step S3, the gesture recognition algorithm Hololens is used to capture the user's hand movements and postures through the equipped depth camera and sensor, and obtain the position, posture and movement of the user's hand in real time. The sensor can capture the user's hand movements, including gesture recognition of one hand or both hands. Hololens can recognize the user's gestures in a ready state (such as the finger up state of Airtap) or a click state (i.e. the finger pressed state of Air tap), while ignoring other gestures. When the user's hand approaches the boundary of the recognizable range, Hololens will inform the direction of the out-of-bounds so that the developer can show the user how to put the hand back into the range that Hololens can detect. When the user makes gestures in the air, the depth camera and sensor can capture these movements and convert them into corresponding instructions. In addition, Hololens also uses machine learning and artificial intelligence technology to continuously optimize and improve the accuracy and sensitivity of gesture interaction by recognizing and analyzing user gestures, so that users can use gestures more naturally to operate augmented reality interfaces and content.

[0076] In a possible embodiment, in the process of identifying the user's hand position, posture, and movement according to the captured user's hand image:

[0077] Determine whether the thumbs of the user's two hands are touching, and determine whether the index fingers of the user's two hands are touching; if the thumbs of the user's two hands are touching, and the index fingers of the user's two hands are touching, get the current time; determine whether the difference between the current time and the last gesture detection time is greater than the preset time threshold, if the difference between the current time and the last gesture detection time is greater than the preset time threshold, update the last gesture detection time and trigger the last gesture function. The implementation code is as follows:

[0078] “aggregator.TryGetJoint(TrackedHandJoint.IndexTip,XRNode.RightHand,out HandJointPose rightIndexPose) gets the world coordinates and rotation angle of a specific joint according to the function.

[0079] Determine whether the index fingers of the two hands are in contact with each other and whether the thumbs are in contact with each other

[0080]

[0081]

[0082] In a possible embodiment, in the process of identifying the user's hand position, posture, and movement according to the captured user's hand image:

[0083] Determine whether the pinkies of the two hands of the user are in contact, obtain the distance between the pinkies of the two hands, and if the distance between the pinkies of the two hands is less than the preset fingertip contact distance threshold, obtain the current time; determine whether the difference between the current time and the last gesture detection time is greater than the preset time threshold, and if the difference between the current time and the last gesture detection time is greater than the preset time threshold, update the last gesture detection time and trigger the last gesture function. The implementation code is as follows:

[0084]

[0085]

[0086] In this embodiment, in step S4, the three-dimensional model corresponding to the corresponding GIS device is selected through the gaze function of the MR device, and then the event is triggered by gestures to display data such as the device name and number, manufacturer, instruction manual, and operation and maintenance manual.

[0087] In a possible embodiment, it also includes: constructing a maintenance knowledge graph, and retrieving the maintenance history and equipment information of the specified GIS equipment through the maintenance knowledge graph; the maintenance history includes maintenance time, maintenance personnel, and maintenance subject information; the retrieval method of the maintenance knowledge graph includes keyword retrieval or voice retrieval.

[0088] In a possible embodiment, it also includes: acquiring image information of the maintenance work site through an external camera, capturing abnormal behavior of maintenance personnel through the image information of the maintenance work site, and issuing a safety warning when the maintenance personnel are captured to have preset abnormal behavior; the preset abnormal behaviors include whether the maintenance personnel are wearing a safety helmet, whether they fall to the ground, whether they are smoking, whether they are wearing gloves, and whether they are beyond the safety area.

[0089] In a possible embodiment, image information of the work site is obtained through an external camera, and based on the Yolov4 training model, the image information of the work site is processed through Opencv, and abnormal behavior of on-site workers is identified to ensure the safety of the workers at work.

[0090] Among them, YOLOv4 is based on a deep convolutional neural network and can detect objects in images in real time with high accuracy. It uses a single neural network, inputs an image and outputs bounding boxes and category probabilities for all objects. In the task of object detection, data labeling is an important link. LabelImg is an open source image labeling tool that is widely used to generate labeled data in YOLO format. This tutorial will detail how to use LabelImg for labeling and export data formats suitable for YOLO object detection. In the field of deep learning, YOLOv4-tiny is an efficient object detection model. By training your own dataset based on YOLOv4-tiny on Colab, you can train the model specifically for your specific target, thereby improving detection performance. Among them, image information is obtained through Hikvision cameras, and the image is processed for image recognition through Opencv using the trained model in Yolov4.

[0091] An application practice process of an embodiment of the present invention is as follows:

[0092] 1. System Preparation

[0093] Equipment debugging and installation: Installation and calibration of MR head-mounted equipment and gesture recognition devices.

[0094] GIS equipment data integration: Integrate the status data, fault information, etc. of GIS equipment into the MR system to ensure real-time data synchronization.

[0095] 2. Build a virtual model

[0096] 3D modeling: Create high-precision 3D models of GIS equipment and import them into MR applications. Virtual annotation: Add key component annotations, maintenance instructions, and operation guides to the virtual model.

[0097] 3. Gesture recognition configuration

[0098] Gesture definition: Define commonly used gesture commands, such as selection, zoom in / out, rotation, start / stop operation, etc.

[0099] 4. Develop MR applications

[0100] Interface design: Design a user-friendly MR interface to ensure that maintenance personnel can view and operate intuitively.

[0101] Function realization: realize equipment inspection, fault diagnosis, maintenance guidance, data recording and other functions.

[0102] Multi-mode interaction: supports multiple interaction methods such as gestures, voice, and sight to improve operational convenience.

[0103] Among them, the system based on MR technology integrates the three-dimensional model of the GIS equipment of the pumped-storage power station with the equipment operation information to achieve real-time monitoring, fault diagnosis and maintenance guidance of the equipment.

[0104] Establishment of 3D virtual model of equipment: Use mixed reality technology to accurately present the appearance, structure and components of the equipment and provide detailed geometric information.

[0105] Interactivity and visualization of models: Through mixed reality technology, operators can interact with virtual models and operate them through head-mounted display devices or other interactive devices. They can rotate, scale and move virtual models to view various angles and details of the equipment. At the same time, virtual models can be superimposed on actual equipment through augmented reality technology to achieve perspective and analysis of the equipment.

[0106] Multiple display functions of virtual models: Mixed reality technology can provide multiple display functions, such as model slicing, transparency, animation demonstration, etc. Through these functions, operators can gain an in-depth understanding of the internal structure and working principle of the equipment, identify potential fault points, and formulate corresponding maintenance plans.

[0107] Real-time update and management of models: The status and configuration of equipment may change, so virtual models need to be updated and managed in real time. Through mixed reality technology, the real-time data and monitoring information of equipment can be associated with the virtual model, and the status and parameters of the model can be updated in time to keep it consistent with the actual equipment.

[0108] By building and managing virtual models, the 3D assisted maintenance technology of pumped storage power station GIS equipment based on mixed reality technology can provide operators with more intuitive and comprehensive equipment information, helping them better understand the structure and working principle of the equipment, accurately locate problems, and develop precise maintenance plans. This will improve the efficiency and accuracy of equipment maintenance and repair, reduce operational risks, and ultimately ensure the normal operation of the equipment.

[0109] 5. Functional module design

[0110] 3D model browsing: Users can view the 3D model of power plant equipment, including detailed structure and layout, through MR equipment.

[0111] 6. Deployment and Training

[0112] System deployment: Deploy the MR system to the maintenance site and integrate it with the existing maintenance process.

[0113] Personnel training: Train maintenance personnel to familiarize them with the operation methods and precautions of the MR system.

[0114] 7. Field Application

[0115] Real-time monitoring: The MR system can be used to monitor the status of GIS equipment in real time and detect potential problems in a timely manner.

[0116] Fault diagnosis and maintenance: Use virtual display maintenance guidance and gesture interaction to perform fault diagnosis and maintenance operations.

[0117] Data recording and analysis: Automatically record the data of the maintenance process to facilitate subsequent analysis and optimize the maintenance strategy.

[0118] Among them, the simulation rehearsal of the project was completed according to the GIS equipment maintenance operation instruction. This operation instruction is applicable to the replacement, cleaning and maintenance of the arc extinguishing chamber of the Beijing Ming Tombs Energy Storage Power Plant of State Grid New Energy Holding Co., Ltd. The operation instruction is compiled based on (State Grid New Energy Holding Co., Ltd. Beijing Ming Tombs Energy Storage Power Plant Enterprise Standard Q / GDW4601-001-2018-12215 High Voltage GIS Equipment Maintenance Procedure). Figure 2 Edit the maintenance work instructions into UI window format. Figure 3 As the basic working conditions, Figure 4 Disassembly and maintenance process of circuit breaker, Figure 5 The maintenance process of the circuit breaker operating mechanism is as follows. According to the maintenance work instructions, the equipment model maintenance animation is produced, and the maintenance of the equipment is simulated and previewed, such as Figure 6 shown.

[0119] In summary, the present invention obtains a three-dimensional model of a designated GIS device, and matches the physical device corresponding to the designated GIS device with the virtual device position in the three-dimensional model corresponding to the designated GIS device; simulates the maintenance process of the designated GIS device through an MR device, and the simulation process performs a maintenance simulation animation demonstration of the virtual device in a virtual environment, and the demonstration objects of the maintenance simulation animation include the maintenance environment, maintenance procedures, on-site dangerous point distribution, and concrete operation standards; selects a GIS device that the user is gazing at through the gaze function of the MR device, captures the user's hand image, identifies the user's hand position, posture and movement based on the captured user hand image, and identifies the user's gesture operation instructions using the user's hand position, posture and movement; triggers a data display event of the GIS device that the user is gazing at according to the user's gesture operation instructions, and the display objects of the data display event include the device name number, manufacturer, instruction manual, and operation and maintenance manual corresponding to the GIS device that the user is gazing at. The present invention realizes automatic analysis and prediction of fault data, reduces maintenance preparation time, and improves maintenance work efficiency; through the research on GIS equipment on-site operation simulation and rehearsal technology, a GIS equipment maintenance on-site operation simulation and rehearsal method is proposed, which simulates the maintenance and maintenance operation process in real time, improves maintenance work efficiency, improves the safety level of the maintenance operation process, and strengthens the professional ability of maintenance personnel. Through the research on abnormal behavior intelligent identification technology at the maintenance operation site, a visual early warning method for the entire maintenance operation process is proposed to reduce operation risks and improve the efficiency and completion effect of the maintenance operation process.

[0120] It should be noted that the method of the embodiment of the present disclosure can be performed by a single device, such as a computer or a server. The method of the present embodiment can also be applied in a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present disclosure, and the multiple devices will interact with each other to complete the described method.

[0121] It should be noted that the above describes some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0122] Example 2

[0123] See also Figure 7 Embodiment 2 of the present invention provides a GIS three-dimensional model and device interaction system based on mixed reality, including:

[0124] The physical-virtual device position matching module 100 is used to obtain a three-dimensional model of a specified GIS device, and to match the physical device corresponding to the specified GIS device with the virtual device position in the three-dimensional model corresponding to the specified GIS device;

[0125] The maintenance process simulation module 200 is used to simulate the maintenance process of the designated GIS equipment through the MR device. The simulation process performs maintenance simulation animation demonstration on the virtual equipment in the virtual environment. The demonstration objects of the maintenance simulation animation include the maintenance environment, maintenance process, on-site dangerous point distribution, and concrete operation standards.

[0126] The gesture operation instruction recognition module 300 is used to select the GIS device that the user is gazing at through the gaze function of the MR device, capture the user's hand image, recognize the user's hand position, posture and movement according to the captured user's hand image, and recognize the user's gesture operation instruction by using the user's hand position, posture and movement;

[0127] The data display module 400 is used to trigger a data display event of the GIS device that the user is gazing at according to the user's gesture operation instruction. The display object of the data display event includes the device name number, manufacturer, instruction manual, and operation and maintenance manual corresponding to the GIS device that the user is gazing at.

[0128] In this embodiment, it also includes:

[0129] The maintenance knowledge graph module 500 is used to construct a maintenance knowledge graph, and retrieve the maintenance history and equipment information of the specified GIS equipment through the maintenance knowledge graph; the maintenance history includes maintenance time, maintenance personnel, and maintenance subject information;

[0130] The retrieval method of the maintenance knowledge graph includes keyword retrieval or voice retrieval.

[0131] In this embodiment, it also includes:

[0132] The abnormal behavior warning module 600 is used to obtain image information of the maintenance operation site through an external camera, capture abnormal behavior of maintenance personnel through the image information of the maintenance operation site, and issue a safety warning when the maintenance personnel are caught with preset abnormal behavior;

[0133] The preset abnormal behaviors include whether the person is wearing a helmet, whether the person has fallen to the ground, whether the person is smoking, whether the person is wearing gloves, and whether the person has exceeded the safety area.

[0134] In this embodiment, in the gesture operation instruction recognition module 300:

[0135] Determine whether the thumb tips of the two hands of the user are in contact, and determine whether the index finger tips of the two hands of the user are in contact;

[0136] If the thumb tips of the user's two hands are in contact, and the index finger tips of the user's two hands are in contact, obtain the current time;

[0137] Determine whether the difference between the current time and the last gesture detection time is greater than a preset time threshold. If the difference between the current time and the last gesture detection time is greater than the preset time threshold, update the last gesture detection time and trigger the function of the last gesture.

[0138] In this embodiment, in the gesture operation instruction recognition module 300:

[0139] Determine whether the pinkies of the two hands of the user are in contact, obtain the distance between the pinkies of the two hands, and if the distance between the pinkies of the two hands is less than the preset fingertip contact distance threshold, obtain the current time;

[0140] Determine whether the difference between the current time and the last gesture detection time is greater than a preset time threshold. If the difference between the current time and the last gesture detection time is greater than the preset time threshold, update the last gesture detection time and trigger the function of the last gesture.

[0141] It should be noted that the information interaction, execution process and other contents between the modules of the above-mentioned system are based on the same concept as the method embodiment in Example 1 of the present application, and the technical effects they bring are the same as those of the method embodiment of the present application. For specific contents, please refer to the description in the method embodiment shown in the previous part of the present application, and will not be repeated here.

[0142] Example 3

[0143] Embodiment 3 of the present invention provides a non-transitory computer-readable storage medium, in which a program code for a method for interaction between a GIS three-dimensional model and a device based on mixed reality is stored. The program code includes instructions for executing the method for interaction between a GIS three-dimensional model and a device based on mixed reality of embodiment 1 or any possible implementation thereof.

[0144] The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0145] Example 4

[0146] Embodiment 4 of the present invention provides an electronic device, including: a memory and a processor;

[0147] The processor and the memory communicate with each other via a bus; the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the mixed reality-based GIS three-dimensional model and device interaction method of Example 1 or any possible implementation thereof.

[0148] Specifically, the processor can be implemented by hardware or by software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc.; when implemented by software, the processor can be a general-purpose processor implemented by reading software codes stored in a memory. The memory can be integrated in the processor or can be located outside the processor and exist independently.

[0149] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital consumer customer line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center.

[0150] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0151] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. GIS 3D model and device interaction method based on mixed reality, It is characterized in that include: Obtaining a three-dimensional model of a specified GIS device, and aligning a physical device corresponding to the specified GIS device with a virtual device position in the three-dimensional model corresponding to the specified GIS device; The maintenance process of the designated GIS equipment is simulated by MR equipment. The maintenance simulation animation of the virtual equipment is demonstrated in a virtual environment. The demonstration objects of the maintenance simulation animation include the maintenance environment, maintenance process, distribution of on-site dangerous points, and concrete operation standards. Selecting a GIS device that the user is gazing at through the gaze function of the MR device, capturing a hand image of the user, recognizing a hand position, posture, and motion of the user according to the captured hand image of the user, and recognizing a gesture operation instruction of the user using the hand position, posture, and motion of the user; A data display event of the GIS device that the user is gazing at is triggered according to the user's gesture operation instruction, and the display object of the data display event includes the device name number, manufacturer, instruction manual, and operation and maintenance manual corresponding to the GIS device that the user is gazing at.

2. The method for interacting with a GIS three-dimensional model and a device based on mixed reality according to claim 1, It is characterized in that Also includes: Construct a maintenance knowledge graph, and retrieve the maintenance history and equipment information of the specified GIS equipment through the maintenance knowledge graph; the maintenance history includes maintenance time, maintenance personnel, and maintenance subject information; The retrieval method of the maintenance knowledge graph includes keyword retrieval or voice retrieval.

3. The method for interacting with a GIS three-dimensional model and a device based on mixed reality according to claim 1, It is characterized in that Also includes: Acquire image information of the maintenance operation site through an external camera, capture abnormal behavior of maintenance personnel through the image information of the maintenance operation site, and issue a safety warning when the maintenance personnel are caught having preset abnormal behavior; The preset abnormal behaviors include whether the person is wearing a helmet, whether the person has fallen to the ground, whether the person is smoking, whether the person is wearing gloves, and whether the person has exceeded the safety area.

4. The method for interacting with a GIS three-dimensional model and a device based on mixed reality according to claim 1, It is characterized in that In the process of recognizing the user's hand position, posture and movement based on the captured image of the user's hand: Determine whether the thumb tips of the two hands of the user are in contact, and determine whether the index finger tips of the two hands of the user are in contact; If the tips of the thumbs of the user's two hands are in contact, and the tips of the index fingers of the user's two hands are in contact, obtain the current time; Determine whether the difference between the current time and the last gesture detection time is greater than a preset time threshold. If the difference between the current time and the last gesture detection time is greater than the preset time threshold, update the last gesture detection time and trigger the function of the last gesture.

5. The method for interacting with a GIS three-dimensional model and a device based on mixed reality according to claim 1, It is characterized in that In the process of recognizing the user's hand position, posture and movement based on the captured image of the user's hand: Determine whether the pinkies of the two hands of the user are in contact, obtain the distance between the pinkies of the two hands, and if the distance between the pinkies of the two hands is less than the preset fingertip contact distance threshold, obtain the current time; Determine whether the difference between the current time and the last gesture detection time is greater than a preset time threshold. If the difference between the current time and the last gesture detection time is greater than the preset time threshold, update the last gesture detection time and trigger the function of the last gesture.

6. GIS 3D model and equipment interaction system based on mixed reality, It is characterized in that include: A physical-virtual device position matching module is used to obtain a three-dimensional model of a specified GIS device, and to match the physical device corresponding to the specified GIS device with the virtual device position in the three-dimensional model corresponding to the specified GIS device; The maintenance process simulation module is used to simulate the maintenance process of the specified GIS equipment through the MR device. The simulation process performs maintenance simulation animation demonstration on the virtual equipment in the virtual environment. The demonstration objects of the maintenance simulation animation include the maintenance environment, maintenance process, on-site dangerous point distribution, and concrete operation standards. A gesture operation instruction recognition module is used to select a GIS device that the user is gazing at through the gaze function of the MR device, capture the user's hand image, recognize the user's hand position, posture and movement based on the captured user's hand image, and recognize the user's gesture operation instruction using the user's hand position, posture and movement; The data display module is used to trigger a data display event of the GIS device that the user is gazing at according to the user's gesture operation instruction. The display object of the data display event includes the device name number, manufacturer, instruction manual, and operation and maintenance manual corresponding to the GIS device that the user is gazing at.

7. The GIS three-dimensional model and device interaction system based on mixed reality according to claim 6, It is characterized in that Also includes: A maintenance knowledge graph module is used to construct a maintenance knowledge graph, and retrieve the maintenance history and equipment information of the specified GIS equipment through the maintenance knowledge graph; the maintenance history includes maintenance time, maintenance personnel, and maintenance subject information; The retrieval method of the maintenance knowledge graph includes keyword retrieval or voice retrieval.

8. The GIS three-dimensional model and device interaction system based on mixed reality according to claim 6, It is characterized in that Also includes: The abnormal behavior warning module is used to obtain image information of the maintenance operation site through an external camera, capture abnormal behavior of maintenance personnel through the image information of the maintenance operation site, and issue a safety warning when the maintenance personnel are caught with preset abnormal behavior; The preset abnormal behaviors include whether the person is wearing a helmet, whether the person has fallen to the ground, whether the person is smoking, whether the person is wearing gloves, and whether the person has exceeded the safety area.

9. The GIS three-dimensional model and device interaction system based on mixed reality according to claim 6, It is characterized in that In the gesture operation instruction recognition module: Determine whether the thumb tips of the two hands of the user are in contact, and determine whether the index finger tips of the two hands of the user are in contact; If the tips of the thumbs of the user's two hands are in contact, and the tips of the index fingers of the user's two hands are in contact, obtain the current time; Determine whether the difference between the current time and the last gesture detection time is greater than a preset time threshold. If the difference between the current time and the last gesture detection time is greater than the preset time threshold, update the last gesture detection time and trigger the function of the last gesture.

10. The GIS three-dimensional model and device interaction system based on mixed reality according to claim 6, It is characterized in that In the gesture operation instruction recognition module: Determine whether the pinkies of the two hands of the user are in contact, obtain the distance between the pinkies of the two hands, and if the distance between the pinkies of the two hands is less than the preset fingertip contact distance threshold, obtain the current time; Determine whether the difference between the current time and the last gesture detection time is greater than a preset time threshold. If the difference between the current time and the last gesture detection time is greater than the preset time threshold, update the last gesture detection time and trigger the function of the last gesture.