Intelligent inspection method and system based on AR (Augmented Reality) technology
By applying AR augmented reality technology in power equipment inspection, combining feature matching, semantic segmentation and target tracking algorithms, efficient equipment identification and real-time tracking are achieved, solving the problems of low efficiency and poor accuracy of existing inspection methods, and improving patrol efficiency and safety.
Patent Information
- Application Number
- CN202411560609.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-04
- Publication Date
- 2025-02-25
AI Technical Summary
The existing power equipment inspection methods are prone to fatigue, errors, labor hours and low efficiency. Especially in large substations, manual inspection is low efficiency and difficult to obtain equipment information in real time.
The intelligent patrol method based on AR augmented reality technology is adopted to identify the device identity information and pose information through feature matching algorithms, and the device area pose information is obtained in combination with the semantic segmentation algorithm, and the target tracking algorithm is used for real-time tracking to achieve the integration of virtual information and real scenes.
It improves the accuracy of equipment identification and positioning, realizes real-time tracking and stable display of virtual information, improves patrol efficiency and accuracy, reduces information acquisition time, and enhances the accuracy and safety of on-site operations.
Smart Images

Figure CN119515350A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of electric power operation and maintenance inspection, and in particular relates to an intelligent inspection method and system based on AR augmented reality technology. Background Art
[0002] In order to ensure the stable and reliable operation of power equipment, routine inspection of equipment in the station has become a key technical means to ensure the safe operation of the power grid. The inspection method of "paper + manual verification" has problems such as easy fatigue, easy error, time-consuming, and low efficiency. In recent years, smart wearable devices based on virtual reality (VR) and augmented reality (AR) technology have been applied in the power industry, providing a new solution to this problem. The application of VR technology in the power grid is to use intelligent computing devices to simulate virtual substations, power equipment and other scenes in three-dimensional space, providing users with an immersive interactive experience in vision, hearing and touch. It is mostly used for power operation and maintenance inspections involving high-risk, high-cost or high-consumption situations. AR technology is a technology that superimposes physical information (visual information, sound, taste, touch, etc.) that is difficult to experience within a certain time and space range in the real world, applies this information to the real world, and is recognized by the user's senses. It can obtain an interactive experience beyond reality and provide technical support for a large number of repetitive tasks in power operation and maintenance. Applying augmented reality technology to on-site operation inspections can integrate virtual information with real scenes, helping inspection personnel or maintenance engineers obtain the status, performance data and fault information of equipment in the station in real time, thereby improving inspection efficiency.
[0003] On-site inspection is to check the operation of equipment in order to timely discover and eliminate equipment defects and unsafe factors, prevent accidents, and ensure the normal operation of the power generation system. Inspection usually includes equipment status inspection, cleanliness assessment and performance monitoring, etc., generally through manual inspection or drone inspection, but there are some disadvantages: high labor cost, manual inspection requires a lot of human resources, especially for large substations; low inspection efficiency, manual inspection usually takes a long time, and during the inspection, on-site personnel cannot directly understand some equipment information, and need to return to the archives to inquire or ask other personnel, which reduces work efficiency; missed inspection and false inspection problems, manual inspection may be affected by the subjective judgment of the inspectors, and cannot accurately obtain equipment information. However, research on related technologies such as on-site equipment inspection based on AR has not been systematically carried out. How to combine AR technology to improve the efficiency of on-site equipment inspection has become a new research goal. Summary of the invention
[0004] In order to solve the above technical problems, the present invention provides an intelligent inspection method and system based on augmented reality technology.
[0005] The technical solution provided by the present invention is as follows:
[0006] An intelligent inspection method based on AR augmented reality technology, comprising:
[0007] Step 1: Identify the identity information and preliminary posture information of the acquisition device in the image based on the feature matching algorithm;
[0008] Step 2: Based on the preliminary pose information, use the semantic segmentation algorithm to obtain the regional pose information of the acquisition device;
[0009] Step 3: query the status attribute information of the device according to the identity information of the acquisition device, and project the queried status attribute information to the area where the acquisition device is located for display.
[0010] Furthermore, in step 1, based on the improved AKAZE algorithm, image matching is performed between the acquisition device during inspection and the target device to obtain the identity information and preliminary posture information of the acquisition device, which specifically includes the following steps:
[0011] Use the ORB algorithm to detect the image feature points of the acquisition device and the target device respectively;
[0012] The M-LDB descriptor is used to describe the image feature points of the acquisition device and the target device respectively;
[0013] By comparing the M-LDB descriptors, feature point matching is performed to identify the identity information of the acquisition device.
[0014] Preferably, after matching feature points by comparing M-LDB descriptors, the GMS algorithm is used to remove mismatched points.
[0015] Furthermore, in step 2, the image of the acquisition device is input into the U-Net network for semantic segmentation to obtain the regional posture information of the acquisition device, and the encoding part of the U-Net network is replaced by the DRN network.
[0016] Preferably, edge detection is performed on the image of the acquisition device, and the edge detection result and the fused image of the acquisition device are used as input of the U-Net network.
[0017] Furthermore, in step 3, the target tracking algorithm is used to track the identified acquisition device to obtain the real-time position and posture of the acquisition device, and the position and posture of the projection device are estimated in real time so that the projection information can be accurately superimposed on the area where the acquisition device is located.
[0018] Furthermore, the status attribute information includes the running status, history records and operation suggestions of the equipment.
[0019] An intelligent inspection system based on the above method comprises:
[0020] Intelligent inspection terminal, used to collect on-site information in real time and transmit it to the background management system, and also used to project and superimpose the equipment status attribute information pushed by the background management system to the designated area;
[0021] The backend management system is used to identify, locate and track the equipment in the images uploaded by the intelligent inspection terminal, query the equipment's status attribute information and push it together with the equipment's real-time posture information to the intelligent inspection terminal.
[0022] Furthermore, the intelligent inspection terminal includes an information collection module, a communication module and a virtual information rendering module. The virtual information rendering module integrates the device status attribute information pushed by the background management system with the scene of the inspection site through Unity3D technology.
[0023] Furthermore, the background management system includes a device identification module, a tracking and registration module, and a posture estimation module: the device identification module ensures that the system can accurately identify and locate the equipment in the on-site operation, providing a basis for the accurate rendering of virtual information; the tracking and registration module uses the target tracking algorithm to track the device position in real time to ensure the consistency between the virtual scene and the real scene; the posture estimation module estimates the posture of the device to ensure the consistency of the visual effect between the projected virtual information image or video and the actual device.
[0024] Compared with the prior art, the present invention has at least the following beneficial effects:
[0025] (1) Efficient and accurate feature matching and segmentation: Through the improved AKAZE feature matching and DRN-UNet semantic segmentation algorithm, the present invention can quickly and accurately identify devices in complex environments. This combination greatly improves the accuracy of device identification and positioning, especially in large-scale industrial scenarios.
[0026] (2) Real-time 3D tracking and registration: Using the KCF target tracking algorithm and camera posture estimation, the present invention can continuously track the position and posture of the device during the inspection process, ensuring the stability and accuracy of the virtual information. The inspectors can obtain the real-time status and information of the device at any time without interrupting the inspection work.
[0027] (3) Real-time display and interaction of augmented reality: Through AR technology, inspection personnel can see virtual information of equipment in real scenes, including equipment operating status, fault records, operation guides, etc. This combination of virtual and real display improves inspection efficiency, reduces the time of information acquisition, and enhances the accuracy and safety of on-site operations.
[0028] (4) Real-time storage and analysis of data: The present invention can upload inspection data and equipment status to the background in real time, store and analyze them, and provide strong support for long-term maintenance of equipment, fault analysis and management of inspection records. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.
[0030] Figure 1 It is a schematic diagram of the framework of an intelligent inspection method provided by an embodiment of the present invention;
[0031] Figure 2 It is a schematic diagram of the module combination of the intelligent inspection system provided by one embodiment of the present invention. DETAILED DESCRIPTION
[0032] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, other embodiments obtained by ordinary technicians in this field without making creative work are all within the scope of protection of the present invention.
[0033] Embodiment 1
[0034] This embodiment provides an intelligent inspection method based on augmented reality technology. Figure 1 As shown in the figure, this method uses the improved AKAZE feature matching algorithm, DRN-UNet semantic segmentation algorithm and KCF target tracking algorithm to achieve accurate identification, real-time positioning and three-dimensional tracking of equipment. The equipment image is collected by the intelligent terminal device, and feature point matching and segmentation are performed. The virtual information is accurately superimposed on the real equipment through three-dimensional tracking and camera posture estimation to display the equipment's operating status, history and operation suggestions.
[0035] The intelligent inspection method provided in this embodiment mainly includes the following steps:
[0036] Step 1: Identify the identity information and rough posture information of the acquisition device in the image based on the feature matching algorithm;
[0037] Step 2: Based on the rough pose information, use the semantic segmentation algorithm to obtain the regional pose information of the acquisition device;
[0038] Step 3: query the status attribute information of the device according to the identity information of the acquisition device, and project the queried status attribute information to the area where the acquisition device is located for display.
[0039] In step 1, based on the improved AKAZE algorithm, the device in the image collected during the inspection is matched with the target device image stored in the database to obtain the identity information and rough location information of the collected device.
[0040] Specifically, this step mainly includes the following steps:
[0041] 1) Feature point detection
[0042] In image processing, the AKAZE algorithm is mainly used for image feature point detection and description. Feature points refer to key locations in an image, reflecting the geometry and surface texture of the device. Although the traditional AKAZE algorithm has good performance, the feature point detection speed is slow. Therefore, this embodiment introduces the ORB (Oriented FAST and Rotated BRIEF) algorithm to speed up the detection of feature points. ORB is an improved FAST (Features from Accelerated Segment Test) feature point detection algorithm that can accelerate image processing through fast corner point detection, making it suitable for real-time inspection tasks.
[0043] FAST detects key points by quickly judging pixels on the circumference of the image. The ORB algorithm extracts feature points and locates the direction of the feature points, making the feature points rotationally invariant. Compared with the FAST algorithm, the ORB algorithm is more suitable for practical applications and has stronger robustness and accuracy. In order to speed up the feature point extraction, the feature point detection of the improved algorithm is based on the ORB algorithm. Due to its computational characteristics, this feature detection algorithm is suitable for efficient corner point detection in real-time applications. If there are continuous n (n = 8, 9, 12, 16) pixels with brightness values higher than I on the circumference, the feature point detection algorithm is suitable for efficient corner point detection in real-time applications. P +h is larger, or larger than I P -h is small, the center pixel is called a corner point. The corner point determination formula is as follows:
[0044]
[0045] In the formula, I x Indicates the pixel value of the xth point around point P, I P Represents the pixel value of point P, and h is the set threshold.
[0046] The feature point description of the improved algorithm uses the M-LDB descriptor, which is an improved local binary descriptor used to describe feature points in an image. It is a feature descriptor commonly used in computer vision and image processing. It is derived from the local binary pattern descriptor and is improved on the basis of the LDB descriptor. It aims to capture local texture information in the image. After improvement, it can better capture texture details and enhance the descriptor's ability to distinguish various computer vision tasks (such as object recognition, facial analysis, and texture classification). The M-LDB descriptor does not sample all segmented grids, but samples grid pixels. After sampling, the effect of using the average value to calculate the approximate division of the grid is good. The formulas are as follows:
[0047]
[0048] Where m represents the total number of grid cells i, L avg (i) represents the average intensity of each grid cell, d u (i) represents the lateral gradient value of the grid i area, d v (i) represents the longitudinal gradient value of the grid i area.
[0049] The operator obtained by the M-LDB descriptor is a binary descriptor, which is simple and easy to calculate and can further improve the running speed of the algorithm. Its formula is as follows:
[0050] G i ={L avg (i)d u (i)d v (i)},
[0051]
[0052] 2) Feature point matching and false matching elimination
[0053] By comparing the M-LDB descriptors between different images for feature matching, the target device in the image is identified. The result generated by the M-LDB descriptor is roughly matched using the BF (Brute Force) algorithm to obtain a matching result. In the feature point matching process, the occurrence of mismatches will lead to errors in device recognition. In order to improve the accuracy of matching, most image matching algorithms use the RANSAC algorithm, but when the amount of feature point sample data is large, the matching time will be longer and the effect is not ideal. Therefore, this embodiment uses the GMS (Grid-based Motion Statistics) algorithm to eliminate mismatched points. The GMS algorithm is a method for feature matching screening. The algorithm is based on the local motion statistics of the image and can effectively eliminate mismatched points, which can further improve the accuracy and robustness of the matching results.
[0054] 3) Output of device information after matching
[0055] Once the feature point matching is completed, the status attribute information of the corresponding device stored in the database or server, such as operating status, history, and operation suggestions, can be obtained. In addition, feature point matching provides preliminary position and posture information of the device, which provides important starting conditions for subsequent semantic segmentation. Through the feature matching results, the approximate area of the device can be locked in the image, reducing the image range that needs to be processed by the subsequent segmentation algorithm, thereby improving efficiency and accuracy. Accurate feature point matching results also provide basic data for subsequent three-dimensional tracking and virtual information overlay, ensuring that the device can still be accurately positioned from different perspectives.
[0056] Feature point matching can only provide the approximate location of the device. Accurate identification of the device location requires further regional segmentation. In step 2, based on the U-Net algorithm, the convolutional layer of its network coding part is replaced with a dilated residual network (DRN), providing a DRN-UNet semantic segmentation algorithm that can accurately extract the contour and boundary information of the device, providing high-precision data support for device positioning and status monitoring.
[0057] U-Net is a classic semantic segmentation network that is particularly good at handling pixel-level classification problems in images. It adopts a "U"-shaped symmetrical structure, consisting of a contraction path (for extracting high-level features) and an expansion path (for restoring image resolution). U-Net can fuse low-level information of the image with high-level features through jump connections, thereby improving segmentation accuracy, and is particularly suitable for applications in small sample environments.
[0058] In order to improve the performance of U-Net, this embodiment replaces its encoding part with DRN. This improvement enhances the ability to capture global context information of the image and the scale invariance of the network, which helps to improve the feature extraction and perception capabilities of the model, enabling the model to process device images of different scales. DRN expands the receptive field through dilated convolution, allowing the model to capture a wider range of context information during the segmentation process while maintaining attention to details. Such a structure is particularly suitable for identifying complex boundaries of devices and ensuring accurate segmentation of device contours.
[0059] In some embodiments, in order to further improve the segmentation accuracy, the device image and the edge detection result of the device image are fused as the input of the semantic segmentation network to enhance the network's perception of the edge features of the image, more accurately capture the contour and boundary information of the target, and thus improve the algorithm's segmentation accuracy for the edge of the target. Edge detection is a commonly used technology in image processing. Classic algorithms include the Sobel operator, Prewitt operator, Canny edge detection, etc., which can accurately identify the edges of devices in the image. Through this additional edge information, the model can better capture the contour and shape of the device and reduce the possibility of mis-segmentation.
[0060] The Sobel operator is an edge detection operator based on first-order differentials, with strong anti-noise capability. Its implementation principle is as follows: first calculate the Sobel operator in the x and y directions respectively, then take the absolute value, and then linearly mix. Assuming that there is an original image Is, calculate the approximate value of the horizontal partial derivative: convolve the Sobel operator with the original image Is to calculate the change in pixel value in the horizontal direction. Taking the 3×3 size Sobel operator as an example, the image gradient values Gx and Gy for edge detection in the horizontal and vertical directions are calculated as follows:
[0061]
[0062] The gradient value of each pixel in the image can be obtained by combining the horizontal and vertical gradient values using the following formula:
[0063]
[0064] Often, an approximation without square root is used for efficiency:
[0065] G|=|G x |+|G y |,
[0066] If the gradient G is greater than a certain threshold, the pixel is considered to be an edge point.
[0067] The gradient direction can then be calculated using the following formula:
[0068]
[0069] The Sobel operator detects edges based on the phenomenon that the weighted difference in grayscale between the upper and lower and left and right neighboring points of a pixel reaches an extreme value at the edge. It has a smoothing effect on noise and can obtain more accurate edge information.
[0070] Semantic segmentation relies on the preliminary device location provided by feature matching. The feature matching results narrow the search range for semantic segmentation, ensuring that the segmentation algorithm can focus on device identification in a specific area, thereby improving efficiency. After semantic segmentation processing, the acquisition device is separated from the image background, and the precise outline, location, and shape information of the device are output. This information provides more accurate input for subsequent virtual information overlay.
[0071] In step 3, the inspection personnel use the smart terminal to collect on-site information (images, etc.) and transmit it to the system database. The target device pre-stored in the database and the currently collected device are matched and identified. If the match is successful, the relevant information and data of the device are read from the background, such as operating status, historical inspection records, etc., and superimposed on the device for display using a projection device (camera).
[0072] In actual inspections, the position and posture of the device may change with the movement of the inspector or changes in the environment, so it is necessary to track the posture of the device in real time to ensure that the device status attribute information can be accurately projected and superimposed on the actual area where the device is located. In this embodiment, the KCF (Kernelized Correlation Filters) target tracking algorithm is used to ensure that the position and posture of the device in three-dimensional space remain stable, providing a basis for the superposition of virtual information and augmented reality display.
[0073] The KCF algorithm is a target tracking algorithm based on correlation filtering, which can efficiently track the target through a linear kernel function. Its principle is to find the position of the target in the image through template matching and filter convolution operations. KCF has the advantages of high computational efficiency and stable target tracking, and is suitable for real-time applications. The KCF algorithm combines the feature points and segmentation information provided by the first two steps to more accurately lock the position and posture of the device. Based on the device feature points, the system can adjust the device's tracking frame in real time to ensure that the device is still accurately tracked at different angles and moving states.
[0074] 3D registration is a key step in augmented reality systems, requiring the system to be able to accurately superimpose virtual information (such as equipment fault information, operation prompts, etc.) onto the equipment in the real scene. Through camera posture estimation, the system can calculate the change in the inspection personnel's perspective and adjust the display position of the virtual information to ensure seamless connection between virtual and reality. The output of target tracking provides reliable registration data for the display of virtual information in augmented reality. By tracking the 3D position of the equipment, the system can ensure that the virtual information is always superimposed on the correct position of the equipment, helping the inspection personnel to accurately obtain the equipment status and operation instructions.
[0075] The above is the main content of the intelligent inspection method provided in this embodiment. By using augmented reality technology to combine virtual and reality, the real-time status of the equipment, operation prompts and other information are superimposed on the field of vision of the inspection personnel, so that the inspection personnel can intuitively obtain the operating status, historical data and other information of the equipment, thereby improving the inspection efficiency and accuracy.
[0076] Embodiment 2
[0077] Corresponding to the above method, this embodiment provides an intelligent inspection system based on augmented reality technology. The inspection personnel interact with the background management system in real time by wearing intelligent terminals (such as AR glasses, etc.). The management system pushes the historical operation records, fault information, operation prompts and other data of the equipment to the terminal device. The virtual information seen by the inspection personnel on the actual equipment is synchronized with the background data. When the inspection personnel approach the equipment, the intelligent terminal collects the on-site image and uploads it to the background system. The background system identifies and locates the target equipment, and pushes the real-time information of the equipment to the terminal device for projection. The system obtains the real-time posture of the equipment through the KCF tracking algorithm, and adjusts the display position of the virtual information in real time to ensure that the virtual information (such as equipment status, operation suggestions) can be accurately superimposed on the actual position of the equipment.
[0078] In summary, if Figure 2 As shown, the intelligent inspection system provided in this embodiment includes at least the following modules:
[0079] The intelligent inspection terminal is worn by the inspection personnel to collect on-site information in real time and transmit it to the background management system. It is also used to project and overlay the equipment-related information pushed by the background management system onto the designated area. The intelligent inspection terminal is subdivided into information collection module, communication module and virtual information rendering module. The virtual information rendering module integrates the virtual information pushed by the background management system with the scene of the inspection site through Unity3D technology to provide enhanced visual perception for the operators.
[0080] The backend management system is used to identify, locate and track the images uploaded by the intelligent inspection terminal, query the relevant information of the equipment and push it together with the acquired real-time posture of the equipment to the intelligent inspection terminal. For the specific method steps of equipment identification, positioning and tracking, please refer to the above-mentioned embodiment one. The backend management system is subdivided into a device identification module, a tracking registration module and a posture estimation module: the device identification module ensures that the system can accurately identify and locate the equipment in the field operation, providing a basis for the accurate rendering of virtual information; the tracking registration module uses the KCF tracking algorithm to track the device position in real time to ensure the consistency of the virtual scene and the real scene; the posture estimation module estimates the posture of the device to ensure the high consistency of the visual effect between the projected virtual image or video and the actual object.
[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Under the concept of the present invention, the technical features in the above embodiments or different embodiments may also be combined, the steps may be implemented in any order, and there are many other changes in different aspects of the present invention as described above, which are not provided in detail for the sake of simplicity. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.
Claims
1. An intelligent inspection method based on AR augmented reality technology, characterized in that: include: Step 1: Identify the identity information and preliminary posture information of the acquisition device in the image based on the feature matching algorithm; Step 2: Based on the preliminary pose information, use the semantic segmentation algorithm to obtain the regional pose information of the acquisition device; Step 3: query the status attribute information of the device according to the identity information of the acquisition device, and project the queried status attribute information to the area where the acquisition device is located for display.
2. The intelligent inspection method according to claim 1, characterized in that: In step 1, based on the improved AKAZE algorithm, the image matching between the acquisition device during inspection and the target device is performed to obtain the identity information and preliminary posture information of the acquisition device, which specifically includes the following steps: Use the ORB algorithm to detect the image feature points of the acquisition device and the target device respectively; The M-LDB descriptor is used to describe the image feature points of the acquisition device and the target device respectively; By comparing the M-LDB descriptors, feature point matching is performed to identify the identity information of the acquisition device.
3. The intelligent inspection method according to claim 2, characterized in that: After matching feature points by comparing M-LDB descriptors, the GMS algorithm is used to eliminate mismatched points.
4. The intelligent inspection method according to claim 1, characterized in that: In step 2, the image of the acquisition device is input into the U-Net network for semantic segmentation to obtain the regional pose information of the acquisition device, and the encoding part of the U-Net network is replaced by the DRN network.
5. The intelligent inspection method according to claim 4, characterized in that: Perform edge detection on the image of the acquisition device, and use the edge detection result and the fused image of the acquisition device as the input of the U-Net network.
6. The intelligent inspection method according to claim 1, characterized in that: In step 3, the target tracking algorithm is used to track the identified acquisition device to obtain the real-time position and posture of the acquisition device, and the position and posture of the projection device are estimated in real time so that the projection information can be accurately superimposed on the area where the acquisition device is located.
7. The intelligent inspection method according to claim 1, characterized in that: The status attribute information includes the running status, historical records and operation suggestions of the device.
8. An intelligent inspection system based on the method according to any one of claims 1 to 7, characterized in that: include: Intelligent inspection terminal, used to collect on-site information in real time and transmit it to the background management system, and also used to project and superimpose the equipment status attribute information pushed by the background management system to the designated area; The backend management system is used to identify, locate and track the equipment in the images uploaded by the intelligent inspection terminal, query the equipment's status attribute information and push it together with the equipment's real-time posture information to the intelligent inspection terminal.
9. The intelligent inspection system according to claim 8, characterized in that: The intelligent inspection terminal includes an information collection module, a communication module and a virtual information rendering module. The virtual information rendering module integrates the device status attribute information pushed by the background management system with the scene of the inspection site through Unity3D technology.
10. The intelligent inspection system according to claim 8, characterized in that: The background management system includes a device identification module, a tracking and registration module, and a posture estimation module: the device identification module ensures that the system can accurately identify and locate the equipment in the field operation, providing a basis for the accurate rendering of virtual information; the tracking and registration module uses the target tracking algorithm to track the device position in real time to ensure the coherence of the virtual scene and the real scene; The posture estimation module estimates the posture of the device to ensure the consistency of the visual effects between the projected virtual information image or video and the actual device.