Hierarchical model dynamic projection method and device based on augmented reality

Abstract

The invention provides a hierarchical model dynamic projection method and device based on augmented reality. The method comprises the steps that RGB image data, depth image data and point cloud data of a power construction environment are collected through multiple sensors; performing data fusion and scene reconstruction based on the RGB image data, the depth image data and the point cloud data to obtain a scene model of the power construction environment; performing hierarchical division on the scene model to obtain a hierarchical model; updating the hierarchical model in real time; real power construction environment information is collected through augmented reality equipment; fusing the updated hierarchical model with the real power construction environment information, so that the hierarchical model is dynamically projected into the real power construction environment; according to the method, the precision and the real-time performance of the augmented reality technology in power construction can be improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a layered model dynamic projection method and device based on augmented reality. Background Art

[0002] Currently, some augmented reality (AR) technologies have been applied to on-site operations in power construction, assisting construction workers with wiring planning or equipment installation through simple 3D model projection and basic environment capture functions. For example, Chinese patent application publication number CN113269832 A proposes an augmented reality navigation method for power operations in extreme weather environments. The method is characterized by the user wearing an augmented display device to locate faulty equipment in a virtual scene. The augmented display device is used to project the virtual scene in front of the user's line of sight. The virtual scene is a scene of the scene to be simulated in an extreme environment. By obtaining an image of the scene to be simulated in a non-extreme environment as an initial map, combined with instance target extraction in the extreme environment, the user locates the faulty equipment in the virtual scene by moving in reality. The positioning adopts a three-dimensional tracking and registration method. For another example, the Chinese patent application with publication number CN119888143A provides an augmented reality power pipeline inspection system for achieving SAR spatial stabilization, including a data acquisition module for acquiring GIS surveying and mapping data, positioning data, and construction CAD drawings of power pipelines; a data normalization module for normalizing the data acquired by the data acquisition module, projecting the content data of the construction CAD drawings into a coordinate system map based on geodetic coordinates, and performing multi-vertex alignment and binding of the construction CAD drawings to the map base map, thereby acquiring a SAR map containing a power pipeline model; an augmented reality module for collecting inspection screen images and inspector positioning information during inspections, acquiring power pipeline model segments within geodetic coordinates through positioning information, and projecting the power pipeline model segments into the inspection screen images through AR technology.

[0003] However, this approach is limited to the overlay of static models, lacking the dynamic presentation of multi-level information in complex scenes. Furthermore, model adjustments are difficult, making it ineffective in responding to real-time changes on the construction site. Furthermore, existing AR systems often rely on a single sensor (such as LiDAR) to acquire environmental information, making it difficult to achieve high-precision environmental modeling and real-time adjustments in complex power construction environments. Furthermore, functions such as occlusion processing and model masking are also relatively limited, failing to meet the precision and real-time requirements of power construction. Summary of the Invention

[0004] The present invention provides a hierarchical model dynamic projection method and device based on augmented reality to solve the problem that existing augmented reality technology cannot meet the accuracy and real-time requirements in power construction.

[0005] A layered model dynamic projection method based on augmented reality, comprising:

[0006] Collect RGB image data, depth image data, and point cloud data of the power construction environment through multiple sensors;

[0007] Performing data fusion and scene reconstruction based on the RGB image data, depth image data, and point cloud data to obtain a scene model of the power construction environment;

[0008] Dividing the scene model into layers to obtain a layered model;

[0009] updating the hierarchical model in real time;

[0010] Collect real-world power construction environment information through augmented reality equipment;

[0011] The updated hierarchical model is integrated with the real electric power construction environment information so that the hierarchical model is dynamically projected into the real electric power construction environment.

[0012] Furthermore, after collecting RGB image data, depth image data, and point cloud data of the power construction environment, the following is also included:

[0013] Denoising is performed on the RGB image data, depth image data, and point cloud data.

[0014] Furthermore, denoising is performed on the RGB image data, the depth image data, and the point cloud data, including:

[0015] For each pixel in the RGB image data and the depth image data, select a neighborhood window of a preset size, calculate a weight based on the positional relationship between each pixel in the neighborhood window and the central pixel, calculate a weighted average of each pixel based on the weight, and update the corresponding pixel value based on the weighted average to obtain a denoised pixel;

[0016] For each point cloud in the point cloud data, a k-neighborhood of a preset size is selected, and a distance weight of the point clouds in the k-neighborhood is calculated. The weighted average position of each point cloud is calculated according to the distance weight, and the position of the corresponding point cloud is updated according to the weighted average position to obtain the denoised point cloud data.

[0017] Furthermore, data fusion and scene reconstruction are performed based on the RGB image data, depth image data, and point cloud data to obtain a scene model of the power construction environment, including:

[0018] Establishing a point cloud model based on the point cloud data;

[0019] Establishing a mapping relationship between the RGB image data and the point cloud model, and a mapping relationship between the depth image data and the point cloud data respectively;

[0020] The color information in the RGB image data and the depth information in the depth image data are fused into the point cloud model to obtain the scene model.

[0021] Furthermore, fusing the color information in the RGB image data and the depth information in the depth image data into the point cloud model includes:

[0022] According to the mapping relationship between the RGB image data and the point cloud model, the color information of the corresponding pixel in the RGB image data is added to the corresponding point cloud in the point cloud model;

[0023] Identify the area to be enhanced in the point cloud model, and merge the depth values ​​of the pixels corresponding to the area to be enhanced in the depth image data with the corresponding point cloud in the area to be enhanced according to a pre-established mapping relationship between the depth image data and the point cloud model.

[0024] Furthermore, the layered model includes a terrain layer, a device layer, a wiring layer, and a temporary facility layer;

[0025] The scene model is divided into layers to obtain a layered model, including:

[0026] Inputting the scene model into a pre-established convolutional neural network for edge recognition, and outputting edge recognition results;

[0027] Hierarchical clustering is performed based on the edge recognition result to obtain a hierarchical model of the terrain layer, equipment layer, wiring layer and temporary facility layer.

[0028] Furthermore, the updated hierarchical model is integrated with the real power construction environment information so that the hierarchical model is dynamically projected into the real power construction environment, including:

[0029] Calibrate the augmented reality device with internal and external parameters to obtain an internal parameter matrix and an external parameter matrix;

[0030] Establishing a projection matrix according to the intrinsic parameter matrix and the extrinsic parameter matrix;

[0031] Obtaining projection position coordinates according to the three-dimensional coordinates of each point cloud in the layered model and the projection matrix;

[0032] Using an extended Kalman filter algorithm to fuse the projected position coordinates with the real power construction environment information, predicting and correcting the projected position coordinates to obtain an estimated value of the projected position coordinates;

[0033] Projection is performed based on the estimated value of the projection position coordinates, and the layered model is rendered.

[0034] Furthermore, rendering the layered model includes:

[0035] Acquire depth data of real objects in the electric power construction environment according to the depth image data, and construct a depth buffer;

[0036] After projecting the layered model into the screen space, obtaining the current depth value of the point cloud in the layered model at the current viewing angle;

[0037] The current depth value of the point cloud corresponding to each screen pixel is compared with the corresponding depth value stored in the depth buffer. If the current depth value is less than or equal to the corresponding depth value stored in the depth buffer, the current point cloud is rendered.

[0038] Furthermore, the projected position coordinates are integrated with the real power construction environment information using an extended Kalman filter algorithm to predict and correct the projected position coordinates, including:

[0039] The position and velocity of the dynamic target are used as the initial state variables, and the covariance matrix, process noise matrix and observation noise matrix are initialized;

[0040] Establishing an observation model based on the coordinates of the feature points of the dynamic target object in the real power construction environment information;

[0041] Establishing the Jacobian matrix of state quantity transfer and the Jacobian matrix of observation function, predicting the state quantity to obtain the predicted state quantity, and performing covariance prediction based on the Jacobian matrix of state quantity transfer to obtain the predicted covariance matrix;

[0042] The Kalman gain is updated according to the Jacobian matrix of the observation function and the predicted covariance matrix, and the covariance matrix is ​​updated according to the updated Kalman gain and the predicted covariance matrix, and the state quantity is updated according to the updated Kalman gain, the updated covariance matrix and the predicted state quantity to obtain an estimated value of the projection position coordinate.

[0043] A layered model dynamic projection device based on augmented reality, comprising:

[0044] The first acquisition module is used to collect RGB image data, depth image data and point cloud data of the power construction environment through multiple sensors;

[0045] A model building module is used to perform data fusion and scene reconstruction based on the RGB image data, depth image data and point cloud data to obtain a scene model of the power construction environment;

[0046] A division module, configured to divide the scene model into layers to obtain a layered model;

[0047] An updating module, configured to update the hierarchical model in real time;

[0048] The second acquisition module is used to control the acquisition of real power construction environment information through augmented reality equipment;

[0049] The projection module is used to integrate the updated hierarchical model with the real power construction environment information so that the hierarchical model is dynamically projected into the real power construction environment.

[0050] Furthermore, after the first acquisition module acquires the RGB image data, depth image data, and point cloud data of the power construction environment, it further includes:

[0051] Denoising is performed on the RGB image data, depth image data, and point cloud data.

[0052] Furthermore, the first acquisition module performs denoising processing on the RGB image data, the depth image data, and the point cloud data, including:

[0053] For each pixel in the RGB image data and the depth image data, select a neighborhood window of a preset size, calculate a weight based on the positional relationship between each pixel in the neighborhood window and the central pixel, calculate a weighted average of each pixel based on the weight, and update the corresponding pixel value based on the weighted average to obtain a denoised pixel;

[0054] For each point cloud in the point cloud data, a k-neighborhood of a preset size is selected, and a distance weight of the point clouds in the k-neighborhood is calculated. The weighted average position of each point cloud is calculated according to the distance weight, and the position of the corresponding point cloud is updated according to the weighted average position to obtain the denoised point cloud data.

[0055] Furthermore, the model building module performs data fusion and scene reconstruction based on the RGB image data, depth image data, and point cloud data to obtain a scene model of the power construction environment, including:

[0056] Establishing a point cloud model based on the point cloud data;

[0057] Establishing a mapping relationship between the RGB image data and the point cloud model, and a mapping relationship between the depth image data and the point cloud data respectively;

[0058] The color information in the RGB image data and the depth information in the depth image data are fused into the point cloud model to obtain the scene model.

[0059] Furthermore, the model building module fuses the color information in the RGB image data and the depth information in the depth image data into the point cloud model, including:

[0060] According to the mapping relationship between the RGB image data and the point cloud model, the color information of the corresponding pixel in the RGB image data is added to the corresponding point cloud in the point cloud model;

[0061] Identify the area to be enhanced in the point cloud model, and merge the depth values ​​of the pixels corresponding to the area to be enhanced in the depth image data with the corresponding point cloud in the area to be enhanced according to a pre-established mapping relationship between the depth image data and the point cloud model.

[0062] Furthermore, the layered model includes a terrain layer, a device layer, a wiring layer, and a temporary facility layer;

[0063] The dividing module divides the scene model into layers to obtain a layered model, including:

[0064] Inputting the scene model into a pre-established convolutional neural network for edge recognition, and outputting edge recognition results;

[0065] Hierarchical clustering is performed based on the edge recognition result to obtain a hierarchical model of the terrain layer, equipment layer, wiring layer and temporary facility layer.

[0066] Furthermore, the projection module integrates the updated hierarchical model with the real power construction environment information so that the hierarchical model is dynamically projected into the real power construction environment, including:

[0067] Calibrate the augmented reality device with internal and external parameters to obtain an internal parameter matrix and an external parameter matrix;

[0068] Establishing a projection matrix according to the intrinsic parameter matrix and the extrinsic parameter matrix;

[0069] Obtaining projection position coordinates according to the three-dimensional coordinates of each point cloud in the layered model and the projection matrix;

[0070] Using an extended Kalman filter algorithm to fuse the projected position coordinates with the real power construction environment information, predicting and correcting the projected position coordinates to obtain an estimated value of the projected position coordinates;

[0071] Projection is performed based on the estimated value of the projection position coordinates, and the layered model is rendered.

[0072] Furthermore, the projection module renders the layered model, including:

[0073] Acquire depth data of real objects in the electric power construction environment according to the depth image data, and construct a depth buffer;

[0074] After projecting the layered model into the screen space, obtaining the current depth value of the point cloud in the layered model at the current viewing angle;

[0075] The current depth value of the point cloud corresponding to each screen pixel is compared with the corresponding depth value stored in the depth buffer. If the current depth value is less than or equal to the corresponding depth value stored in the depth buffer, the current point cloud is rendered.

[0076] Furthermore, the projection module uses an extended Kalman filter algorithm to fuse the projected position coordinates with the real power construction environment information, and predicts and corrects the projected position coordinates, including:

[0077] The position and velocity of the dynamic target are used as the initial state variables, and the covariance matrix, process noise matrix and observation noise matrix are initialized;

[0078] Establishing an observation model based on the coordinates of the feature points of the dynamic target object in the real power construction environment information;

[0079] Establishing the Jacobian matrix of state quantity transfer and the Jacobian matrix of observation function, predicting the state quantity to obtain the predicted state quantity, and performing covariance prediction based on the Jacobian matrix of state quantity transfer to obtain the predicted covariance matrix;

[0080] The Kalman gain is updated according to the Jacobian matrix of the observation function and the predicted covariance matrix, and the covariance matrix is ​​updated according to the updated Kalman gain and the predicted covariance matrix, and the state quantity is updated according to the updated Kalman gain, the updated covariance matrix and the predicted state quantity to obtain an estimated value of the projection position coordinate.

[0081] The method and device for dynamic projection of layered models based on augmented reality provided by the present invention have at least the following beneficial effects:

[0082] (1) Through multi-sensor fusion technology and dynamic hierarchical model generation, real-time environmental perception, accurate model projection, and automatic dynamic update are achieved, providing high-precision augmented reality guidance for power construction;

[0083] (2) Through the extended Kalman filter (EKF) algorithm, the system can update the projection position according to sensor data at any time during the construction process to ensure the real-time consistency between the model and the on-site scene;

[0084] (3) Based on multi-sensor fusion technology, the system can accurately obtain three-dimensional information of the construction site and improve the accuracy of construction operations through precise layered model projection;

[0085] (4) Different levels of information in the construction scene, such as terrain, equipment, wiring, etc., can be dynamically displayed to facilitate layered operations by construction personnel and improve the efficiency of information management.

[0086] (5) When changes occur on site, the system can automatically detect the changes and adjust the projection model in real time, so that construction personnel can always refer to the accurate layered model for construction operations, with high real-time performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] Figure 1 This is a flowchart of an embodiment of the method for dynamic projection of a layered model based on augmented reality provided by the present invention.

[0088] Figure 2 The present invention provides a flowchart of an embodiment of establishing a scene model in the layered model dynamic projection method based on augmented reality.

[0089] Figure 3 This is a flowchart of an embodiment of model layering in the layered model dynamic projection method based on augmented reality provided by the present invention.

[0090] Figure 4 This is a flowchart of an embodiment of model projection in the layered model dynamic projection method based on augmented reality provided by the present invention.

[0091] Figure 5 The present invention provides a flowchart of an embodiment of position correction in the layered model dynamic projection method based on augmented reality.

[0092] Figure 6 This is a structural diagram of an embodiment of the layered model dynamic projection device based on augmented reality provided by the present invention. DETAILED DESCRIPTION

[0093] In order to better understand the above technical solution, the above technical solution will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0094] refer to Figure 1 In some embodiments, a method for dynamic projection of a layered model based on augmented reality is provided, comprising:

[0095] S1, collect RGB image data, depth image data and point cloud data of the power construction environment through multiple sensors;

[0096] S2. Performing data fusion and scene reconstruction based on the RGB image data, depth image data, and point cloud data to obtain a scene model of the power construction environment;

[0097] S3, dividing the scene model into layers to obtain a layered model;

[0098] S4, updating the hierarchical model in real time;

[0099] S5. Collect real power construction environment information through augmented reality equipment;

[0100] S6. Fusing the updated hierarchical model with the real electric power construction environment information, so that the hierarchical model is dynamically projected into the real electric power construction environment.

[0101] Specifically, in step S1, because a single sensor is inadequate for the complex power construction environment, the system employs multi-sensor fusion technology, including RGB cameras, depth cameras, and LiDAR, to acquire accurate environmental information. For example, the RGB camera captures color information and construction details, the depth camera captures distance data, and the LiDAR provides precise 3D point cloud information. This multi-sensor collaboration enables comprehensive perception in complex power construction scenarios.

[0102] Furthermore, after collecting RGB image data, depth image data, and point cloud data of the power construction environment, the following is also included:

[0103] Denoising is performed on the RGB image data, depth image data, and point cloud data.

[0104] Specifically, a weighted average method may be used to perform denoising on the RGB image data, depth image data, and point cloud data, specifically including:

[0105] For each pixel in the RGB image data and the depth image data, select a neighborhood window of a preset size, calculate a weight based on the positional relationship between each pixel in the neighborhood window and the central pixel, calculate a weighted average of each pixel based on the weight, and update the corresponding pixel value based on the weighted average to obtain a denoised pixel;

[0106] For each point cloud in the point cloud data, a k-neighborhood of a preset size is selected, and a distance weight of the point clouds in the k-neighborhood is calculated. The weighted average position of each point cloud is calculated according to the distance weight, and the position of the corresponding point cloud is updated according to the weighted average position to obtain the denoised point cloud data.

[0107] Specifically, a weighted averaging method is used to denoise RGB image data, effectively smoothing image noise while preserving edge information. A weighted averaging method is also used to denoise depth image data, with weights adjusted based on distance or gradient changes to avoid blurred boundaries, remove noise points caused by depth measurement errors, and improve depth map quality.

[0108] The weighted average method is used to denoise the point cloud data, which can remove outliers or measurement errors in the point cloud data and improve the model accuracy.

[0109] Further, refer to Figure 2 In step S2, data fusion and scene reconstruction are performed based on the RGB image data, depth image data, and point cloud data to obtain a scene model of the power construction environment, including:

[0110] S21, establishing a point cloud model based on the point cloud data;

[0111] S22, establishing a mapping relationship between the RGB image data and the point cloud model, and a mapping relationship between the depth image data and the point cloud data;

[0112] S23: Fusing the color information in the RGB image data and the depth information in the depth image data into the point cloud model to obtain the scene model.

[0113] Specifically, in step S21, a point cloud registration algorithm is used to combine the point cloud data to form an accurate scene model. Specifically, an ICP (Iterative Closest Point)-based registration method can be used to reconstruct the 3D point cloud, matching point cloud data from different perspectives to the same coordinate system to generate a detailed point cloud model.

[0114] Furthermore, in step S22, pixel coordinates in the RGB image data are mapped to point cloud data based on the camera's intrinsic and extrinsic parameter matrices. For point cloud data acquired by laser, a mapping relationship between the depth image data and the point cloud data can be established through a registration algorithm. If the point cloud data is generated from the depth image data, a natural mapping relationship exists between them.

[0115] Furthermore, in step S23, the color information in the RGB image data and the depth information in the depth image data are fused into the point cloud model, including:

[0116] According to the mapping relationship between the RGB image data and the point cloud model, the color information of the corresponding pixel in the RGB image data is added to the corresponding point cloud in the point cloud model;

[0117] Identify the area to be enhanced in the point cloud model, and merge the depth values ​​of the pixels corresponding to the area to be enhanced in the depth image data with the corresponding point cloud in the area to be enhanced according to a pre-established mapping relationship between the depth image data and the point cloud model.

[0118] Specifically, the color information of the corresponding pixels in the RGB image data is appended to the corresponding point cloud in the point cloud model, so that the point cloud in the point cloud model is colored. There may be fuzzy areas in the point cloud model, and some areas of the depth map data may be clearer or more accurate than the point cloud data. Therefore, the fuzzy areas in the point cloud model are identified as areas to be enhanced. Based on the pre-established mapping relationship between the depth image data and the point cloud model, the depth values ​​of the pixels in the depth image data corresponding to the area to be enhanced are merged with the corresponding point cloud in the area to be enhanced, that is, the Z coordinates of the point cloud in the area to be enhanced are replaced with the corresponding depth values.

[0119] Further, refer to Figure 3 ,In step S3, the hierarchical model includes a terrain layer, a device layer, a wiring layer, and a temporary facility layer;

[0120] The scene model is divided into layers to obtain a layered model, including:

[0121] S31, inputting the scene model into a pre-established convolutional neural network for edge recognition, and outputting an edge recognition result;

[0122] S32. Perform hierarchical clustering based on the edge recognition result to obtain a hierarchical model of the terrain layer, equipment layer, wiring layer, and temporary facility layer.

[0123] In step S31, edge recognition in the point cloud model is implemented based on a convolutional neural network (CNN). The scene model is input into the convolutional neural network, and the edge recognition result is output.

[0124] Specifically, in step S32, multi-scale clustering is adopted. Clustering can be performed first at the local scale to obtain small components such as wires and bolts, and then small categories can be merged at the global scale. For example, multiple small components can be combined into equipment, and finally a hierarchical structure of terrain, equipment, wiring, and temporary facilities is formed.

[0125] Utilizing 3D reconstruction technology, a point cloud model of the construction environment is generated and divided into different layers. These layers include terrain, equipment, and wiring paths. This effectively displays equipment layout and wiring routing in a layered manner, facilitating step-by-step construction.

[0126] Furthermore, in step S4, the hierarchical model is updated in real time according to real-time changes in power construction (such as addition of new equipment or changes in the environment).

[0127] Specifically, RGB image data, depth image data and point cloud data can be collected in real time through multiple sensors, the newly collected point cloud data can be aligned with the existing scene model through the registration algorithm, and the RGB image data and depth image data can be fused into the updated scene model respectively to realize the update of the hierarchical model.

[0128] Furthermore, in step S5, the augmented reality device may be wearable AR glasses, a helmet, etc., or a mobile terminal.

[0129] refer to Figure 4 In step S6, the updated hierarchical model is integrated with the real power construction environment information so that the hierarchical model is dynamically projected into the real power construction environment, including:

[0130] S61, calibrating the internal and external parameters of the augmented reality device to obtain an internal parameter matrix and an external parameter matrix;

[0131] S62, establishing a projection matrix according to the intrinsic parameter matrix and the extrinsic parameter matrix;

[0132] S63, obtaining projection position coordinates according to the three-dimensional coordinates of each point cloud in the layered model and the projection matrix;

[0133] S64, using an extended Kalman filter algorithm to fuse the projected position coordinates with the real power construction environment information, predicting and correcting the projected position coordinates to obtain an estimated value of the projected position coordinates;

[0134] S65 , performing projection based on the estimated value of the projection position coordinates, and rendering the layered model.

[0135] Rendering and projection are key technologies used by augmented reality systems to display layered models in physical space. Through this technology, the system projects the generated 3D model into the construction environment, helping construction workers achieve precise positioning and visual operations.

[0136] Specifically, in step S61, the extrinsic parameter matrix of the augmented reality device includes a rotation matrix and a translation matrix.

[0137] Furthermore, in step S62 and step S63, the spatial coordinate transformation is performed using the intrinsic parameter matrix and the extrinsic parameter matrix of the augmented reality device, and the projection matrix formula established is as follows:

[0138] P proj =K·{R|T}·P worlk ; (1)

[0139] Among them, K is the intrinsic parameter matrix of the augmented reality device, R is the rotation matrix, T is the translation matrix, P proj is the projection position coordinate, Pworld Point cloud coordinates in a hierarchical model.

[0140] Further, refer to Figure 5 In step S64, the projected position coordinates are integrated with the real power construction environment information using an extended Kalman filter algorithm to predict and correct the projected position coordinates, including:

[0141] S64a, taking the position and velocity of the dynamic target as initial state variables, and initializing the covariance matrix, process noise matrix, and observation noise matrix;

[0142] S64b, establishing an observation model according to the coordinates of the feature points of the dynamic target objects in the real power construction environment information;

[0143] S64c, establishing a Jacobian matrix for state quantity transfer and a Jacobian matrix for the observation function, predicting the state quantity to obtain a predicted state quantity, and performing covariance prediction based on the Jacobian matrix for state quantity transfer to obtain a predicted covariance matrix;

[0144] S64d, update the Kalman gain according to the Jacobian matrix of the observation function and the predicted covariance matrix, and update the covariance matrix according to the updated Kalman gain and the predicted covariance matrix, update the state quantity according to the updated Kalman gain, the updated covariance matrix and the predicted state quantity, and obtain the estimated value of the projection position coordinate.

[0145] Specifically, in step S64a, the position and speed of the dynamic target are variables related to the projection position coordinates, which are used as state quantities to initialize the state quantity x t , as well as the covariance matrix P, the process noise matrix and the observation noise matrix.

[0146] The state equation of the state quantity is:

[0147] x t =f(x t-1 ,u t )+w t ; (2)

[0148] Among them, x t represents the state quantity, f(x t-1 ,u t ) represents the nonlinear state transfer function, u t represents the control input vector, w t represents process noise.

[0149] Furthermore, in step S64b, the coordinates of the feature points of the dynamic target objects in the real power construction environment information are z=[x obs ,y obs ,zobs ] 2 ; It can be extracted through the camera + SLAM system of the augmented reality device.

[0150] The observation model is as follows:

[0151] z k =h(x k )+v k ; (3)

[0152] Among them, z k represents the observation value at time k, h(x k ) represents the nonlinear observation function, v k represents the observation noise.

[0153] Furthermore, in step S64c, the Jacobian matrix of the state quantity transfer is defined as:

[0154]

[0155] Among them, F t Represents the Jacobian matrix of state quantity transfer, and f represents the nonlinear state transfer function.

[0156] The Jacobian matrix of the observation function is as follows:

[0157]

[0158] Among them, H t represents the Jacobian matrix of the observation function, and h represents the nonlinear observation function.

[0159] The predicted state quantity is:

[0160]

[0161] in, represents the predicted state quantity, represents the state estimate at the previous moment, f represents the nonlinear state transfer function, u t represents the control input vector.

[0162] The forecast covariance matrix is ​​expressed by the following formula:

[0163]

[0164] Among them, P t|t-1 Represents the prediction covariance matrix at the current moment, F t The Jacobian matrix representing the state transfer, P t-1|t-1 represents the posterior covariance matrix of the previous moment, Q t represents the process noise covariance matrix, Indicates F t The transposed matrix of .

[0165] Furthermore, in step S64d, the updated Kalman gain is as follows:

[0166]

[0167] Among them, K t represents the Kalman gain, R t represents the observation noise covariance matrix, H t represents the Jacobian matrix of the observation function, Indicates H t The transposed matrix of .

[0168] The updated covariance matrix is:

[0169] P t|t =(IK t H t )P t|t-1 ; (9)

[0170] Among them, P t|t represents the updated covariance matrix and I represents the identity matrix.

[0171] The updated state is:

[0172]

[0173] Among them, x t|t represents the updated state quantity, Represents the predicted state quantity, K t represents the Kalman gain, z t Represents the observation value, and h represents the nonlinear observation function.

[0174] After obtaining the updated state quantity, the corrected projection position coordinates are calculated based on the updated state quantity to obtain an estimated value of the projection position coordinates, and projection is performed based on a position corresponding to the estimated value of the projection position coordinates.

[0175] Specifically, in step S65, the multi-layered model data is rendered using the Unity engine or other real-time rendering platform, and the model details are enhanced by shaders and light and shadow processing during rendering to make it more three-dimensional.

[0176] To enhance realism, occlusion processing techniques can be applied to remove invisible parts of the layered model, ensuring that the projection effect is consistent with the occlusion relationship of the real object. Occlusion processing is mainly based on a depth test algorithm, which hides the occluded parts of the model through depth comparison. Each point cloud in the layered model has a pre-stored depth value.

[0177] Therefore, in step S65, rendering the layered model includes:

[0178] Acquire depth data of real objects in the electric power construction environment according to the depth image data, and construct a depth buffer;

[0179] After projecting the layered model into the screen space, obtaining the current depth value of the point cloud in the layered model at the current viewing angle;

[0180] The current depth value of the point cloud corresponding to each screen pixel is compared with the corresponding depth value stored in the depth buffer. If the current depth value is less than or equal to the corresponding depth value stored in the depth buffer, the current point cloud is rendered, otherwise it will be ignored.

[0181] Specifically, each point cloud in the layered model has a depth value (z value), which indicates its distance from the camera; a depth buffer is maintained to record the depth value of the real object closest to the augmented reality device at each point cloud position under the current perspective; when rendering a virtual object, if its depth value is greater than the depth value stored in the depth buffer (that is, it is behind the real object), the pixel is not rendered; otherwise, it is rendered.

[0182] This method ensures that the relevant parts of the model are displayed only when the view is not blocked by other objects. For opaque objects, the algorithm automatically blocks out the parts of the background that are blocked by the object, ensuring that the display of virtual objects is consistent with the occlusion in the real world.

[0183] For example, when viewing a red distribution box through AR glasses, the system projects a cable model behind it. Without occlusion processing, the cable would appear to "penetrate" the box, appearing unrealistic. With this occlusion processing, the system detects the distribution box between the user and the cable, assuming it should be obscured by the box. Therefore, only the cable in front of the box is rendered, while the cable behind it is not displayed. The visual effect is that the cable is "hidden" behind the box, just as in reality.

[0184] Through these steps, the system can accurately simulate the occlusion effect of objects in the real environment, making the augmented reality experience more realistic.

[0185] This method uses multiple sensors and a point cloud change monitoring algorithm to detect changes in the construction site. When displacement or changes in the model's position are detected, a dynamic projection algorithm automatically activates to adjust the projection content, ensuring accurate, real-time projection.

[0186] Dynamic projection works in conjunction with the layered model to synchronize updates. Using a model synchronization algorithm, the system renders the latest layered structure in real time and projects the updated layered model onto the target area of ​​the construction site. For newly added equipment or tool models, the system automatically performs incremental updates to avoid misalignment or ghosting in the projection.

[0187] The method provided in this embodiment is further described below through specific application scenarios:

[0188] During transmission line construction, the paths of line towers and conductors need to avoid existing obstacles such as buildings, trees, and roads. Using the augmented reality-based layered model dynamic projection method provided in this embodiment, the planned line model can be superimposed on the on-site environment in real time, allowing construction workers to intuitively observe whether the path selection conflicts with on-site obstacles, ensuring the accuracy of wiring planning. The specific operations are as follows:

[0189] Environmental scanning and modeling: The system uses multi-sensor equipment to scan the site, digitize information such as terrain and obstacles in the transmission line path into a scene model, and update it in real time.

[0190] Path Planning and Dynamic Adjustment: The preset transmission line path is marked on the scene model, and the different construction layers (such as infrastructure layer, environment layer, and preset line layer) are intuitively displayed through a layered model. If the system detects that the current line path intersects with an obstacle, it will automatically generate a recommended detour path and dynamically display it on the AR device to ensure the safety and rationality of the line path.

[0191] Substation equipment installation and wiring optimization:

[0192] During substation construction, the proper installation and wiring layout of equipment are crucial. The augmented reality-based layered model dynamic projection method provided in this embodiment can project a 3D model of equipment onto the physical scene according to the construction schedule, helping construction workers determine the exact installation location of equipment and optimize wiring paths to avoid intersections and interference.

[0193] Equipment layer projection: According to the construction progress of the equipment, the system can project different equipment models onto the site layer by layer, allowing construction personnel to view the positional relationship of the equipment on the site through AR devices.

[0194] Wiring Optimization: The system simulates wiring based on site conditions, presenting the wiring paths between devices through a layered model to avoid interference caused by intersecting paths or insufficient space. Construction personnel can then implement wiring according to the system's recommended routing, ensuring stable and efficient wiring.

[0195] refer to Figure 6In some embodiments, a device for dynamically projecting a layered model based on augmented reality is further provided, comprising:

[0196] The first acquisition module 201 is used to collect RGB image data, depth image data and point cloud data of the power construction environment through multiple sensors;

[0197] A model building module 202 is configured to perform data fusion and scene reconstruction based on the RGB image data, depth image data, and point cloud data to obtain a scene model of the power construction environment;

[0198] A division module 203 is used to divide the scene model into layers to obtain a layered model;

[0199] An updating module 204, configured to update the hierarchical model in real time;

[0200] The second collection module 205 is used to control the collection of real power construction environment information through augmented reality equipment;

[0201] The projection module 206 is configured to integrate the updated hierarchical model with the real electric power construction environment information, so that the hierarchical model is dynamically projected into the real electric power construction environment.

[0202] Furthermore, after the first acquisition module 201 acquires the RGB image data, depth image data, and point cloud data of the power construction environment, it further includes:

[0203] Denoising is performed on the RGB image data, depth image data, and point cloud data.

[0204] Furthermore, the first acquisition module performs denoising processing on the RGB image data, the depth image data, and the point cloud data, including:

[0205] For each pixel in the RGB image data and the depth image data, select a neighborhood window of a preset size, calculate a weight based on the positional relationship between each pixel in the neighborhood window and the central pixel, calculate a weighted average of each pixel based on the weight, and update the corresponding pixel value based on the weighted average to obtain a denoised pixel;

[0206] For each point cloud in the point cloud data, a k-neighborhood of a preset size is selected, and a distance weight of the point clouds in the k-neighborhood is calculated. The weighted average position of each point cloud is calculated according to the distance weight, and the position of the corresponding point cloud is updated according to the weighted average position to obtain the denoised point cloud data.

[0207] Furthermore, the model building module 202 performs data fusion and scene reconstruction based on the RGB image data, depth image data, and point cloud data to obtain a scene model of the power construction environment, including:

[0208] Establishing a point cloud model based on the point cloud data;

[0209] Establishing a mapping relationship between the RGB image data and the point cloud model, and a mapping relationship between the depth image data and the point cloud data respectively;

[0210] The color information in the RGB image data and the depth information in the depth image data are fused into the point cloud model to obtain the scene model.

[0211] Furthermore, the model building module 202 fuses the color information in the RGB image data and the depth information in the depth image data into the point cloud model, including:

[0212] According to the mapping relationship between the RGB image data and the point cloud model, the color information of the corresponding pixel in the RGB image data is added to the corresponding point cloud in the point cloud model;

[0213] Identify the area to be enhanced in the point cloud model, and merge the depth values ​​of the pixels corresponding to the area to be enhanced in the depth image data with the corresponding point cloud in the area to be enhanced according to a pre-established mapping relationship between the depth image data and the point cloud model.

[0214] Furthermore, the layered model includes a terrain layer, a device layer, a wiring layer, and a temporary facility layer;

[0215] The division module 203 divides the scene model into layers to obtain a layered model, including:

[0216] Inputting the scene model into a pre-established convolutional neural network for edge recognition, and outputting edge recognition results;

[0217] Hierarchical clustering is performed based on the edge recognition result to obtain a hierarchical model of the terrain layer, equipment layer, wiring layer and temporary facility layer.

[0218] Furthermore, the projection module 206 integrates the updated hierarchical model with the real power construction environment information so that the hierarchical model is dynamically projected into the real power construction environment, including:

[0219] Calibrate the augmented reality device with internal and external parameters to obtain an internal parameter matrix and an external parameter matrix;

[0220] Establishing a projection matrix according to the intrinsic parameter matrix and the extrinsic parameter matrix;

[0221] Obtaining projection position coordinates according to the three-dimensional coordinates of each point cloud in the layered model and the projection matrix;

[0222] Using an extended Kalman filter algorithm to fuse the projected position coordinates with the real power construction environment information, predicting and correcting the projected position coordinates to obtain an estimated value of the projected position coordinates;

[0223] Projection is performed based on the estimated value of the projection position coordinates, and the layered model is rendered.

[0224] Furthermore, the projection module 206 renders the layered model, including:

[0225] Acquire depth data of real objects in the electric power construction environment according to the depth image data, and construct a depth buffer;

[0226] After projecting the layered model into the screen space, obtaining the current depth value of the point cloud in the layered model at the current viewing angle;

[0227] The current depth value of the point cloud corresponding to each screen pixel is compared with the corresponding depth value stored in the depth buffer. If the current depth value is less than or equal to the corresponding depth value stored in the depth buffer, the current point cloud is rendered.

[0228] Furthermore, the projection module 206 uses an extended Kalman filter algorithm to fuse the projected position coordinates with the real power construction environment information, and predicts and corrects the projected position coordinates, including:

[0229] The position and velocity of the dynamic target are used as the initial state variables, and the covariance matrix, process noise matrix and observation noise matrix are initialized;

[0230] Establishing an observation model based on the coordinates of the feature points of the dynamic target object in the real power construction environment information;

[0231] Establishing the Jacobian matrix of state quantity transfer and the Jacobian matrix of observation function, predicting the state quantity to obtain the predicted state quantity, and performing covariance prediction based on the Jacobian matrix of state quantity transfer to obtain the predicted covariance matrix;

[0232] The Kalman gain is updated according to the Jacobian matrix of the observation function and the predicted covariance matrix, and the covariance matrix is ​​updated according to the updated Kalman gain and the predicted covariance matrix, and the state quantity is updated according to the updated Kalman gain, the updated covariance matrix and the predicted state quantity to obtain an estimated value of the projection position coordinate.

[0233] The above embodiments provide a method and device for dynamically projecting a layered model based on augmented reality, which have at least the following beneficial effects:

[0234] (1) Through multi-sensor fusion technology and dynamic hierarchical model generation, real-time environmental perception, accurate model projection, and automatic dynamic update are achieved, providing high-precision augmented reality guidance for power construction;

[0235] (2) Through the extended Kalman filter (EKF) algorithm, the system can update the projection position according to sensor data at any time during the construction process to ensure the real-time consistency between the model and the on-site scene;

[0236] (3) Based on multi-sensor fusion technology, the system can accurately obtain three-dimensional information of the construction site and improve the accuracy of construction operations through precise layered model projection;

[0237] (4) Different levels of information in the construction scene, such as terrain, equipment, wiring, etc., can be dynamically displayed to facilitate layered operations by construction personnel and improve the efficiency of information management.

[0238] (5) When changes occur on site, the system can automatically detect the changes and adjust the projection model in real time, so that construction personnel can always refer to the accurate layered model for construction operations, with high real-time performance.

[0239] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they are aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the invention. Thus, the present invention is intended to include such changes and modifications as fall within the scope of the claims and their equivalents.

Claims

1. A layered model dynamic projection method based on augmented reality, characterized in that: include: Collect RGB image data, depth image data, and point cloud data of the power construction environment through multiple sensors; Performing data fusion and scene reconstruction based on the RGB image data, depth image data, and point cloud data to obtain a scene model of the power construction environment; Dividing the scene model into layers to obtain a layered model; updating the hierarchical model in real time; Collect real-world power construction environment information through augmented reality equipment; The updated hierarchical model is integrated with the real electric power construction environment information so that the hierarchical model is dynamically projected into the real electric power construction environment.

2. The method according to claim 1, characterized in that After collecting RGB image data, depth image data, and point cloud data of the power construction environment, it also includes: Denoising is performed on the RGB image data, depth image data, and point cloud data.

3. The method according to claim 2, characterized in that Denoising the RGB image data, the depth image data, and the point cloud data includes: For each pixel in the RGB image data and the depth image data, select a neighborhood window of a preset size, calculate a weight based on the positional relationship between each pixel in the neighborhood window and the central pixel, calculate a weighted average of each pixel based on the weight, and update the corresponding pixel value based on the weighted average to obtain a denoised pixel; For each point cloud in the point cloud data, a k-neighborhood of a preset size is selected, and a distance weight of the point clouds in the k-neighborhood is calculated. The weighted average position of each point cloud is calculated according to the distance weight, and the position of the corresponding point cloud is updated according to the weighted average position to obtain the denoised point cloud data.

4. The method according to claim 1, wherein Data fusion and scene reconstruction are performed based on the RGB image data, depth image data, and point cloud data to obtain a scene model of the power construction environment, including: Establishing a point cloud model based on the point cloud data; Establishing a mapping relationship between the RGB image data and the point cloud model, and a mapping relationship between the depth image data and the point cloud data respectively; The color information in the RGB image data and the depth information in the depth image data are fused into the point cloud model to obtain the scene model.

5. The method according to claim 4, characterized in that Fusion of the color information in the RGB image data and the depth information in the depth image data into the point cloud model includes: According to the mapping relationship between the RGB image data and the point cloud model, the color information of the corresponding pixel in the RGB image data is added to the corresponding point cloud in the point cloud model; Identify the area to be enhanced in the point cloud model, and merge the depth values ​​of the pixels corresponding to the area to be enhanced in the depth image data with the corresponding point cloud in the area to be enhanced according to a pre-established mapping relationship between the depth image data and the point cloud model.

6. The method according to claim 1, wherein The layered model includes a terrain layer, a device layer, a wiring layer, and a temporary facility layer; The scene model is divided into layers to obtain a layered model, including: Inputting the scene model into a pre-established convolutional neural network for edge recognition, and outputting edge recognition results; Hierarchical clustering is performed based on the edge recognition result to obtain a hierarchical model of the terrain layer, equipment layer, wiring layer and temporary facility layer.

7. The method according to claim 1, characterized in that The updated hierarchical model is integrated with the real power construction environment information so that the hierarchical model is dynamically projected into the real power construction environment, including: Calibrate the augmented reality device with internal and external parameters to obtain an internal parameter matrix and an external parameter matrix; Establishing a projection matrix according to the intrinsic parameter matrix and the extrinsic parameter matrix; Obtaining projection position coordinates according to the three-dimensional coordinates of each point cloud in the layered model and the projection matrix; Using an extended Kalman filter algorithm to fuse the projected position coordinates with the real power construction environment information, predicting and correcting the projected position coordinates to obtain an estimated value of the projected position coordinates; Projection is performed based on the estimated value of the projection position coordinates, and the layered model is rendered.

8. The method according to claim 7, characterized in that Rendering the layered model includes: Acquire depth data of real objects in the electric power construction environment according to the depth image data, and construct a depth buffer; After projecting the layered model into the screen space, obtaining the current depth value of the point cloud in the layered model at the current viewing angle; The current depth value of the point cloud corresponding to each screen pixel is compared with the corresponding depth value stored in the depth buffer. If the current depth value is less than or equal to the corresponding depth value stored in the depth buffer, the current point cloud is rendered.

9. The method according to claim 7, characterized in that The projected position coordinates are integrated with the real power construction environment information using an extended Kalman filter algorithm to predict and correct the projected position coordinates, including: The position and velocity of the dynamic target are used as the initial state variables, and the covariance matrix, process noise matrix and observation noise matrix are initialized; Establishing an observation model based on the coordinates of the feature points of the dynamic target object in the real power construction environment information; Establishing the Jacobian matrix of state quantity transfer and the Jacobian matrix of observation function, predicting the state quantity to obtain the predicted state quantity, and performing covariance prediction based on the Jacobian matrix of state quantity transfer to obtain the predicted covariance matrix; The Kalman gain is updated according to the Jacobian matrix of the observation function and the predicted covariance matrix, and the covariance matrix is ​​updated according to the updated Kalman gain and the predicted covariance matrix, and the state quantity is updated according to the updated Kalman gain, the updated covariance matrix and the predicted state quantity to obtain an estimated value of the projection position coordinate.

10. A layered model dynamic projection device based on augmented reality, characterized in that: include: The first acquisition module is used to collect RGB image data, depth image data and point cloud data of the power construction environment through multiple sensors; A model building module is used to perform data fusion and scene reconstruction based on the RGB image data, depth image data and point cloud data to obtain a scene model of the power construction environment; A division module, configured to divide the scene model into layers to obtain a layered model; An updating module, configured to update the hierarchical model in real time; The second acquisition module is used to control the acquisition of real power construction environment information through augmented reality equipment; The projection module is used to integrate the updated hierarchical model with the real power construction environment information so that the hierarchical model is dynamically projected into the real power construction environment.

Images