Automatic inspection method and system for interior of building based on VR and unmanned aerial vehicle

Abstract

The invention discloses a building interior automatic inspection method and system based on VR and an unmanned aerial vehicle, and the method comprises the following steps: obtaining the original data of multiple sensors, carrying out the data noise reduction and synchronization processing through adaptive Kalman filtering, extracting dynamic features through employing a tensor manifold learning method, and detecting the environment change through employing a dynamic feature topology analysis method. Environment reconstruction is carried out through a weighted graph convolutional network; carrying out model registration by adopting a graph matching network alignment method, extracting semantic features through a depth graph neural network, calculating an area change probability by utilizing a conditional random field model, and carrying out information fusion through a Bayesian reasoning network; performing path optimization based on an improved A * algorithm and dynamic planning, and generating a smooth trajectory by adopting a Bezier curve; performing rendering optimization through a hierarchical detail technology, and establishing an operation mapping relation; and performing state prediction and control instruction generation by adopting model prediction control. According to the invention, the automation level, safety and efficiency of inspection are improved.

Description

Technical Field

[0001] The present invention relates to an automatic inspection method, in particular to an automatic inspection method and system for interior of a building based on VR and unmanned aerial vehicles. Background Art

[0002] Internal inspection of buildings is a key link in ensuring the safe operation of building facilities. Traditional manual inspection methods have problems such as low efficiency, high cost, and high risk. With the increasing number and complexity of large-scale building facilities, higher requirements are placed on the accuracy, real-time and safety of inspections. The automatic inspection system based on VR and drones can realize remote operation and intelligent decision-making, effectively reduce personnel risks, and improve inspection efficiency and data collection quality. At the same time, this technology can build digital twin scenarios, support historical data retrospective analysis and predictive maintenance, which is of great significance to improving the level of intelligent operation and maintenance of building facilities.

[0003] At present, the internal inspection technology of buildings mainly focuses on environmental perception based on a single sensor, simple path planning and manual remote control operation. In the field of environmental perception, the sensor data processing method based on Kalman filtering is mainly used, but the fusion ability of multi-source heterogeneous data is limited. Path planning mostly uses traditional A* algorithm or RRT algorithm, which is difficult to adapt to dynamic environmental changes. The remote control system mainly relies on 2D image transmission and keyboard and mouse control interface, with poor operating experience and large delay. In terms of environmental reconstruction, the traditional registration method based on feature point matching is mainly used, which is difficult to handle the dynamic changes of complex scenes. Drone control mainly uses classic control algorithms such as PID, which have insufficient control accuracy and robustness in complex environments.

[0004] The existing technology has the following specific problems: First, in the processing of multi-source sensor data, traditional feature extraction methods are difficult to maintain the high-dimensional structural characteristics of the data, resulting in incomplete expression of environmental features. Second, the environmental change detection adopts a method based on Euclidean distance, which cannot effectively capture topological structure changes, affecting the recognition accuracy of dynamic features. Third, in the model registration process, traditional feature matching methods are difficult to handle local occlusion and dynamic changes, affecting the accuracy of scene reconstruction. Fourth, path planning fails to fully consider the distribution of environmental risks, and the generated trajectory has a curvature discontinuity problem, affecting the execution reliability. Fifth, the VR rendering system has serious delays when processing large-scale scenes, affecting the interactive experience. Sixth, the drone control system has poor adaptability to environmental disturbances, insufficient predictive control accuracy, and it is difficult to ensure the stable execution of inspection tasks. These technical problems seriously restrict the actual application effect of the automatic inspection system inside the building. Summary of the invention

[0005] The purpose of the invention is to provide an automatic inspection method for the interior of a building based on VR and drones, in order to solve at least one technical problem existing in the prior art.

[0006] Technical solution, according to one aspect of the present application, provides a method for automatic inspection of the interior of a building based on VR and drones, comprising the following steps:

[0007] Step S1, obtaining multi-sensor raw data, using adaptive Kalman filtering to perform data noise reduction and synchronization processing, and obtaining a preprocessed sensor data matrix; according to the preprocessed sensor data matrix, using the tensor manifold learning method to extract dynamic features, and obtaining an environmental feature matrix; based on the environmental feature matrix, using the dynamic feature topology analysis method to detect environmental changes, and obtaining an environmental change feature vector; according to the environmental feature matrix and the environmental change feature vector, using a weighted graph convolutional network to reconstruct the environment, and obtaining a real-time dynamic environment model;

[0008] Step S2, obtaining a real-time dynamic environment model and pre-stored BIM model data, using a graph matching network alignment method to perform model registration, and obtaining an alignment transformation matrix; extracting semantic features through a deep graph neural network based on the real-time dynamic environment model and the alignment transformation matrix to obtain a semantic feature map; based on the environmental change feature vector and the semantic feature map, using a conditional random field model to calculate the probability of regional changes to obtain a dynamic regional probability map; based on the semantic feature map and the dynamic regional probability map, information fusion is performed through a Bayesian inference network to obtain a semantically annotated scene representation;

[0009] Step S3, obtaining the pre-stored inspection task parameters and semantically annotated scene representation, processing them through the hierarchical task decomposition algorithm, and obtaining a subtask sequence; calculating the task risk coefficient according to the dynamic area probability map and the subtask sequence, and obtaining the task risk matrix; based on the task risk matrix and the semantically annotated scene representation, using the improved A* algorithm combined with dynamic programming to perform path optimization, and obtaining the optimal inspection path; according to the optimal inspection path and the dynamic area probability map, using the Bezier curve to generate a smooth trajectory and calculate the correction parameters, and obtaining the real-time trajectory correction parameters;

[0010] Step S4, obtaining a real-time dynamic environment model, simplifying data through adaptive meshing, and obtaining a simplified environment model; calculating an observation perspective sequence according to the optimal inspection path and the simplified environment model to obtain a perspective sequence; based on the simplified environment model and the perspective sequence, performing rendering optimization using hierarchical detail technology to obtain low-latency VR rendering data; establishing an operation mapping relationship according to the low-latency VR rendering data to obtain an interactive control instruction set;

[0011] Step S5, obtain real-time trajectory correction parameters and interactive control instruction sets, perform state prediction through model predictive control, and obtain a predicted state sequence; solve the quadratic programming problem based on the predicted state sequence and the optimal inspection path to obtain a UAV control instruction sequence; based on the UAV control instruction sequence, execute control instructions and collect feedback data to obtain system state feedback data.

[0012] According to another aspect of the present application, there is also provided a building interior automatic inspection system based on VR and drones, comprising:

[0013] at least one processor; and,

[0014] a memory communicatively connected to at least one of the processors; wherein,

[0015] The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the automatic inspection method for the interior of a building based on VR and a drone as described in any of the above technical solutions.

[0016] Beneficial effect: the present invention improves the automation level, safety and efficiency of inspection. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a flow chart of the method of the present invention.

[0018] Figure 2 It is a flow chart of step S1 of the present invention.

[0019] Figure 3 It is a flow chart of step S2 of the present invention.

[0020] Figure 4 It is a flow chart of step S3 of the present invention.

[0021] Figure 5 It is a flow chart of step S4 of the present invention.

[0022] Figure 6 It is a flow chart of step S5 of the present invention. DETAILED DESCRIPTION

[0023] like Figure 1 As shown, according to one aspect of the present application, a method for automatic inspection of the interior of a building based on VR and a drone is provided, comprising the following steps:

[0024] Step S1, obtaining multi-sensor raw data, using adaptive Kalman filtering to perform data noise reduction and synchronization processing, and obtaining a preprocessed sensor data matrix; according to the preprocessed sensor data matrix, using the tensor manifold learning method to extract dynamic features, and obtaining an environmental feature matrix; based on the environmental feature matrix, using the dynamic feature topology analysis method to detect environmental changes, and obtaining an environmental change feature vector; according to the environmental feature matrix and the environmental change feature vector, using a weighted graph convolutional network to reconstruct the environment, and obtaining a real-time dynamic environment model;

[0025] Step S2, obtaining a real-time dynamic environment model and pre-stored BIM model data, using a graph matching network alignment method to perform model registration, and obtaining an alignment transformation matrix; extracting semantic features through a deep graph neural network based on the real-time dynamic environment model and the alignment transformation matrix to obtain a semantic feature map; based on the environmental change feature vector and the semantic feature map, using a conditional random field model to calculate the probability of regional changes to obtain a dynamic regional probability map; based on the semantic feature map and the dynamic regional probability map, information fusion is performed through a Bayesian inference network to obtain a semantically annotated scene representation;

[0026] Step S3, obtaining the pre-stored inspection task parameters and semantically annotated scene representation, processing them through the hierarchical task decomposition algorithm, and obtaining a subtask sequence; calculating the task risk coefficient according to the dynamic area probability map and the subtask sequence, and obtaining the task risk matrix; based on the task risk matrix and the semantically annotated scene representation, using the improved A* algorithm combined with dynamic programming to perform path optimization, and obtaining the optimal inspection path; according to the optimal inspection path and the dynamic area probability map, using the Bezier curve to generate a smooth trajectory and calculate the correction parameters, and obtaining the real-time trajectory correction parameters;

[0027] Step S4, obtaining a real-time dynamic environment model, simplifying data through adaptive meshing, and obtaining a simplified environment model; calculating an observation perspective sequence according to the optimal inspection path and the simplified environment model to obtain a perspective sequence; based on the simplified environment model and the perspective sequence, performing rendering optimization using hierarchical detail technology to obtain low-latency VR rendering data; establishing an operation mapping relationship according to the low-latency VR rendering data to obtain an interactive control instruction set;

[0028] Step S5, obtain real-time trajectory correction parameters and interactive control instruction sets, perform state prediction through model predictive control, and obtain a predicted state sequence; solve the quadratic programming problem based on the predicted state sequence and the optimal inspection path to obtain a UAV control instruction sequence; based on the UAV control instruction sequence, execute control instructions and collect feedback data to obtain system state feedback data.

[0029] Through adaptive Kalman filtering and tensor manifold learning methods, efficient processing and feature extraction of multi-source heterogeneous sensor data are achieved, providing a reliable foundation for environmental perception. The environmental change detection method based on continuous homology analysis can accurately capture the dynamic characteristics of the environment and provide real-time risk assessment for path planning. The real-time environment model and the BIM model are accurately aligned through the graph matching network, providing rich semantic information for scene understanding. In terms of path planning, the improved A* algorithm combined with dynamic planning realizes the optimal path generation considering risk weights, and the smoothness and executableness of the trajectory are ensured by the Bezier curve generation algorithm. The VR interactive system uses hierarchical detail technology to achieve low-latency rendering, and provides an intuitive human-computer interaction interface combined with standardized control mapping. Finally, the drone control system based on model predictive control ensures the precise execution of inspection tasks. The overall solution significantly improves the automation level, safety and efficiency of internal building inspections through multi-level optimization strategies, and is particularly suitable for performing high-precision inspection tasks in complex and dynamic indoor environments. This solution not only solves the limitations of traditional inspection methods in complex environments, but also provides safer and more efficient inspection solutions through intelligent technical means, which has important practical application value.

[0030] like Figure 2 As shown, according to one aspect of the present application, step S1 is specifically:

[0031] Step S11, acquiring multi-sensor raw data including lidar point cloud data, depth camera data and inertial measurement unit data, and performing time series noise reduction processing on the data using an adaptive Kalman filter to obtain noise-reduced sensor data; performing time synchronization and spatial registration processing on the noise-reduced sensor data to obtain a preprocessed sensor data matrix;

[0032] The adaptive Kalman filter method is: the state estimation equation X(k|k) = X(k|k-1) + K(k)[Z(k)- H(k)X(k|k-1)]; the adaptive gain matrix K(k) = P(k|k-1)H(k) T [H(k)P(k|k-1)H(k) T + λ(k)R(k)]⁻¹; dynamic noise covarianceλ(k) = tr[C(k)] / tr[R(k)]; innovation covariance matrixC(k) = 1 / M ∑[v(i)v(i) T], i from k-M+1 to k; innovation sequence v(k) = Z(k) - H(k)X(k|k-1); state prediction equation X(k|k-1) = Φ(k,k-1)X(k-1|k-1); where: X(k|k) is the optimal state estimate at time k; X(k|k-1) is the state prediction value at time k; Z(k) is the measurement value at time k; H(k) is the measurement matrix; P(k|k-1) is the prediction error covariance matrix; R(k) is the measurement noise covariance matrix; M is the innovation window length; Φ(k,k-1) is the state transfer matrix; tr[·] represents the trace operation of the matrix.

[0033] Step S12, obtaining the preprocessed sensor data matrix, and reconstructing it into a fourth-order tensor including space dimension, time dimension, feature dimension and relationship dimension; performing nonlinear dimensionality reduction operation on the fourth-order tensor on the tensor manifold to obtain intermediate features that maintain the intrinsic structure of the data; solving the optimal feature representation through the Riemann optimization method based on the intermediate features to obtain the environmental feature matrix;

[0034] The algorithm for performing nonlinear dimensionality reduction operations on tensor manifolds to obtain intermediate features that preserve the intrinsic structure of the data is:

[0035] Feature extraction function F(T) = ∑[Wi·φi(T)]·exp(-γD(T,Mi)); where the tensor distance metric D(T,Mi) = ||T×1U1 T ×2U2 T ×3U3 T - Mi||F; orthogonal projection matrix update Ui = argmin∑||T×iUi T -Mi||F + α||Ui T Ui - I||F; local feature map φi(T) = σ(Vec(T×1Pi1×2Pi2×3Pi3)); where: T is the input multidimensional tensor; Mi is the reference point on the manifold; Wi is the adaptive weight; γ is the kernel function parameter; ×i represents the i-th modular product of the tensor; Ui is the i-th orthogonal basis matrix; ||·||F is the Frobenius norm; α is the regularization parameter; Pi is the projection matrix; Vec(·) represents tensor vectorization; σ(·) is a nonlinear activation function.

[0036] Step S13, obtaining the current environment feature matrix and the stored environment feature matrix of the previous moment, constructing a continuous homology structure in the feature space; performing time series analysis on the continuous homology structure to extract topological invariants; identifying key environmental change points according to changes in the topological invariants, and obtaining an environmental change feature vector;

[0037] The timing analysis of the continuous coherent structure is as follows:

[0038] The persistent homology characteristic function β(f,t) = rank(Hk(X≤t)) - rank(η(Hk(X≤t-ε)→Hk(X≤t))); where the distance function d(x,y) = ||f(x) - f(y)||2 + λ·KL(p(x)||p(y)); the submanifold decomposition X≤t = {x∈X | f(x)≤t}; the persistent graph G(V,E,w) = {(vi,vj,wij) | wij = exp(-d(vi,vj) / σ 2 )}; where: Hk represents the k-order homology group; η is the homology mapping; ε is the persistence parameter; f(x) is the point cloud feature function; p(x) is the local probability distribution; KL(·||·) is the KL divergence; σ is the Gaussian kernel parameter; rank(·) represents the rank calculation.

[0039] Step S14, obtain the environmental feature matrix and the environmental change feature vector, and construct a weighted graph structure; perform graph convolution operation on the weighted graph structure to fuse spatial information; integrate the time series information of the graph convolution result to obtain a real-time dynamic environment model.

[0040] By introducing an adaptive Kalman filter to process multi-sensor data, the adaptive fusion of sensor data with different sampling rates and different noise characteristics is realized, reducing the impact of data noise on subsequent processing; at the same time, the tensor manifold learning method is used for feature extraction to maintain the high-order structural relationship of the data. Compared with the traditional matrix decomposition method, it better captures the spatiotemporal correlation between sensor data. Through the dynamic feature topological analysis method, the topological features of environmental changes can be detected in real time, avoiding the problem of being susceptible to noise interference in traditional methods. Finally, the weighted graph convolutional network is used for environmental reconstruction, and the adaptive weight allocation mechanism ensures that the reconstruction results achieve a balance between local details and global consistency. This multi-level data processing architecture enables the system to maintain stable perception and reconstruction performance in complex dynamic environments, and can still ensure the accuracy and real-time performance of environmental reconstruction even when there is partial loss or interference of sensor data.

[0041] like Figure 3 As shown, according to one aspect of the present application, step S2 is specifically:

[0042] Step S21, obtaining a real-time dynamic environment model and pre-stored BIM model data, converting the two models into a graph structure representation with node attributes; for the converted graph structure, calculating the graph embedding vectors of nodes and edges through a graph attention network; based on the calculated graph embedding vectors, performing a soft matching operation to obtain an initial correspondence relationship; according to the initial correspondence relationship, calculating the final model alignment parameters through iterative optimization to obtain an alignment transformation matrix;

[0043] Perform matching operations, specifically: graph matching score S(X) = tr(X T KX) + vec(X) T Wvec(X); where the kernel matrix K(ia,jb) = exp(-||fi - fj|| 2 / σf 2 - ||pi - pj|| 2 / σp 2 ); spatial compatibility matrix W(ia,jb;kc,ld) = exp(-||(dia,jb - dkc,ld)|| 2 / σd 2 );Sinkhorn iteration X(t+1) = T(exp(βK·X(t)·K));where: X is the soft matching matrix; fi is the node feature; pi is the spatial position; dia,jb is the geodesic distance on the graph; T is the Sinkhorn normalization operator; β is the temperature parameter; vec(·) represents matrix vectorization; tr(·) represents the trace of the matrix.

[0044] Step S22, obtaining a real-time dynamic environment model and an alignment transformation matrix, performing spatial transformation on the real-time dynamic environment model to obtain an aligned environment model; constructing a multi-layer depth map structure for the aligned environment model; extracting local and global semantic features through a depth map neural network according to the multi-layer depth map structure; fusing the extracted local and global semantic features to obtain a semantic feature map;

[0045] Step S23, obtaining the environmental change feature vector and the semantic feature map, constructing the potential function of the conditional random field; calculating the state transition probability of each area in the scene according to the constructed potential function; performing time series smoothing on the calculated state transition probability; generating a dynamic area probability map representing the dynamic characteristics of the area according to the smoothed probability value;

[0046] Step S24, obtain the semantic feature map and the dynamic area probability map, and construct the node relationship of the Bayesian network; calculate the conditional probability distribution between the features according to the node relationship; perform probabilistic reasoning operations based on the conditional probability distribution; combine the reasoning results with the semantic information to obtain a semantically annotated scene representation.

[0047] The specific model of conditional probability is: conditional probability model P(Y|X) = exp(∑θkfk(X,Y)) / Z(X); where potential function fk(X,Y) = ∑wijφ(yi,yj)ψ(xi,xj) + ∑viϕ(yi)η(xi); normalization factor Z(X) = ∑exp(∑θkfk(X,Y ’)); parameter update θk(t+1) = θk(t) + α(EPdata[fk] - EPmodel[fk]); where: Y is the label sequence; X is the observation sequence; θk is the model parameter; wij is the weight between nodes; vi is the node weight; φ, ψ, ϕ, η are the feature functions; EPdata is the empirical expectation; EPmodel is the model expectation; α is the learning rate.

[0048] By introducing the graph matching network alignment method, the registration problem between the real-time dynamic environment model and the BIM model is solved. Compared with the traditional ICP algorithm, this method can handle a large range of initial pose deviations and has stronger robustness to local occlusion. The deep graph neural network is used to extract semantic features, which not only extracts the geometric features of the scene, but also integrates the semantic information, improving the accuracy of scene understanding. The introduction of the conditional random field model enables the system to consider spatial contextual relationships, effectively reducing misclassification, especially in areas where the environment is dynamically changing. Finally, information fusion is performed through the Bayesian inference network, realizing the probabilistic fusion of semantic information and geometric information, and improving the reliability and integrity of scene representation. This hierarchical semantic understanding framework enables the system to accurately identify and locate various objects in complex indoor environments, providing reliable semantic navigation information for subsequent path planning.

[0049] like Figure 4 As shown, according to one aspect of the present application, step S3 is specifically:

[0050] Step S31, obtaining pre-stored inspection task parameters and semantically annotated scene representations, extracting task constraints and scene semantic information; constructing a task dependency graph based on the extracted constraints and semantic information; performing a hierarchical clustering operation on the task dependency graph; and generating a subtask sequence containing priority information based on the clustering results;

[0051] Step S32, obtaining a dynamic area probability map and a subtask sequence, extracting a dynamic probability value of each subtask-related area; calculating an environmental uncertainty index based on the extracted dynamic probability value; weightedly combining the environmental uncertainty index with the difficulty of subtask execution; and generating a task risk matrix that characterizes the risk level of each subtask based on the weighted combination result;

[0052] Step S33, obtaining the task risk matrix and the scene representation of the semantic annotation, constructing a search space that takes risk weight into consideration; in the constructed search space, calculating the initial path using the improved A* algorithm; applying the dynamic programming method to the initial path for global optimization; generating the optimal inspection path that meets the constraint conditions based on the optimization results;

[0053] A* algorithm and dynamic programming, specifically: heuristic function h(n) = ω1·d(n,g) + ω2·r(n) + ω3·c(n); where risk assessment r(n) = ∑p(oi)·exp(-d(n,oi) / σ 2 ); cost function g(n) = min{g(m)+ d(m,n)·(1 + λ·r(n))}; dynamic programming value function V(s) = min{c(s,a) + γ·∑P(s ’ |s,a)V(s ’ )}; where: d(n,g) is the Euclidean distance from the node to the target; oi is the obstacle; p(oi) is the obstacle risk probability; c(n) is the node congestion; λ is the risk weight; γ is the discount factor; P(s ’ |s,a) are state transition probabilities; ω1, ω2, ω3 are adaptive weight coefficients.

[0054] Step S34, obtain the optimal inspection path and dynamic area probability map, extract the path key point sequence; calculate the Bezier curve control point parameters according to the key point sequence; generate a smooth trajectory curve based on the control point parameters; for the generated trajectory curve, calculate the real-time adjustment amount in combination with the dynamic area probability to obtain the real-time trajectory correction parameter.

[0055] The generation algorithm of the Bezier curve is as follows: Bezier curve equation B(t) = ∑(nk)·(1-t)^(nk)·t^k·Pk; where the control point optimization Pk* = argmin{∑||B ’ (ti) - vi|| 2 + α·∑||B ’’ (ti)|| 2 +β·R(ti)}; Risk weight function R(t) = ∑p(oi)·exp(-d(B(t),oi) / σ 2 ); Tangent vector constraint B ’ (ti) = λi·vi; where: n is the curve order; (nk) is the binomial coefficient; t∈[0,1] is the parameter; Pk is the control point; vi is the desired velocity direction; B ’ (t),B ’’ (t) are the first and second order derivatives of the curve respectively; α, β are smoothness weights; λi is the velocity scaling factor.

[0056] Through the hierarchical task decomposition algorithm, the complex inspection task is decomposed into a series of subtasks with clear priorities, avoiding the combinatorial explosion problem of task planning in traditional methods. The introduction of the calculation mechanism of the task risk coefficient not only considers the static risk of the environment, but also integrates the potential risks brought by dynamic changes, thereby improving the safety of task execution. The improved A* algorithm combines the dynamic planning method for path optimization. While ensuring the optimality of the path, it improves the planning efficiency in a dynamic environment by dynamically adjusting the heuristic function. The introduction of the Bezier curve realizes the smoothing of the trajectory, avoiding the mutation problem of the traditional piecewise linear path during execution. At the same time, through the calculation of the real-time trajectory correction parameters, the system can dynamically adjust the execution trajectory according to environmental changes, improving the adaptability and safety of the actual inspection process. This multi-level task planning and trajectory generation method significantly improves the execution efficiency and reliability of the inspection task, especially in scenarios with large dynamic changes in the environment.

[0057] like Figure 5 As shown, according to one aspect of the present application, step S4 is specifically:

[0058] Step S41, obtaining a real-time dynamic environment model, calculating the geometric complexity of each area of ​​the model; constructing an adaptive meshing strategy based on the calculated geometric complexity; performing meshing operations on the model to obtain a multi-resolution mesh structure; based on the multi-resolution mesh structure, retaining key geometric features to obtain a simplified environment model;

[0059] Step S42, obtaining an optimal inspection path and a simplified environment model, extracting key observation points on the path; calculating scene visibility analysis indicators for each key observation point; optimizing camera parameters according to the visibility analysis indicators; and generating a viewing angle sequence containing position and orientation information based on the optimized camera parameters;

[0060] Step S43, obtaining a simplified environment model and a view sequence, and constructing a scene hierarchical detail model; calculating the rendering priority of each part of the scene according to the observation position in the view sequence; dynamically adjusting the model detail level based on the rendering priority; and compressing the adjusted rendering data in real time to obtain low-latency VR rendering data;

[0061] Hierarchical detail rendering is as follows: level of detail function LOD(v) = floor(log2(d(v) / ε)); mesh simplification cost E(v) = ∑wiQi(v); visual importance weight w(v) = exp(-θ(v) / σθ 2 )·exp(-d(v) / σd 2), compression rate adaptation function R(b) = R0·(1 + α·||∇I(b)||), where: d(v) is the viewpoint distance; ε is the reference distance; Qi(v) is the quadratic error metric; θ(v) is the viewing angle; b is the image block; ∇I is the image gradient; R0 is the reference compression rate; α is the adaptation coefficient.

[0062] Step S44, obtaining low-latency VR rendering data, extracting interactive elements in the VR scene; establishing a correspondence between interactive operations and control instructions for the interactive elements; generating a standardized control mapping table based on the correspondence; generating an interactive control instruction set for drone control based on the control mapping table and the interaction rules.

[0063] Efficient VR rendering and interactive control are achieved through adaptive meshing and hierarchical detail technology. This method first realizes the intelligent simplification of the environment model based on the adaptive meshing of geometric complexity, significantly reducing the amount of data while maintaining key geometric features. In the process of generating the view sequence, the visibility analysis and camera parameter optimization are used to ensure the optimal coverage and visual effect of the observation view. The application of hierarchical detail technology enables the rendering process to dynamically adjust the model detail level according to the observation position, and further reduces the data transmission delay through the real-time compression algorithm. In terms of interactive control, by establishing a standardized control mapping relationship, the precise correspondence between VR operation and drone control instructions is achieved, providing an intuitive human-computer interaction interface. This method not only solves the problem of real-time rendering of large-scale scene data, but also ensures the smoothness of the interactive experience and the accuracy of control through multi-level optimization strategies. It is particularly suitable for remote control inspection tasks in complex indoor environments.

[0064] like Figure 6 As shown, according to one aspect of the present application, step S5 is specifically:

[0065] Step S51, obtaining real-time trajectory correction parameters and interactive control instruction sets, constructing a system state space model; calculating a multi-step prediction interval according to the state space model; performing rolling optimization calculation on the state evolution within the prediction interval; generating a prediction state sequence containing position, attitude and speed information based on the optimization calculation results;

[0066] Specifically: prediction state equation x(k+i|k) = Ax(k+i-1|k) + Bu(k+i-1|k) + w(k+i-1|k); cost function J = ∑||x(k+i|k) - xr||Q + ∑||u(k+i|k)||R; constraint condition C(x,u) = {g(x,u)≤0, h(x,u)=0}; adaptive prediction model A(k) = A0 + ΔA(k), where ΔA(k) = K·∑e(i)x(i)T / ||x(i)|| 2 ; Where: x is the state vector; u is the control input; w is the prediction error; xr is the reference trajectory; Q, R are weight matrices; g, h are inequality and equality constraints; K is the adaptive gain; e is the prediction error.

[0067] Step S52, obtaining the predicted state sequence and the optimal inspection path, establishing the state deviation evaluation index; constructing the objective function of the quadratic programming according to the evaluation index; applying the convex optimization method to the objective function to calculate the optimal control quantity; based on the optimal control quantity, generating the UAV control instruction sequence including thrust and torque instructions;

[0068] The objective function is min J = x T Qx + u T Ru + 2x T Nu; dynamic constraint x* = Ax + Bu; control constraints umin ≤ u ≤ umax, |u*| ≤ vmax; augmented Lagrangian function L(x,u,λ,μ) = J + λ T (Ax+Bu) +μ T max(0,g(x,u)); optimal control law u*(t) = -R⁻¹(B T λ(t) + N T x(t)); where: Q, R are the state and control weight matrices; N is the cross weight matrix; λ, μ are Lagrange multipliers; g(x,u) is the inequality constraint function; vmax is the control change rate limit. T represents the matrix transpose.

[0069] Step S53, obtain the UAV control instruction sequence, convert the control instruction into an actuator drive signal; collect the UAV motion state data according to the drive signal; perform real-time filtering and fusion processing on the collected state data; based on the processed data, generate system state feedback data including position, attitude and sensor state.

[0070] Precise control of the UAV is achieved through model predictive control and real-time feedback mechanism. The method first performs multi-step prediction based on the system state space model, and continuously updates the prediction results through the rolling optimization method, thereby improving the accuracy of state prediction. In the process of generating control instructions, by solving the quadratic programming problem considering multiple constraints, it is ensured that the control instructions both meet the dynamic constraints and minimize the control cost. The generation of actuator drive signals adopts real-time filtering and fusion processing to effectively suppress the influence of sensor noise. Through the real-time acquisition and processing of system state feedback data, a closed-loop control architecture is constructed, which enables the control system to quickly respond to environmental changes and mission requirements. This method not only improves the accuracy and stability of UAV control, but also enhances the robustness of the system through an adaptive adjustment mechanism. It is particularly suitable for performing precise inspection tasks in dynamically changing indoor environments.

[0071] According to one aspect of the present application, step S12 is specifically:

[0072] Step S121: Obtain the preprocessed sensor data matrix, extract the spatial position and timestamp information, and construct an initial space-time two-dimensional data structure; based on the initial two-dimensional data structure, calculate the correlation coefficient between the sensor data and generate a relationship weight matrix; based on the sensor type and data characteristics, construct a feature description vector; combine and reconstruct the space-time structure, the relationship weight matrix and the feature description vector to obtain a fourth-order feature tensor including space dimension, time dimension, feature dimension and relationship dimension.

[0073] Step S122: Obtain a fourth-order feature tensor and construct a tangent space mapping of the tensor manifold; calculate a local linear approximation matrix based on the tangent space mapping; perform principal geodesic analysis on the manifold based on the local linear approximation matrix to obtain nonlinear feature components; orthogonalize the nonlinear feature components to obtain a reduced-dimensional feature representation that maintains the intrinsic structure of the data, that is, to obtain intermediate features that maintain the intrinsic structure of the data.

[0074] Step S123: Obtain the reduced-dimensional feature representation and construct an objective function based on the Riemann metric; calculate the Riemann gradient and the Riemann-Hessian matrix according to the objective function; based on the Riemann gradient and the Riemann-Hessian matrix, execute the conjugate gradient method to solve the optimal parameters; substitute the optimal parameters into the feature mapping function to obtain the environmental feature matrix.

[0075] By constructing a fourth-order feature tensor and tensor manifold learning method, efficient feature extraction and dimensionality reduction of multi-source heterogeneous sensor data are achieved. Specifically, by first reconstructing the space-time structure, relationship weight matrix and feature description vector into a fourth-order tensor, the complete information of the four dimensions of space, time, features and relationships between the data is retained, avoiding the information loss of traditional dimensionality reduction methods when processing multidimensional data. Secondly, when performing nonlinear dimensionality reduction operations on tensor manifolds, the local geometric structure of the data is maintained by constructing tangent space mapping and local linear approximation matrices, so that the reduced dimensionality features can better express the essential characteristics of the original data. Finally, the optimal feature representation is solved by an optimization method based on Riemannian metric. The convergence and numerical stability of the feature extraction process are ensured by gradient descent and conjugate gradient method on Riemann manifolds, and the obtained environmental feature matrix has stronger expression ability and robustness. Compared with traditional linear dimensionality reduction methods, this method can better handle nonlinear and high-dimensional data, reduce computational complexity, and improve the accuracy of feature extraction.

[0076] According to one aspect of the present application, step S13 is specifically:

[0077] Step S131: Obtain the current environment feature matrix and the stored environment feature matrix of the previous moment, and calculate the difference measure between the feature matrices; construct a multi-scale simplicial complex sequence based on the difference measure; calculate the connected components and the hollow structure based on the simplicial complex sequence; organize the connected components and the hollow structure into a hierarchical continuous homology structure.

[0078] Step S132: Obtain the persistent homology structure and extract the zero-dimensional and one-dimensional homology group information; calculate the Betti number sequence based on the homology group information; construct a persistent barcode based on the Betti number sequence; perform stability analysis on the persistent barcode to obtain the topological invariant.

[0079] Step S133: Obtain topological invariants and construct an importance scoring function based on the feature life cycle; identify key topological events in temporal changes based on the scoring function; calculate significance indicators of environmental changes based on key topological events; map the significance indicators to the original feature space to obtain an environmental change feature vector.

[0080] The continuous homology analysis method is used to detect environmental changes, which can accurately capture the dynamic characteristics of the environment and reliably identify the change points. This method first constructs a multi-scale simplicial complex sequence and calculates the continuous homology structure, which can characterize the evolution process of environmental characteristics from a topological perspective. Compared with the traditional change detection method based on Euclidean distance, it has stronger geometric invariance and scale adaptability. By extracting zero-dimensional and one-dimensional homology group information and calculating the Betti number sequence, it can quantitatively characterize the connectivity and cavity structure changes of environmental characteristics, so as to accurately identify the key topological events in the environment. Based on the importance scoring mechanism of the feature life cycle, the short-term topological changes caused by noise can be effectively filtered out, and only the significant environmental change features are retained. The final environmental change feature vector contains both the change information of the spatial structure and the continuity of the temporal evolution, providing a reliable basis for subsequent environmental reconstruction and path planning.

[0081] According to one aspect of the present application, step S14 is specifically:

[0082] Step S141: Obtain the environment feature matrix and the environment change feature vector, and construct the node relationship of the time series graph; calculate the spatial adjacency matrix according to the node relationship; assign dynamic weights to the adjacency matrix based on the change feature vector; combine the dynamic weights and the adjacency relationship to construct a weighted graph structure.

[0083] Step S142: Obtain a weighted graph structure and design multi-layer graph convolution kernel parameters; perform weighted message passing according to the weighted graph structure and the graph convolution kernel; apply a nonlinear activation function to the passing result; retain multi-scale features through jump connections to obtain spatial aggregation features.

[0084] Step S143: Obtain weighted graph structure and spatial aggregation features, and construct a temporal attention module; fuse features of different time steps according to the attention weights; perform weighted graph convolution operations on the fused features; integrate graph structure information through residual connections to obtain spatiotemporal fusion features.

[0085] Step S144: Acquire spatiotemporal fusion features and weighted graph structure, and establish a decoder based on the graph structure; map the features to three-dimensional space according to the decoder; perform weighted graph convolution optimization on the mapping results; and obtain a real-time dynamic environment model through structural consistency constraints.

[0086] Efficient environment reconstruction is achieved through weighted graph convolutional networks, which effectively integrates spatial and temporal information. First, when constructing the weighted graph structure, the environmental change feature vector is used as the basis for dynamic weight allocation, so that the graph structure can adaptively reflect the local similarity and global connectivity of the environment. When performing graph convolution operations, the architecture of multi-layer graph convolution kernels and jump connections is adopted, which not only ensures the depth of feature extraction, but also avoids the gradient vanishing problem and enhances the expression ability of the model. The introduction of the temporal attention module enables the model to adaptively focus on important features at different time steps and improves the ability to capture dynamic changes. The stability and accuracy of the reconstruction results are ensured by residual connections and structural consistency constraints. This method can not only accurately reconstruct the static environment structure, but also effectively express the dynamic change characteristics of the environment, significantly improve the accuracy and real-time performance of environment reconstruction, and provide a reliable environment representation for subsequent navigation decisions.

[0087] According to one aspect of the present application, step S21 is specifically:

[0088] Step S211: Acquire the real-time dynamic environment model and pre-stored BIM model data, extract the geometric feature points and topological relationships in the model; construct local descriptors based on the geometric feature points; generate attribute-weighted nodes based on the local descriptors and topological relationships; encode the spatial and semantic relationships between nodes as edge attributes to obtain an attribute graph structure, i.e., a graph structure representation.

[0089] Step S212: Obtain the attribute graph structure and construct a calculation graph of the multi-head attention layer; according to the calculation graph, perform self-attention operations on node attributes; perform relational attention operations on edge attributes; fuse the results of node attention and relational attention to obtain a graph embedding vector.

[0090] Step S213: Obtain the graph embedding vector and calculate the similarity matrix between nodes; according to the similarity matrix, apply the Sinkhorn algorithm to solve the optimal transmission problem; based on the optimal transmission result, generate a probability matching matrix; filter through the confidence threshold to obtain the initial correspondence.

[0091] Step S214: obtain the initial corresponding relationship and construct a rigid transformation constraint based on the corresponding relationship; design an optimization objective function based on Lie algebra according to the constraint conditions; use the trust region method to iteratively solve the optimal transformation parameters; convert the optimal parameters into a homogeneous transformation form to obtain an alignment transformation matrix.

[0092] The real-time dynamic environment model and the BIM model are accurately aligned through the graph matching network. The method first encodes the geometric feature points and topological relationships into an attribute-weighted graph structure, which not only retains the geometric information of the model, but also introduces semantic relationships as edge attributes to improve the distinguishability of feature representation. The introduction of the multi-head attention mechanism enables the model to adaptively learn the importance weights of nodes and edges. Through the dual effects of self-attention and relational attention, the graph embedding vector is enhanced to express local details and global structures. When calculating the model correspondence, the Sinkhorn algorithm is used to solve the optimal transmission problem, which has better global consistency than the traditional matching method based on distance metric. Finally, through iterative optimization based on Lie algebra, the rigid transformation parameters are accurately solved to ensure the accuracy of the alignment result. This method can not only process large-scale point cloud data, but also adapt to the real-time registration requirements in dynamic environments, significantly improving the robustness and efficiency of model alignment.

[0093] According to one aspect of the present application, step S23 is specifically:

[0094] Step S231: Obtain the environmental change feature vector and semantic feature map, extract the regional semantic label and location information; construct the node potential function based on the semantic label; construct the edge potential function based on the location information and spatial relationship; combine the node potential function and the edge potential function to obtain the conditional random field potential function.

[0095] Step S232: Obtain the potential function of the conditional random field and construct a probability graph model based on the potential function; perform conditional probability decomposition according to the probability graph model; calculate local and global conditional probabilities based on the decomposition results; and obtain the conditional random field model through conditional probability integration.

[0096] Step S233: Obtain a conditional random field model, and calculate the state transition probability between regions based on the model; perform maximum likelihood estimation based on the state transition probability; update the probability distribution based on the likelihood estimation result; and obtain a state transition probability matrix through iterative optimization.

[0097] Step S234: obtain the state transition probability matrix and establish a time series probability filter; according to the filter, smooth the state transition sequence; based on the smoothing result, execute the spatial consistency constraint; obtain the dynamic area probability map through the probability distribution after time series smoothing.

[0098] The conditional random field model is used to achieve accurate probabilistic modeling of dynamic areas in the scene. First, the potential function constructed based on the environmental change feature vector and the semantic feature map not only considers the semantic attributes of the region, but also includes the spatial position relationship, so that the model can fully capture the structural characteristics of the scene. In the process of probabilistic modeling, the conditional probability decomposition and maximum likelihood estimation are used to effectively reduce the computational complexity while ensuring the accuracy of the probability estimation. The introduction of the temporal probability filter enables the model to smoothly process the state transition sequence, suppress the influence of temporary disturbances, and improve the stability of the probability prediction. By applying spatial consistency constraints, the continuity of the probability distribution of adjacent regions is ensured, and the fragmentation of the probability prediction results is avoided. This method can not only accurately identify the dynamic areas in the scene, but also quantitatively characterize the uncertainty of regional changes, providing a reliable risk assessment basis for subsequent path planning.

[0099] According to one aspect of the present application, step S33 is specifically:

[0100] Step S331: Obtain the task risk matrix and the semantically annotated scene representation, and extract the three-dimensional grid representation of the scene; calculate the accessibility index of each grid based on the grid representation; assign a risk weight to each reachable grid based on the risk matrix; combine the accessibility index and the risk weight to construct a search space that takes the risk weight into consideration.

[0101] Step S332: Obtain a search space that takes risk weights into consideration and construct an adaptive heuristic function; dynamically update the node expansion priority based on the heuristic function; perform path search using an improved A* algorithm based on a priority queue; and obtain an initial path by optimizing the search process through pruning.

[0102] Step S333: Obtain the initial path, construct a global cost function including path smoothness and risk constraints; establish a dynamic programming state transfer equation based on the global cost function; perform global dynamic programming optimization based on the state transfer equation; perform constraint verification on the optimization results to obtain a global optimization path.

[0103] Step S334: obtain the global optimization path and extract the key path points of the path; calculate the motion continuity constraints based on the path points; perform path refinement processing based on the constraints; obtain the optimal inspection path through feasibility verification and fine-tuning.

[0104] The optimal path planning considering risk weights is realized by combining the improved A* algorithm with dynamic programming. The method first constructs a search space based on risk weights. By combining the task risk matrix with the scene semantic information, the path planning process can adaptively avoid high-risk areas. The design of the adaptive heuristic function enables the search process to dynamically adjust the node expansion strategy according to the local environmental characteristics, significantly improving the search efficiency. In the global optimization stage, by constructing a cost function containing path smoothness and risk constraints, it is ensured that the planned path not only satisfies the motion constraints but also minimizes the risk exposure. The introduction of dynamic programming enables the optimization process to find the global optimal solution while considering multiple objectives, avoiding the problem of local optimality. Finally, through path refinement and feasibility verification, the actual executability of the planning results is guaranteed. This method can not only quickly generate safe inspection paths in complex environments, but also adjust the planning strategy in real time according to the dynamic changes of the environment, significantly improving the safety and efficiency of inspection tasks.

[0105] According to one aspect of the present application, step S34 is specifically:

[0106] Step S341: Obtain the optimal inspection path and dynamic area probability map, extract the turning position and orientation information of the path; calculate the curvature change characteristics according to the turning position; perform key point screening based on the curvature characteristics; obtain the path key point sequence through spatial sampling optimization.

[0107] Step S342: Obtain a sequence of key points on the path, and calculate the tangent vectors of adjacent key points; construct the endpoint constraints of the Bezier curve based on the tangent vectors; optimize the control point positions based on the endpoint constraints; and obtain the Bezier curve control point parameters by solving the geometric constraints.

[0108] Step S343: Obtain the control point parameters of the Bezier curve and construct a piecewise Bezier curve equation; calculate the continuity constraint of the trajectory according to the curve equation; perform curve fitting optimization based on the continuity constraint; and obtain a smooth trajectory curve by adjusting the parameters.

[0109] Step S344: Obtain a smooth trajectory curve and a dynamic area probability map, extract trajectory sampling points; calculate local risk assessment values ​​according to the sampling points; generate a trajectory offset vector based on the risk assessment; and obtain real-time trajectory correction parameters through dynamic weighted fusion.

[0110] The real-time correction of smooth trajectory is realized by Bezier curve generation algorithm. The method first extracts the key points of the path through curvature feature analysis, and ensures the rationality of the key point distribution based on spatial sampling optimization, avoiding the feature loss that may be caused by the traditional equidistant sampling method. When constructing the Bezier curve, the continuity and smoothness of the trajectory at the key points are ensured through tangent vector constraints and control point optimization. The fitting process of the piecewise Bezier curve adopts continuity constraints to ensure the continuity of high-order derivatives between adjacent segments and improve the execution stability of the trajectory. Finally, by incorporating the dynamic area probability map information into the trajectory correction process, real-time trajectory adjustment based on risk assessment is realized, enabling the UAV to dynamically avoid high-risk areas while maintaining the smoothness of the trajectory. This method not only improves the geometric quality of the trajectory, but also enhances the safety and robustness of the UAV movement. It is particularly suitable for performing precise inspection tasks in dynamically changing indoor environments.

[0111] According to one aspect of the present application, step S43 is specifically:

[0112] Step S431: Obtain a simplified environment model and a view sequence, extract the geometry and texture information of the model; construct a multi-resolution hierarchy based on the geometry and texture information; establish a geometry simplification rule based on the hierarchy and space division strategy; apply the geometry simplification rule to the model to obtain a scene hierarchical detail model.

[0113] Step S432: Obtain a scene hierarchical detail model and a viewing angle sequence, and calculate the viewing distance of each part of the scene according to the observation position; based on the viewing distance information, evaluate the visual importance of geometric details; combine the visual importance and computing resource constraints to construct a priority evaluation function; perform priority evaluation on each part of the scene to obtain a rendering priority table.

[0114] Step S433: Obtain the scene hierarchical detail model and rendering priority table, and dynamically select the appropriate detail level according to the priority; perform mesh simplification operations based on the selected detail level; perform topology optimization on the simplified mesh; and generate hierarchical rendering data based on the optimization results.

[0115] Step S434: Obtain hierarchical rendering data, and select an adaptive compression algorithm based on data features; encode geometry and texture data according to the compression algorithm; accelerate the compression process through parallel computing; organize the compression results to obtain low-latency VR rendering data.

[0116] The optimization of low-latency VR rendering is achieved through hierarchical detail technology. The method first constructs a multi-resolution hierarchy, and establishes adaptive simplification rules based on the analysis of geometric and texture information, which effectively reduces the complexity of the model. In the rendering priority calculation, by combining the view distance information and the visual importance evaluation, the dynamic adjustment of the scene details is realized, which ensures the visual effect and optimizes the allocation of computing resources. The mesh simplification operation adopts a topology optimization strategy, which significantly reduces the amount of geometric data while maintaining the visual quality of the model. Finally, the rendering data is encoded through an adaptive compression algorithm, and the compression process is accelerated by parallel computing, which significantly reduces the data transmission delay. This method not only solves the real-time rendering problem in VR scenes, but also ensures the rendering quality through a multi-level optimization strategy, providing operators with a smooth interactive experience.

[0117] According to one aspect of the present application, step S51 is specifically:

[0118] Step S511: Acquire real-time trajectory correction parameters and interactive control instruction sets to construct the UAV dynamics equation; establish a state space model based on the dynamics equation; calculate the system response characteristics based on the state space model; obtain the state space model parameters through parameter identification.

[0119] Step S512: Obtain state space model parameters and determine the prediction interval range; construct a state evolution equation based on the prediction interval; calculate the state transfer matrix based on the evolution equation; and obtain a state prediction model through constraint processing.

[0120] Step S513: Obtain a state prediction model and perform rolling optimization calculations; predict multi-step state evolution based on the optimization calculations; perform error compensation based on state evolution; and obtain optimized prediction results through dynamic updating.

[0121] Step S514: Obtain the optimized prediction results and apply the model predictive control algorithm; perform state correction according to the control algorithm; perform time series organization based on the correction results; and obtain the predicted state sequence through data fusion.

[0122] The accurate prediction of the state of the UAV is achieved through model predictive control. The method first establishes an accurate state space model through the dynamic equation, and uses parameter identification technology to improve the accuracy of the model. In the prediction process, by determining the appropriate prediction interval and constructing the state evolution equation, a reliable prediction of the future state of the UAV is achieved. The introduction of rolling optimization calculation enables the prediction process to continuously update the model parameters to adapt to the dynamic changes of the environment and system state. Through error compensation and dynamic update mechanisms, the accuracy and robustness of the prediction results are significantly improved. This method can not only accurately predict the motion state of the UAV, but also adjust the control strategy in time, providing a reliable decision-making basis for real-time trajectory correction.

[0123] According to one aspect of the present application, step S52 is specifically:

[0124] Step S521: Obtain the predicted state sequence and the optimal inspection path, extract the state variables and path reference values; calculate the position deviation and posture deviation according to the state variables and reference values; construct a time-varying weight matrix based on the deviation data; combine the weight matrix with the deviation data to obtain the state deviation evaluation index.

[0125] Step S522: Obtain state deviation evaluation indicators and establish UAV dynamic constraints; design control input restrictions based on dynamic constraints; construct an inequality constraint matrix based on control restrictions and state constraints; combine the constraint matrix with the evaluation indicators to obtain the quadratic programming objective function.

[0126] Step S523: Obtain the quadratic programming objective function and construct an augmented Lagrangian function; calculate the first-order optimality condition based on the augmented Lagrangian function; perform interior point method iterative solution based on the optimality condition; verify the convergence of the iterative result to obtain the optimal control quantity.

[0127] Step S524: Obtain the optimal control quantity and map the control quantity to the actuator space; calculate the thrust distribution plan according to the actuator characteristics; generate the actuator control signal based on the distribution plan; organize the control signal in time sequence to obtain the UAV control instruction sequence.

[0128] The accurate calculation of the UAV control instructions is achieved through the quadratic programming algorithm. The method first constructs a state evaluation index that takes into account the position deviation and attitude deviation, and dynamically adjusts the importance of different deviation terms through a time-varying weight matrix. In the constraint processing, the dynamic constraints and control input restrictions of the UAV are considered at the same time to ensure the executable nature of the control instructions. The quadratic programming problem is solved by the augmented Lagrangian method, which not only ensures the convergence of the solution but also improves the computational efficiency. Finally, the control instructions are ensured to be accurately executed through actuator space mapping and thrust distribution optimization. This method can not only generate the optimal control instruction sequence, but also ensure the stability and safety of the control process, significantly improving the control accuracy of the UAV in complex environments.

[0129] In this embodiment, tensor manifold learning (TML) is used in S1. Compared with time series matrix decomposition, the advantages are:

[0130] TML operates directly in high-dimensional tensor space, maintaining the original multi-dimensional structure of the data. However, time series matrix decomposition requires flattening high-dimensional data into a matrix, which inevitably loses the coupling information between dimensions. TML uses tensor product operations (T×1U1 T ×2U2 T ×3U3 T) maintains the interaction of features in each dimension and can better capture the spatiotemporal correlation of sensor data. TML introduces the nonlinear mapping function φi(T) and the kernel function exp(-γD(T,Mi)), which can handle the nonlinear characteristics of the data. The learning of nonlinear patterns is achieved through the local feature mapping φi(T) = σ(Vec(T×1Pi1×2Pi2×3Pi3)), while matrix decomposition is mainly based on linear assumptions. TML dynamically adjusts the importance of different features through adaptive weights Wi, so that the model can adapt to environmental changes. Orthogonal constraint term ||Ui T Ui - I||F ensures the stability of feature extraction and avoids feature degradation. TML uses the parallelism of tensor decomposition and has higher computational efficiency than layer-by-layer matrix decomposition. The selection of the reference point Mi reduces the computational complexity and achieves efficient feature extraction.

[0131] In step S13, the advantage of dynamic feature topological analysis (DFTA) over threshold segmentation is that DFTA captures the topological invariants of the data through the continuous homology feature function β(f, t), maintaining the essential characteristics of environmental changes. Traditional threshold segmentation only focuses on pixel-level differences, is easily affected by noise, and cannot identify topological structure changes. DFTA implements multi-scale analysis through the continuous parameter ε, and can detect environmental changes at different scales. Submanifold decomposition X≤t provides a continuous scale representation, while threshold segmentation can only work on a single scale. DFTA introduces the KL divergence term KL(p(x)||p(y)) to take into account the local probability distribution and can more accurately describe environmental changes. The distance function d(x, y) comprehensively considers geometric and statistical characteristics to improve the robustness of change detection. The continuous graph G(V, E, w) can dynamically update the environmental representation and adapt to the continuous changes of the environment. The Gaussian kernel parameter σ provides adaptive local feature extraction capabilities, which is better than static segmentation with a fixed threshold.

[0132] In another embodiment of the present application, step S21 may also be:

[0133] Step S21: Acquire the real-time dynamic environment model and pre-stored BIM model data, extract feature descriptors and topological relationships to construct a graph structure, and obtain an attribute graph model; according to the attribute graph model, calculate the graph embedding through a multi-layer graph attention network to obtain a graph embedding vector; based on the graph embedding vector, perform soft matching optimization and iterative solution to obtain an alignment transformation matrix.

[0134] Step S211: Acquire the real-time dynamic environment model and pre-stored BIM model data, extract the point cloud key points and geometric features, and obtain a feature point set; calculate the local geometric descriptor and semantic descriptor based on the feature point set to obtain a multimodal descriptor; based on the multimodal descriptor and the spatial position relationship, construct a graph structure with node and edge attributes to obtain an attribute graph model.

[0135] Step S212: Obtain the attribute graph model and the multimodal descriptor, construct a multi-head self-attention layer to calculate the node representation, and obtain the node embedding vector; according to the node embedding vector, calculate the edge representation through the edge attention network to obtain the edge embedding vector; based on the node embedding vector and the edge embedding vector, perform feature aggregation and nonlinear transformation to obtain the graph embedding vector.

[0136] Step S213: Obtain the graph embedding vector and the multimodal descriptor, calculate the cross-modal similarity matrix, and obtain the similarity distribution; according to the similarity distribution, solve the optimal transmission problem through Sinkhorn iteration to obtain the soft matching matrix; based on the soft matching matrix and the confidence threshold, perform consistency filtering to obtain the initial correspondence.

[0137] Step S214: Obtain the initial correspondence and multimodal descriptors, construct a rigid transformation constraint based on Lie group, and obtain the transformation constraint condition; according to the transformation constraint condition, optimize the transformation parameters by iterative nearest point algorithm to obtain the optimized transformation parameters; based on the optimized transformation parameters and geometric consistency check, perform transformation matrix calculation to obtain the alignment transformation matrix.

[0138] In another embodiment of the present application, S43 may also be:

[0139] Step S43: Obtain a simplified environment model and a viewing sequence, construct a hierarchical rendering scheduling model including geometric complexity classification and rendering priority, and obtain rendering scheduling parameters; according to the rendering scheduling parameters, perform adaptive LOD (Level of Detail) control and real-time performance monitoring to obtain rendering performance indicators; based on the rendering performance indicators and rendering scheduling parameters, obtain low-latency VR rendering data through adaptive compression and transmission optimization.

[0140] Step S431: Obtain a simplified environment model and a viewing sequence, calculate the geometric complexity score of each area of ​​the model, and obtain a complexity scoring matrix; according to the complexity scoring matrix, perform octree-based spatial partitioning to obtain a multi-level model structure; based on the multi-level model structure and a preset performance threshold, generate LOD levels through recursive subdivision to obtain a hierarchical detail model.

[0141] Step S432: Obtain the hierarchical detail model and view sequence, calculate visibility based on the frustum culling and occlusion culling algorithms, and obtain visibility data; calculate the rendering cost function based on the visibility data and the observation distance to obtain the rendering cost matrix; based on the rendering cost matrix and the system resource status, obtain the rendering scheduling parameters through the priority sorting algorithm.

[0142] Step S433: Obtain rendering scheduling parameters and hierarchical detail model, perform real-time frame rate monitoring and resource utilization analysis, and obtain performance monitoring data; dynamically adjust the LOD switching threshold according to the performance monitoring data to obtain LOD control parameters; based on the LOD control parameters, obtain an optimized rendering model through adaptive mesh simplification.

[0143] Step S434: Obtain an optimized rendering model and performance monitoring data, perform block-based geometry compression and texture compression, and obtain compressed rendering data; calculate transmission priority and cache strategy based on the compressed rendering data to obtain transmission control parameters; based on the transmission control parameters and bandwidth status, obtain low-latency VR rendering data through adaptive streaming.

[0144] In another embodiment of the present application, S51 may also be:

[0145] Step S51: obtain real-time trajectory correction parameters and interactive control instruction set, calculate system dynamics model parameters and response characteristics, and obtain the system response model; according to the system response model, obtain model correction parameters through online parameter identification and adaptive filtering; based on the model correction parameters and the system response model, perform state prediction and uncertainty quantification to obtain a predicted state sequence.

[0146] Step S511: obtain real-time trajectory correction parameters and interactive control instruction set, fit system parameters by least square method, and obtain initial model parameters; perform single-step response test according to the initial model parameters to obtain response test data; based on the response test data, update model parameters by recursive least square method to obtain system response model.

[0147] Step S512: Obtain the system response model and model correction parameters, construct a state equation taking uncertainty into account, and obtain a state prediction equation; calculate a multi-step prediction matrix based on the state prediction equation to obtain a state transfer matrix; based on the state transfer matrix and system constraints, obtain prediction model parameters through robustness analysis.

[0148] Step S513: Obtain prediction model parameters and state transfer matrix, perform Monte Carlo simulation to calculate uncertainty propagation, and obtain uncertainty estimation; according to the uncertainty estimation, obtain prediction interval parameters through adaptive prediction interval adjustment; based on the prediction interval parameters and historical prediction errors, perform online model correction to obtain optimized prediction results.

[0149] Step S514: Obtain optimized prediction results and uncertainty estimates, combine multi-model predictions through the Bayesian fusion algorithm, and obtain fused prediction results; calculate the prediction confidence interval based on the fused prediction results to obtain the prediction confidence; based on the prediction confidence and system state constraints, obtain the prediction state sequence through serialization processing.

[0150] According to another aspect of the present application, there is also provided a building interior automatic inspection system based on VR and drones, comprising:

[0151] at least one processor; and,

[0152] a memory communicatively connected to at least one of the processors; wherein,

[0153] The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the automatic inspection method for the interior of a building based on VR and a drone as described in any of the above technical solutions.

[0154] It should be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present invention will not further describe various possible combinations.

Claims

1. An automatic inspection method for interior of a building based on VR and drones, characterized in that: The steps include: Step S1, obtaining raw data from multiple sensors, performing data noise reduction and synchronization processing using an adaptive Kalman filter, and obtaining a preprocessed sensor data matrix; according to the preprocessed sensor data matrix, extracting dynamic features using a tensor manifold learning method, and obtaining an environmental feature matrix; Based on the environmental feature matrix, the dynamic feature topology analysis method is used to detect environmental changes and obtain the environmental change feature vector. According to the environmental feature matrix and the environmental change feature vector, the environment is reconstructed through the weighted graph convolution network to obtain a real-time dynamic environment model. Step S2, obtaining a real-time dynamic environment model and pre-stored BIM model data, using a graph matching network alignment method to perform model registration, and obtaining an alignment transformation matrix; extracting semantic features through a deep graph neural network based on the real-time dynamic environment model and the alignment transformation matrix to obtain a semantic feature map; based on the environmental change feature vector and the semantic feature map, using a conditional random field model to calculate the probability of regional changes to obtain a dynamic regional probability map; based on the semantic feature map and the dynamic regional probability map, information fusion is performed through a Bayesian inference network to obtain a semantically annotated scene representation; Step S3, obtaining the pre-stored inspection task parameters and semantically annotated scene representation, processing them through a hierarchical task decomposition algorithm, and obtaining a subtask sequence; According to the dynamic area probability map and subtask sequence, the task risk coefficient is calculated to obtain the task risk matrix; Based on the task risk matrix and semantically annotated scene representation, the improved A* algorithm combined with dynamic programming is used to optimize the path and obtain the optimal inspection path; According to the optimal inspection path and the dynamic area probability map, the Bezier curve is used to generate a smooth trajectory and calculate the correction parameters to obtain the real-time trajectory correction parameters; Step S4, obtaining a real-time dynamic environment model, simplifying data through adaptive meshing, and obtaining a simplified environment model; calculating an observation perspective sequence according to the optimal inspection path and the simplified environment model to obtain a perspective sequence; based on the simplified environment model and the perspective sequence, performing rendering optimization using hierarchical detail technology to obtain low-latency VR rendering data; establishing an operation mapping relationship according to the low-latency VR rendering data to obtain an interactive control instruction set; Step S5, obtain real-time trajectory correction parameters and interactive control instruction sets, perform state prediction through model predictive control, and obtain a predicted state sequence; solve the quadratic programming problem based on the predicted state sequence and the optimal inspection path to obtain a UAV control instruction sequence; based on the UAV control instruction sequence, execute control instructions and collect feedback data to obtain system state feedback data.

2. The method for automatic inspection of interior of a building based on VR and drone according to claim 1 is characterized in that: Step S1 is specifically as follows: Step S11, acquiring multi-sensor raw data including lidar point cloud data, depth camera data and inertial measurement unit data, and performing time series noise reduction processing on the data using an adaptive Kalman filter to obtain noise-reduced sensor data; performing time synchronization and spatial registration processing on the noise-reduced sensor data to obtain a preprocessed sensor data matrix; Step S12, obtaining the preprocessed sensor data matrix, and reconstructing it into a fourth-order tensor including space dimension, time dimension, feature dimension and relationship dimension; performing nonlinear dimensionality reduction operation on the fourth-order tensor on the tensor manifold to obtain intermediate features that maintain the intrinsic structure of the data; solving the optimal feature representation through the Riemann optimization method based on the intermediate features to obtain the environmental feature matrix; Step S13, obtaining the current environment feature matrix and the stored environment feature matrix of the previous moment, constructing a continuous homology structure in the feature space; performing time series analysis on the continuous homology structure to extract topological invariants; identifying key environmental change points according to changes in the topological invariants, and obtaining an environmental change feature vector; Step S14, obtain the environmental feature matrix and the environmental change feature vector, and construct a weighted graph structure; perform graph convolution operation on the weighted graph structure to fuse spatial information; integrate the time series information of the graph convolution result to obtain a real-time dynamic environment model.

3. The method for automatic inspection of interior of a building based on VR and drone according to claim 1 is characterized in that: Step S2 is specifically as follows: Step S21, obtaining a real-time dynamic environment model and pre-stored BIM model data, converting the two models into a graph structure representation with node attributes; for the converted graph structure, calculating the graph embedding vectors of nodes and edges through a graph attention network; based on the calculated graph embedding vectors, performing a soft matching operation to obtain an initial correspondence relationship; According to the initial correspondence, the final model alignment parameters are calculated through iterative optimization to obtain the alignment transformation matrix; Step S22, obtaining a real-time dynamic environment model and an alignment transformation matrix, performing spatial transformation on the real-time dynamic environment model to obtain an aligned environment model; constructing a multi-layer depth map structure for the aligned environment model; and extracting local and global semantic features through a deep map neural network based on the multi-layer depth map structure; The extracted local and global semantic features are fused to obtain a semantic feature map; Step S23, obtaining the environment change feature vector and the semantic feature map, constructing the potential function of the conditional random field; and calculating the state transition probability of each area in the scene according to the constructed potential function; The calculated state transition probability is subjected to time series smoothing processing; a dynamic region probability map representing the dynamic characteristics of the region is generated according to the probability value after smoothing processing; Step S24, obtain the semantic feature map and the dynamic area probability map, and construct the node relationship of the Bayesian network; calculate the conditional probability distribution between the features according to the node relationship; perform probabilistic reasoning operations based on the conditional probability distribution; combine the reasoning results with the semantic information to obtain a semantically annotated scene representation.

4. The method for automatic inspection of interior of a building based on VR and drone according to claim 1 is characterized in that: Step S3 is specifically as follows: Step S31, obtaining pre-stored inspection task parameters and semantically annotated scene representations, extracting task constraints and scene semantic information; constructing a task dependency graph based on the extracted constraints and semantic information; Perform hierarchical clustering operations on the task dependency graph; based on the clustering results, generate a subtask sequence containing priority information; Step S32, obtaining a dynamic region probability map and a subtask sequence, and extracting a dynamic probability value of each subtask-related region; Calculate environmental uncertainty index based on the extracted dynamic probability value; The environmental uncertainty index and the difficulty of subtask execution are weighted together; Based on the weighted combination results, a task risk matrix is ​​generated to characterize the risk level of each subtask; Step S33: Obtain the task risk matrix and the semantically annotated scene representation, and construct a search space that takes risk weights into consideration; In the constructed search space, the improved A* algorithm is used to calculate the initial path; The initial path is globally optimized using the dynamic programming method; based on the optimization results, the optimal inspection path that meets the constraints is generated; Step S34, obtaining the optimal inspection path and the dynamic area probability map, extracting the path key point sequence; and calculating the Bezier curve control point parameters according to the key point sequence; Based on the control point parameters, a smooth trajectory curve is generated; for the generated trajectory curve, the real-time adjustment amount is calculated in combination with the dynamic area probability to obtain the real-time trajectory correction parameter.

5. The method for automatic inspection of interior of a building based on VR and drone according to claim 1 is characterized in that: Step S4 is specifically as follows: Step S41, obtaining a real-time dynamic environment model, calculating the geometric complexity of each area of ​​the model; constructing an adaptive grid division strategy according to the calculated geometric complexity; Perform meshing operations on the model to obtain a multi-resolution mesh structure; Based on the multi-resolution grid structure, the key geometric features are retained to obtain a simplified environment model; Step S42, obtaining an optimal inspection path and a simplified environment model, extracting key observation points on the path; calculating scene visibility analysis indicators for each key observation point; optimizing camera parameters according to the visibility analysis indicators; and generating a viewing angle sequence containing position and orientation information based on the optimized camera parameters; Step S43, obtaining a simplified environment model and a view sequence, and constructing a scene hierarchical detail model; and calculating the rendering priority of each part of the scene according to the observation position in the view sequence; Dynamically adjust the model detail level based on rendering priority; compress the adjusted rendering data in real time to obtain low-latency VR rendering data; Step S44: obtaining low-latency VR rendering data and extracting interactive elements in the VR scene; For interactive elements, establish the correspondence between interactive operations and control instructions; According to the corresponding relationship, a standardized control mapping table is generated; based on the control mapping table and the interaction rules, an interactive control instruction set for UAV control is generated.

6. The method for automatic inspection of interior of a building based on VR and drone according to claim 1 is characterized in that: Step S5 is specifically as follows: Step S51, obtaining real-time trajectory correction parameters and interactive control instruction sets, constructing a system state space model; calculating a multi-step prediction interval according to the state space model; performing rolling optimization calculation on the state evolution within the prediction interval; generating a prediction state sequence containing position, attitude and speed information based on the optimization calculation results; Step S52, obtaining the predicted state sequence and the optimal inspection path, establishing the state deviation evaluation index; constructing the objective function of the quadratic programming according to the evaluation index; applying the convex optimization method to the objective function to calculate the optimal control quantity; based on the optimal control quantity, generating the UAV control instruction sequence including thrust and torque instructions; Step S53, obtaining a UAV control instruction sequence, converting the control instruction into an actuator drive signal; collecting the UAV motion state data according to the drive signal; and performing real-time filtering and fusion processing on the collected state data; Based on the processed data, system status feedback data including position, attitude and sensor status is generated.

7. The method for automatic inspection of interior of a building based on VR and drone according to claim 1 is characterized in that: Step S12 is specifically as follows: Step S121, obtaining the preprocessed sensor data matrix, extracting the spatial position and timestamp information, and constructing an initial space-time two-dimensional data structure; according to the initial two-dimensional data structure, calculating the correlation coefficient between the sensor data, and generating a relationship weight matrix; constructing a feature description vector based on the sensor type and data features; The space-time structure, relationship weight matrix and feature description vector are combined and reconstructed to obtain a fourth-order feature tensor including space dimension, time dimension, feature dimension and relationship dimension; Step S122, obtaining a fourth-order characteristic tensor, constructing a tangent space mapping of the tensor manifold; and calculating a local linear approximation matrix according to the tangent space mapping; Based on the local linear approximation matrix, the principal geodesic analysis on the manifold is performed to obtain the nonlinear characteristic components; Orthogonalize the nonlinear feature components to obtain a reduced-dimensional feature representation that maintains the intrinsic structure of the data, that is, to obtain an intermediate feature that maintains the intrinsic structure of the data; Step S123, obtain the reduced-dimensional feature representation, and construct an objective function based on the Riemann metric; according to the objective function, calculate the Riemann gradient and the Riemann-Hessian matrix; based on the Riemann gradient and the Riemann-Hessian matrix, execute the conjugate gradient method to solve the optimal parameters; substitute the optimal parameters into the feature mapping function to obtain the environmental feature matrix.

8. The method for automatic inspection of interior of a building based on VR and drone according to claim 1 is characterized in that: Step S14 is specifically as follows: Step S141, obtaining an environmental feature matrix and an environmental change feature vector, constructing a time series graph node relationship; calculating a spatial adjacency matrix based on the node relationship; and assigning a dynamic weight to the adjacency matrix based on the change feature vector; Combine dynamic weights and adjacency relationships to build a weighted graph structure; Step S142, obtaining a weighted graph structure and designing multi-layer graph convolution kernel parameters; performing weighted message passing according to the weighted graph structure and the graph convolution kernel; applying a nonlinear activation function to the passing result; retaining multi-scale features through skip connections to obtain spatial aggregation features; Step S143, obtaining weighted graph structure and spatial aggregation features, constructing a temporal attention module; fusing features of different time steps according to the attention weights; performing weighted graph convolution operations on the fused features; integrating graph structure information through residual connections to obtain spatiotemporal fusion features; Step S144, obtaining spatiotemporal fusion features and weighted graph structure, and establishing a decoder based on the graph structure; according to the decoder, mapping the features to three-dimensional space; Weighted graph convolution optimization is performed on the mapping results; through structural consistency constraints, a real-time dynamic environment model is obtained.

9. The method for automatic inspection of interior of a building based on VR and drone according to claim 1 is characterized in that: Step S21 is specifically as follows: Step S211, obtaining a real-time dynamic environment model and pre-stored BIM model data, extracting geometric feature points and topological relationships in the model; constructing local descriptors based on the geometric feature points; generating attribute-weighted nodes based on the local descriptors and topological relationships; encoding the spatial and semantic relationships between nodes as edge attributes to obtain an attribute graph structure, i.e., a graph structure representation; Step S212: Obtain the attribute graph structure and construct a calculation graph of the multi-head attention layer; perform self-attention operation on the node attributes according to the calculation graph; perform relational attention operation on the edge attributes; fuse the results of node attention and relational attention to obtain a graph embedding vector; Step S213, obtaining a graph embedding vector and calculating a similarity matrix between nodes; According to the similarity matrix, the Sinkhorn algorithm is applied to solve the optimal transmission problem; based on the optimal transmission result, a probability matching matrix is ​​generated; through confidence threshold filtering, the initial correspondence is obtained; Step S214, obtain the initial corresponding relationship, construct a rigid transformation constraint based on the corresponding relationship; design an optimization objective function based on Lie algebra according to the constraint conditions; use the trust region method to iteratively solve the optimal transformation parameters; convert the optimal parameters into a homogeneous transformation form to obtain an alignment transformation matrix.

10. An automatic inspection system for interior of a building based on VR and drones, characterized in that: include: at least one processor; as well as, a memory communicatively connected to at least one of the processors; wherein, The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the automatic inspection method for the interior of a building based on VR and a drone as described in any one of claims 1 to 9.

Images