Method for performing high-resolution reconstruction on ground data for VR / AR (Virtual Reality / Augmented Reality) operation

Abstract

The invention discloses a VR / AR operation-oriented high-resolution reconstruction method for ground data, and the method comprises the steps: constructing an ST-SAN model which comprises a support vector machine-recursive feature elimination SVM-RFE module, a Conv-LSTM layer, a soft attention layer and a full connection layer; the method comprises the following steps of: extracting features which are most significant to prediction in an input vector by support vector machine-recursive feature elimination (SVM-RFE), capturing the spatial-temporal correlation in data by a Conv-LSTM layer, dynamically adjusting the importance of each feature by a soft attention layer, and reconstructing the input ranging data by a full connection layer.

Description

Technical Field

[0001] The present invention belongs to the field of data processing, and specifically, is a method for improving data quality and reconstructing high resolution of ground data observed by an unmanned aerial vehicle for VR / AR operations. Background of the Invention

[0003] Using VR / AR to operate drones to detect the ground terrain layer height map and measure the ground in combination with the semantic analysis results of the image is a low-altitude economic direction with a relatively wide application range. In this very valuable direction, there are two common preprocessing problems:

[0004] 1. When existing drones measure the distance to the ground, they generally use laser rangefinders. The laser rangefinder measures a straight-line distance. If the drone maintains a stable horizontal posture, then this straight-line distance is indeed the actual height of the drone to the current ground. However, due to the influence of the atmosphere, it is difficult for the drone to maintain a stable horizontal posture, so this value is often interfered by noise.

[0005] 2. In order to improve efficiency, the flight speed of drones is generally relatively fast, and the spatial intervals they measure are generally relatively large. Sometimes, in order to accurately display the ground conditions on the observer's VR / AR display, or to make more accurate measurements, it is necessary to reconstruct the data at high resolution.

[0006] Due to noise interference, when using drones to realize various low-altitude economic applications, it is necessary to improve the data quality and reconstruct high resolution of the ground data observed from the drone's perspective. Summary of the invention

[0007] Based on existing artificial intelligence, drone technology, and virtual reality technology, the present invention proposes a method for improving data quality and reconstructing high resolution of ground data observed from the perspective of a drone.

[0008] The method for high-resolution reconstruction of ground data for VR / AR operation of the present invention comprises the following steps: collecting ranging data and auxiliary data; constructing an ST-SAN model for reconstructing the ranging data; training the ST-SAN model using the collected ranging data and auxiliary data; obtaining a trained ST-SAN model; and inputting the ranging data to be reconstructed and the auxiliary data into the trained ST-SAN model for reconstructing the ranging data.

[0009] Further, the ST-SAN model includes a support vector machine-recursive feature elimination SVM-RFE module, a Conv-LSTM layer, a soft attention layer and a fully connected layer connected in sequence;

[0010] The ranging data and auxiliary data are input into the support vector machine-recursive feature elimination SVM-RFE for feature selection to extract the most meaningful features for prediction. The selected features are input into the Conv-LSTM layer to capture the spatiotemporal correlation in the data; then they are input into the soft attention layer to dynamically adjust the importance of each feature; finally, the features processed by the soft attention layer are input into the fully connected layer to generate the reconstructed ranging data.

[0011] Furthermore, in the soft attention layer, the weight associated with each position in the input sequence is calculated based on the given query q, and then the input sequence is weighted summed according to the weight to generate a context vector that the ST-SAN model pays attention to, which serves as the output of the soft attention layer.

[0012] Given a query q and a sequence of key-value pairs {(k1, v1), (k2, v2), ..., (k n , v n )};

[0013] The specific process of generating the context vector is as follows:

[0014] (1) Calculate the attention scores through a scoring function score(·)

[0015] score(q,k i )=softmax(q·k i T )

[0016] Among them, score(q, k i ) represents the query q and key k i The attention score between i T Indicates key k i The transpose of , · represents the dot product operation of the vector, and the softmax() function is used to normalize the attention score to make it conform to the probability distribution;

[0017] (2) Calculate the weighted sum of attention scores:

[0018]

[0019] Among them, Attention(q,{k i},{v i}) indicates that according to the query q, the value sequence {v i}The context vector obtained by weighted summation.

[0020] Furthermore, the Adam optimizer is used to train the ST-SAN model.

[0021] Furthermore, in the process of training the ST-SAN model, the mean square error is selected as the loss function.

[0022] Furthermore, the cross-validation method was used to verify the generalization ability of the ST-SAN model.

[0023] Furthermore, in the ablation experiment, the CNN part in the Conv-LSTM layer used 3 convolutional layers, the convolution kernel size was 3x3, the activation function was ReLU, the number of convolution kernels were 64, 128 and 256 respectively, and the pooling layer used a 2x2 maximum pooling operation; the LSTM part used a bidirectional LSTM with a hidden state dimension of 256, and the dimension of the input sequence was 128.

[0024] Furthermore, the auxiliary data refers to data that will affect the ranging data during the ranging process. The auxiliary data includes the altimeter data Dhigh of the drone, the meteorological variables MD obtained by the meteorological sensor, the drone attitude data DP during measurement, and the ground semantic information NLP recognized in the current image of the drone;

[0025] The meteorological variable MD includes wind direction and wind speed; the UAV attitude data includes coordinate data of the UAV's built-in GPS, the UAV's pitch angle, speed and angular velocity.

[0026] Beneficial effect: The method for high-resolution reconstruction of ground data for VR / AR operation of the present invention adopts a support vector machine-recursive feature elimination SVM-RFE module, a Conv-LSTM layer, a soft attention layer and a fully connected layer; the support vector machine-recursive feature elimination SVM-RFE extracts the most meaningful features for prediction in the input vector, the Conv-LSTM layer captures the spatiotemporal correlation in the data, the soft attention layer dynamically adjusts the importance of each feature, and then the input ranging data is reconstructed through the fully connected layer. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a network structure diagram of the model of the present invention;

[0028] Figure 2 It is the Conv-LSTM structure diagram in the present invention. DETAILED DESCRIPTION

[0029] The method of the present invention for high-resolution reconstruction of ground data for VR / AR operation specifically refers to reconstruction of ranging data observed by drones; using laser rangefinder data, drone altimeter, meteorological variables, drone attitude data, and image-based semantic data as inputs of the ST-SAN model to estimate the ground height data (e.g., floor height map) for drone inspections. By building a model and establishing the relationship between these variables, the estimation of ground height data can be achieved.

[0030] The present invention combines the soft attention mechanism with Conv-LSTM, proposes a space-time soft attention network model ST-SAN (Space-Time Soft Attention Network, ST-SAN) model, and adds the support vector machine-recursive feature elimination (SVM-RFE) method to the ST-SAN model to explain the importance of features. The ST-SAN model proposed in the present invention constructs the relationship between the laser rangefinder data and other auxiliary data, and the formula is as follows:

[0031] NewHigh=f(OHigh,Dhigh,MD,DP,NLP)

[0032] Among them, NewHigh represents the newly estimated ground height data;

[0033] OHigh indicates the original distance measurement data obtained by the laser rangefinder on the drone;

[0034] Dhigh refers to the drone's built-in altimeter, which is relative to the horizontal plane of the entire earth, not the height to the current ground.

[0035] MD represents the meteorological variables obtained by the meteorological sensors on the drone, specifically the information on wind direction and wind speed in three dimensions, a total of six parameters. The meteorological data here are not weather forecast data.

[0036] DP stands for drone attitude data, including 3 coordinate data of the drone's built-in GPS, 3 attitude angles such as the drone's built-in pitch angle, and 6 data such as the drone's built-in speed and angular velocity, a total of 12 parameters;

[0037] NLP represents the ground semantic information identified from the drone image at the current position of the drone. The categories of semantic information include 8 basic categories formulated by the National Remote Sensing Center, such as water surface, grassland, and road, plus 1 unrecognizable category, a total of 9 categories. In terms of specific algorithms, the reference for the recognition algorithm of ground semantic information NLP here is (Hu Kai et al., Multi-Scale Feature Aggregation Network for Water Area Segmentation. RemoteSensing. 2022, 14, 206.). The data here is the data recognized later after the drone returns. In other words, this algorithm is suitable for reconstructing data after drone inspection, rather than real-time implementation.

[0038] f() represents the ST-SAN model of the present invention, which models the relationship between output and input data.

[0039] The ST-SAN model combines the soft attention mechanism with Conv-LSTM, and adds the support vector machine-recursive feature elimination (SVM-RFE) method to the model to explain the importance of features.

[0040] like Figure 1 As shown, first, the ST-SAN model of the present invention inputs a data set of multi-dimensional and multi-spatiotemporal features into the model, performs feature selection through support vector machine-recursive feature elimination SVM-RFE, and extracts the most meaningful features for prediction. This step ensures that the model can focus on the most critical features for data prediction of the laser rangefinder, thereby simplifying the data dimension and improving the model performance and the interpretability of the model. Then, the selected features are input into the Conv-LSTM layer to capture the spatiotemporal correlation in the data. Conv-LSTM combines the advantages of convolutional neural networks and long short-term memory networks, and can simultaneously process the spatial and temporal features of the data, thereby improving the model's ability to recognize complex data patterns.

[0041] On the basis of the Conv-LSTM layer, the soft attention mechanism is applied to highlight key features by assigning different weights. The soft attention mechanism can dynamically adjust the importance of each feature, so that the model pays more attention to those features that have a significant impact on the prediction results, thereby improving the accuracy of the prediction. Finally, the features processed by the soft attention layer are input to the fully connected layer to generate the prediction results of the laser rangefinder data. The fully connected layer converts the features output by the soft attention layer into the final prediction value through weighted summation.

[0042] 1. Conv-LSTM

[0043] The Convolutional Long Short-Term Memory Network (Conv-LSTM) model introduces convolution calculation based on the LSTM model, which enables the model to obtain time series relationships and extract spatial features similar to convolution layers. Its structure diagram is shown below: Figure 2 As shown, it can be expressed as follows.

[0044]

[0045]

[0046]

[0047]

[0048]

[0049] In the formula,

[0050] * indicates convolution operation; X t is the input of the neuron at time t; in this specific implementation, Xt is specifically a vector representation of (OHigh, Dhigh, MD, DP, NLP);

[0051] H t-1 is the output of the neuron at time t-1;

[0052] W xi , W xf , W xc and W xo It is the convolution operation on X t Weight;

[0053] W hi , W hf , W hc and W ho Yes H t-1 The weight of

[0054] b i 、b f 、b c and b o is the bias value for the convolution operation.

[0055] Among them, i t Control the input gate, use the sigmoid function to select new input data, and the output of the previous neuron to determine the input information of the neuron. t Controls the forget gate, the input parameters are the same as the input gate, and can determine how much information from the previous neuron's output enters the neuron. tis the cell state of the neuron at time t. At the same time, new data is input, and the two are reorganized into a new cell state. Finally, the output gate outputs the cell state in a certain proportion through gate control, that is, the output H of the current cell is obtained. t It should be noted that X, C, H, i, f, and o are all three-dimensional tensors, and their last two dimensions represent the spatial information of rows and columns.

[0056] In this way, the convolutional long short-term memory network Conv-LSTM can obtain spatiotemporal sequence features, enabling the network to better solve spatiotemporal sequence prediction problems.

[0057] 2. Soft Attention Mechanism

[0058] The soft attention mechanism is a commonly used technique in deep learning. It allows the model to dynamically focus on different parts of the input sequence when processing sequence data, thereby improving the performance of the model. In the soft attention mechanism, each input part is assigned an attention weight, which is a real number between 0 and 1, indicating the importance of the part in the output. The sum of all attention weights is equal to 1, which can be achieved by normalizing the attention weights. The principle of the soft attention mechanism is to calculate the weights associated with each position in the input sequence based on a given query, and then perform a weighted summation of the input sequence according to these weights to generate a context vector that the model pays attention to. This context vector can be used to guide the model to make better predictions or generation in specific tasks. Its core formula is as follows:

[0059] Given a query q, a sequence of key-value pairs {(k1, v1), (k2, v2), ..., (k n , v n )}, the calculation process of the soft attention mechanism is as follows:

[0060] [1]. Calculate attention scores:

[0061]

[0062] Among them, score(q, k i ) represents the query q and key k i The attention score between i T Indicates key k i The transpose of , represents the dot product operation of the vector, and the softmax() function is used to normalize the attention score to make it conform to the probability distribution.

[0063] [2] Calculate the weighted sum:

[0064]

[0065] Among them, Attention(q,{k i},{v i}) indicates that according to the query q, the value sequence {v i}The context vector obtained by weighted summation.

[0066] Through the above two steps, the soft attention mechanism dynamically calculates the attention weights of different positions according to the given query and key-value pair sequence, and combines the value sequence to obtain the final context vector, thereby improving the performance and accuracy of the model.

[0067] The soft attention mechanism is selected in this invention mainly based on the following reasons:

[0068] [1]. Effectiveness in processing spatiotemporal data:

[0069] The soft attention mechanism can dynamically focus on different parts of the input data, which is especially important when processing spatiotemporal data. Here, when a drone is patrolling, the ground will not change much except for moving objects on the ground, that is, its spatial data is generally stable, so the soft attention mechanism can help the model better capture these characteristics, thereby improving prediction accuracy.

[0070] [2] Advantages of combining with Conv-LSTM:

[0071] Conv-LSTM can effectively process spatiotemporal sequence data, but it may have certain limitations in capturing long-term dependencies. The soft attention mechanism can help Conv-LSTM better capture long-term dependencies by weighting the inputs of different time steps to enhance the model's memory and selection capabilities.

[0072] [3]. Computational efficiency and model complexity:

[0073] Compared with some complex attention mechanisms, the soft attention mechanism is simpler in computational complexity. For the processing of large-scale data sets, this can reduce the consumption of computing resources while still maintaining good performance.

[0074] In summary, the reason why the present invention chooses the soft attention mechanism is mainly based on its effectiveness in processing spatiotemporal data, the advantages of combining with Conv-LSTM, and the balance between computational efficiency and model complexity. These characteristics make the soft attention mechanism an ideal choice for building the ST-SAN model, thereby better achieving the goal of estimating ground data during each inspection.

[0075] 3. SVM-RFE

[0076] SVM-RFE (Support Vector Machine-Recursive Feature Elimination) is a feature selection algorithm that combines the ideas of support vector machine (SVM) and recursive feature elimination (RFE). Support vector machine is used to evaluate the importance of features, and recursive feature elimination is used to iteratively select and exclude features until a preset stopping condition is reached.

[0077] Support vector machine (SVM) is a supervised learning algorithm used for binary classification and regression tasks. In binary classification problems, SVM separates samples of different categories by finding an optimal hyperplane in the feature space. When training SVM, the learned weight vector can reflect the importance of the feature. Generally speaking, the larger the absolute value of the feature weight, the greater its influence on the classification decision.

[0078] Recursive feature elimination (RFE) is a feature selection technique whose basic idea is to select feature subsets by continuously building a model and removing the least important features. It does this by removing features with smaller weights in each iteration until a predetermined stopping condition is reached.

[0079] Based on the above two points, SVM-RFE combines the classification performance of SVM and the feature selection ability of RFE. It can automatically sort and eliminate features according to their weights without manually specifying the importance of features, effectively reducing the number of features and improving the operating efficiency and generalization ability of the model. This method has potential advantages in reducing the risk of overfitting, improving model interpretability, and performing feature selection on large-scale data sets.

[0080] In the training process of the ST-SAN model of the present invention, the present invention first asked the drone pilot to collect 900,000 points of data, including 100,000 of each of 9 types, with heights ranging from 10 to 500 meters and wind speeds ranging from level 1 to level 3. Then 9,000 data were marked by the pilot as a test set, and the rest were training sets. The present invention conducted 100 rounds of iterative training and continuously monitored the performance of the model. Based on the performance of the validation set, the study adjusted the early stopping strategy to ensure that the model achieves optimal performance during the training process. The study adopted the Adam optimizer with good convergence performance, and its initial learning rate was set to 0.001, and dynamically adjusted according to the performance of the validation set. In the selection of the loss function, the ST-SAN model of the present invention uses the mean squared error (MSE) to effectively measure the gap between the predicted value and the true value. During the training process, the batch size was set to 32 to ensure the stable training of the ST-SAN model, and the back propagation algorithm was used to minimize the loss function. The model parameters were continuously optimized through the training data to improve the prediction accuracy and accurately capture the patterns and laws in the data. During the validation phase, the model was evaluated using a variety of performance metrics to ensure robustness and high accuracy on different data sets. In addition, a cross-validation method was used to further verify the generalization ability of the model, prevent overfitting, and ensure that the model has good prediction performance in practical applications. The entire process is designed to maximize the use of spatiotemporal data and improve the accuracy and reliability of ground height data prediction.

[0081] In addition, in the ablation experiment, the CNN part of Conv-LSTM uses 3 convolution layers, the convolution kernel size is 3x3, the activation function is ReLU, the number of convolution kernels is 64, 128 and 256 respectively, and the pooling layer uses a 2x2 maximum pooling operation. The LSTM part uses a bidirectional LSTM with a hidden state dimension of 256, and the dimension of the input sequence is 128. The soft attention mechanism uses a single-layer attention mechanism, the input and output vector dimensions are both 256, and the attention weight is calculated by dot product. In each combination, the study uses the same data set for training and testing to ensure the comparability of the results.

[0082] In summary, the ST-SAN model proposed in the present invention has the following advantages:

[0083] (1) Integrated spatiotemporal feature extraction and temporal dependency modeling: Conv-LSTM combines the advantages of CNN and LSTM, and can simultaneously capture spatial features and temporal dependencies in spatiotemporal sequence data. This means that the model can effectively extract information about spatiotemporal changes from the data, enabling it to more accurately understand the patterns and trends of spatiotemporal sequence data.

[0084] (2) Dynamically focus on different parts of the data: The soft attention mechanism allows the model to dynamically adjust the degree of attention according to different parts of the input data. This means that the model can selectively focus on certain parts based on the importance and contextual information of different parts of the input data, thereby more effectively utilizing the key information in the data and improving the performance and generalization ability of the model.

[0085] Improved model interpretability and feature selection capabilities: SVM-RFE can help the model automatically select the most representative feature subset, improving the generalization ability and operating efficiency of the model. At the same time, it provides model interpretability, making the model's decision-making process more interpretable and understandable. This means that the present invention can better understand the working principle of the model and adjust and optimize the model as needed.

[0086] By implementing the present invention, users can improve the data quality and reconstruct high resolution of ground data observed from the perspective of the drone when using the drone to realize various low-altitude economic applications under noise interference.

Claims

1. A method for reconstructing ground data, characterized in that: The method comprises the following steps: collecting ranging data and auxiliary data; Construct ST-SAN model; The ST-SAN model is trained using the collected ranging data and auxiliary data to obtain a trained ST-SAN model; The ranging data to be reconstructed and the auxiliary data are input into the trained ST-SAN model to reconstruct the ranging data.

2. The method for reconstructing ground data according to claim 1, characterized in that: The ST-SAN model includes a support vector machine-recursive feature elimination SVM-RFE module, a Conv-LSTM layer, a soft attention layer and a fully connected layer connected in sequence; The ranging data and auxiliary data are input into the support vector machine-recursive feature elimination SVM-RFE for feature selection. The selected features are input into the Conv-LSTM layer to capture the spatiotemporal correlation in the data, and then input into the soft attention layer to dynamically adjust the importance of each feature. Finally, the features processed by the soft attention layer are input into the fully connected layer to generate the reconstructed ranging data.

3. The method for reconstructing ground data according to claim 2, characterized in that: In the soft attention layer, the weight associated with each position in the input sequence is calculated based on the given query q, and then the input sequence is weighted summed according to the weight to produce a context vector that the ST-SAN model pays attention to, which is used as the output of the soft attention layer.

4. The method for reconstructing ground data according to claim 3, characterized in that: Given a query q and a sequence of key-value pairs {(k1,v1),(k2,v2),...,(k n ,v n )}; The specific process of generating the context vector is as follows: (1) Calculate attention scores: Among them, score(q,k i ) represents the query q and key k i The attention score between Indicates key k i The transpose of , · represents the dot product operation of the vector, and the softmax() function is used to normalize the attention score to make it conform to the probability distribution; (2) Calculate the weighted sum of attention scores: Among them, Attention(q,{k i },{v i }) indicates that according to the query q, the value sequence {v i }The context vector obtained by weighted summation.

5. The method for reconstructing ground data according to claim 1, characterized in that: The Adam optimizer is used to train the ST-SAN model.

6. The method for reconstructing ground data according to claim 1, characterized in that: In the process of training the ST-SAN model, the mean square error is selected as the loss function.

7. The method for reconstructing ground data according to claim 1, characterized in that: The cross-validation method was used to verify the generalization ability of the ST-SAN model.

8. The method for reconstructing ground data according to claim 2, characterized in that: In the ablation experiment, the CNN part of Conv-LSTM uses 3 convolution layers, the convolution kernel size is 3x3, the activation function is ReLU, the number of convolution kernels is 64, 128 and 256 respectively, and the pooling layer uses a 2x2 maximum pooling operation; The LSTM part uses a bidirectional LSTM with a hidden state dimension of 256 and an input sequence dimension of 128.

9. The method for reconstructing ground data according to claim 1, characterized in that: The auxiliary data refers to data that will affect the ranging data during the ranging process.

10. The method for reconstructing ground data according to claim 1, characterized in that: The auxiliary data includes the altimeter data Dhigh of the drone, the meteorological variables MD obtained by the meteorological sensor, the drone attitude data DP during measurement, and the ground semantic information NLP recognized in the current image of the drone; The meteorological variables MD include wind direction and wind speed; The drone attitude data includes the coordinate data of the drone's own GPS, the drone's pitch angle, speed and angular velocity.

Images