A data assimilation method and system for virtual-real integrated simulation
Through the multimodal fusion of UWB/RTK and RGB image data and the combination of double-layer LSTM and Kalman filtering, the problem of the difference between the state of the virtual and real model in virtual and real simulation is solved, and high-precision virtual and real simulation is achieved.
Patent Information
- Application Number
- CN202411088015.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-08-09
AI Technical Summary
The state difference between the virtual model and the real model in the existing virtual and real simulation experiments leads to simulation failure, and the existing methods do not fully utilize multimodal data in model state prediction, which cannot meet the requirements of high dynamic characteristics and high precision.
UWB or RTK devices are used to obtain real-time position coordinate data and RGB character image data captured by multiple cameras. State estimation is performed through data preprocessing and a double-layer LSTM neural network model, and state correction is performed by combining ensemble Kalman filtering to realize virtual and real simulation high-precision model construction.
Through multimodal data fusion and state correction, high-precision model construction of virtual and real simulation and state prediction in dynamic environments are realized, improving the credibility and operation speed of the simulation.
Smart Images

Figure CN118629094B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of electronic digital data processing, and in particular to a data assimilation method and system for virtual-real integrated simulation. Background Art
[0002] Traditional data assimilation algorithms have been widely used in modeling fields such as atmospheric environment, water conservancy, and geophysics, but they are rarely used in simulation experiments. At present, in the simulation experiment combining virtual and real, there is a data fault caused by the asynchronous operation of the virtual digital world and the physical world simulation process, which makes the accuracy and real-time of the virtual model's position and posture, operating status, simulation process, etc. lag behind the real world, seriously affecting the credibility and running speed of the simulation; at the same time, limited by the level of modeling and simulation technology, the existing test simulation system is difficult to accurately locate and predict the nonlinear complex human behavior model, and fails to adapt to the high dynamic characteristics of the simulation test process. There are problems such as the single posture recognition and positioning means, low virtual model state prediction accuracy, and high dynamic operation delay.
[0003] At present, most of the data assimilation methods for experimental simulation use sequential data assimilation methods to combine real-time observation data. They mainly use machine learning to predict model states, and then combine Kalman filtering, ensemble Kalman filtering, Bayesian filtering and other methods to continuously correct the optimal state of the virtual model to continuously approach the real model, and finally complete the state update of the virtual simulation model. Other schemes use continuous data assimilation methods, which mainly use real observations and model state data within a certain period of time, and fit the model's predicted trajectory to the real observation data through methods such as three-dimensional variation or four-dimensional variation.
[0004] In the equipment test simulation scenario of virtual-real integration, the virtual world that runs synchronously with the physical world has high requirements for the real-time and reliability of the simulation model. The existing continuous data assimilation method focuses on the historical observation and model data within a certain period of time. As the twin system runs, the complex behavior changes of the model will cause the state difference between the virtual model and the real model to continue to expand, resulting in simulation failure. The filtering-based sequential assimilation method combines real-time observation data and model state to estimate the state of the twin virtual model, but it mostly uses machine learning methods to predict the model state, and does not fully utilize the multi-modal data of the real model such as RTK, UWB, RGB images, etc. for position and attitude positioning and model construction. The effect of model construction and the prediction accuracy of state estimation cannot meet the differentiated needs of virtual-real integration simulation, and the update and correction method of the twin model lacks further optimization. Summary of the invention
[0005] In view of this, the present invention provides a data assimilation method and system for virtual-reality integrated simulation, which can utilize real-time observation data and prediction data to achieve highly reliable virtual-reality integrated simulation experiments through data preprocessing, state estimation, state updating and other methods.
[0006] To this end, the present invention provides the following technical solutions:
[0007] In one aspect, the present invention discloses a data assimilation method for virtual-real integrated simulation, the method comprising:
[0008] Acquire real-time position coordinate data collected by a UWB or RTK device, and perform data preprocessing on the real-time position coordinate data to obtain positioning data;
[0009] Acquire RGB character image data captured by multiple cameras, and perform data preprocessing on the RGB character image data to obtain key point data of human body position and posture;
[0010] The positioning data is fused with the human body position and posture key point data to obtain the human body posture information in the real world;
[0011] Inputting the posture information of the human body in the real world into a state estimation model based on a double-layer LSTM neural network model to estimate the future posture state, and obtaining predicted posture information;
[0012] Based on the predicted posture information and the posture information of the human body in the real world, state correction is performed through ensemble Kalman filtering to obtain a corrected posture state;
[0013] The corrected posture state and the posture information of the human body in the real world are used as inputs of the state estimation model to obtain the predicted posture information at the next moment.
[0014] Furthermore, real-time position coordinate data collected by the UWB or RTK device is obtained, and data preprocessing is performed on the real-time position coordinate data to obtain positioning data, including:
[0015] Record the real-time position coordinate data collected by UWB or RTK equipment, and use The sub-Lagrangian interpolation function eliminates the phenomenon of missing data values;
[0016] The non-local mean interpolation method is used to denoise the data after missing value processing to obtain missing and noise-free positioning data.
[0017] Furthermore, the non-local mean interpolation method is used to denoise the data after missing value processing, including:
[0018] Select the target denoising point and search point , and Centered search window and search point The neighborhood window centered on ;
[0019] By weight factor To express and Similarity of neighborhood:
[0020] ;
[0021] in:
[0022] ;
[0023] ;
[0024] is the normalization coefficient, is the smoothing parameter, is the neighborhood window range, is the Gaussian weighted Euclidean distance, is the standard deviation of the Gaussian kernel;
[0025] pass The similarity of the point neighborhood is calculated by weighting its value :
[0026] ;
[0027] Slide the second neighborhood window to get the next similarity measurement point , and similarly calculate and Similarity and weighted values of neighborhood windows;
[0028] The value obtained after denoising the target denoising point using the non-local mean method NLM:
[0029] ;
[0030] in, It is the UWB data corresponding to the target in the positioning data.
[0031] Furthermore, RGB character image data captured by multiple cameras is obtained, and data preprocessing is performed on the RGB character image data to obtain key point data of human body position and posture, including:
[0032] The collected RGB person image data is input into a trained neural network for posture recognition, and the human body position and posture key point data are output; the neural network is a multi-view 3D posture recognition network.
[0033] Furthermore, the neural network is a VoxelPose network; the neural network performs posture recognition, including: obtaining a cube space positioning proposal of the target through a voxel proposal network CPN module of the VoxelPose network; obtaining spatial coordinate information of each key point of the target of each proposal through a posture regression network PRN module of the VoxelPose network, each key point representing a human joint or a human body part.
[0034] Furthermore, the positioning data is fused with the human body position and posture key point data to obtain the human body posture information in the real world, including:
[0035] For the human body position and posture key point data, select key points , assign weight ;
[0036] The UWB data corresponding to the target in the positioning data Assign weight , and ;
[0037] Calculate the fused position based on the weights:
[0038] ;
[0039] Calculate the offset of fused position and posture data:
[0040] ;
[0041] Correct the posture positioning of other key points according to the offset to obtain the posture information of the human body in the real world:
[0042] ;
[0043] in, , etc. represent the positions of other key points of the body and the offset The sum represents the corrected positioning data, and the position data are expressed in three-dimensional space. Coordinate representation.
[0044] Furthermore, the initialization prediction process of the state estimation model includes:
[0045] Training the first layer of LSTM: Starting from the time series window, the data in the window is fed into the first layer of LSTM for processing, using the observed partial sequence Data and Labels Train the first layer of LSTM, where As a label for the current round;
[0046] Obtaining the optimal model weights: In each training cycle, by calculating The predicted value of Verify the loss between
[0047] Training the second LSTM layer: In the validation step of the first LSTM layer, determine the number of training cycles with the minimum loss, use the same number of training cycles, and the same weights and bias vectors to train the second LSTM layer; use the trained second LSTM layer to train the input sequence Make predictions, predict Pose status information for each time step ;
[0048] Data correction: Change the predicted value Input the Kalman filter for correction and get the correction value .
[0049] Furthermore, the prediction process of the state estimation model includes:
[0050] Using the input sequence and label of the previous time step Train the first layer of LSTM and With the incoming observations Weighted combination , put it into the sliding window as a new sequence value, and predict the next time step.
[0051] Further, based on the predicted posture information and the posture information of the human body in the real world, state correction is performed through ensemble Kalman filtering to obtain a corrected posture state, including:
[0052] (1) Update the covariance matrix :
[0053] Using the prediction sequence state set Estimate the mean of the predicted states ;
[0054] Calculate the deviation of the state : ;
[0055] in, It is iThe deviation between the predicted state of each set member and the mean of the predicted state;
[0056] Calculate the covariance matrix : ;
[0057] in, represents the estimated value of the state covariance matrix during the prediction update phase;
[0058] (2) Calculate the Kalman gain : ;
[0059] in, is the covariance matrix of the predicted state, is the observation matrix, which is used to describe the mapping relationship between the state vector and the observation vector; is the covariance matrix of the observation noise;
[0060] (3) Correction prediction Status, get the corrected value : ;
[0061] in, is the Kalman gain, The time step is When , the observation value of the pose information is collected, is the observation matrix.
[0062] In another aspect, the present invention further provides a data assimilation system for virtual-real integrated simulation, the system comprising:
[0063] A preprocessing module is used to obtain real-time position coordinate data collected by a UWB or RTK device, and perform data preprocessing on the real-time position coordinate data to obtain positioning data; obtain RGB human image data captured by multiple cameras, and perform data preprocessing on the RGB human image data to obtain human body position and posture key point data; perform data fusion on the positioning data and the human body position and posture key point data to obtain human body position and posture information in the real world;
[0064] A posture prediction module is used to input the posture information of the human body in the real world into a state estimation model based on a double-layer LSTM neural network model to estimate the future posture state and obtain predicted posture information;
[0065] A posture correction module, used for performing state correction by ensemble Kalman filtering based on the predicted posture information and the posture information of the human body in the real world to obtain a corrected posture state;
[0066] The model updating module is used to use the corrected posture state and the posture information of the human body in the real world as inputs of the state estimation model to obtain the predicted posture information at the next moment.
[0067] Advantages and positive effects of the present invention:
[0068] The present invention obtains accurate posture observation information of the human body in the real physical world by preprocessing multimodal data, and then obtains the prediction information of the human body posture data at the next moment through a posture state prediction method based on a double-layer LSTM. The virtual model is continuously corrected in accordance with the real-time observation and prediction information to complete the data assimilation process of the model, thus realizing the construction of a high-precision model in the context of combining virtual and real. The present invention adopts a data preprocessing method. On the one hand, the positioning data is collected by using an RTK / UWB positioning device, and then the data is preprocessed by Lagrange interpolation and non-local mean method to obtain missing and noise-free positioning data. On the other hand, the RGB image collected by the visual sensor is used to obtain the human body position and posture key point data through the posture recognition network VoxelPose, and the two are multi-modal data fused to finally obtain the precise posture data of the human body in the real world; then the precise posture data of the human body in the real world is input into a posture state prediction network based on a double-layer LSTM, and the precise state prediction information of the model in a dynamic environment is obtained through continuous training iteration and prediction, and then the predicted posture information is input into an ensemble Kalman filter algorithm, and the state correction process of the model is completed through processes such as updating the covariance matrix and the Kalman gain, and finally the information of the correction model is combined with the real and self information as the input of the state estimation at the next moment, thereby forming an effective closed loop of the prediction and correction process. BRIEF DESCRIPTION OF THE DRAWINGS
[0069] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0070] FIG1 is a block diagram of data stream transmission according to an embodiment of the present invention;
[0071] FIG2 is a diagram of a process of preprocessing collected data in an embodiment of the present invention;
[0072] FIG3 is a flowchart of NLM data denoising process according to an embodiment of the present invention;
[0073] FIG4 is a flow chart of VoxelPose network posture recognition in an embodiment of the present invention;
[0074] FIG5 is a diagram of a hybrid architecture of a double-layer LSTM neural network and an ensemble Kalman filter in an embodiment of the present invention;
[0075] FIG6 is a structural diagram of an LSTM recurrent network according to an embodiment of the present invention;
[0076] FIG7 is a diagram of the LSTM hidden layer structure in an embodiment of the present invention;
[0077] FIG8 is a prediction scheme diagram of a double-layer LSTM time series prediction model in an embodiment of the present invention;
[0078] FIG9 is a flow chart of a double-layer LSTM time series prediction model according to an embodiment of the present invention;
[0079] Fig.10 This is a flow chart of a state correction of the collective Kalman filter in an embodiment of the present invention. DETAILED DESCRIPTION
[0080] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0081] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0082] In the context of virtual-real combination simulation experiments, the existing modeling and simulation technology level is difficult to ensure the synchronous operation of the virtual digital environment and the real event process, resulting in the accuracy and real-time performance of the virtual model's position and posture, operating status, simulation process, etc. lagging behind the real world, seriously affecting the credibility and running speed of the simulation. In this regard, the present invention proposes a data assimilation method and system for virtual-real combination integrated simulation, which effectively promotes the confidence of virtual-real combination simulation, and eliminates the data gap between the physical environment and the virtual simulation system by real-time fusion of real measurement data. The method includes three parts: data preprocessing, state estimation and state correction. First, the present invention fuses UWB / RTK and camera device perception data in a data preprocessing manner to achieve the output of accurate posture data of the human body in the real world. Then, the accurate posture data of the human body in the real world completes the prediction of the future posture state through the state estimation model; the posture state is corrected by combining the predicted posture information and the accurate posture data of the human body in the real world. Finally, the corrected posture state is fed back to the state estimation model to guide the state estimation model to predict the state information more accurately.
[0083] like Figure 1 As shown, a data assimilation method for virtual-real integrated simulation in an embodiment of the present invention can combine state observation values and state prediction values to achieve accurate estimation and prediction of posture states through data assimilation technology, and specifically includes the following steps:
[0084] Step 1: Data preprocessing:
[0085] Preprocess the collected real-world data. There are two main data sources: real-time position coordinate data collected by UWB (UltraWideband, ultra-wideband is a carrier-free communication technology that uses nanosecond to microsecond non-sinusoidal narrow pulses to transmit data) / RTK (Real-time kinematic, carrier phase difference technology, which is a real-time differential method for processing the carrier phase observations of two measurement stations. The carrier phase collected by the base station is sent to the user receiver for coordinate difference calculation) and RGB (RGB color mode is a color standard in the industry. Various colors are obtained by changing the three color channels of red (R), green (G), and blue (B) and superimposing them with each other) character images captured by multiple cameras to estimate character posture data;
[0086] Figure 2The present invention demonstrates the process of preprocessing and collecting real-world data, which mainly includes two parts: the first part is to preprocess the real-time position coordinate data collected by UWB / RTK and other devices. This step aims to solve the problems of missing values, outliers and redundancy that may exist in the data to ensure the accuracy and real-time nature of the position data. The second part is to estimate the character's posture through RGB character images captured by multiple cameras and extract posture data from them. Finally, the two parts of data are fused and unified to obtain complete character posture data.
[0087] The real-world location data acquisition method of the present invention is mainly based on UWB, RTK and other methods. The UWB / RTK device can be bound to the corresponding position of the back waist to facilitate the fusion with the mid-hip key point closest to the position in the posture data (the "mid-hip" key point usually refers to a virtual key point located in the middle of the human hip. It is not an actual detectable body part, but a reference point used to assist in understanding the posture and movement of the human body). The UWB / RTK device can be used to collect real-time location coordinate data. The data stream collected by UWB / RTK can be described as:
[0088] (1)
[0089] in is the tag number of UWB / RTK, To locate the coordinate data, The time of data collection.
[0090] In the posture estimation part, the posture information of the target is obtained through the RGB image data collected by multiple cameras. The data stream collected by the camera can also be described as:
[0091] (2)
[0092] in, The ID of each target in the image, the ID and the tag number of the UWB device Correspondingly, the camera captures the target image carrying the UWB device. It is the image data collected by multiple cameras. The time of data collection.
[0093] To obtain the target pose information, follow these steps:
[0094] Step 1.1: Acquire real-time position coordinate data collected by a UWB or RTK device, and perform data preprocessing on the real-time position coordinate data to obtain positioning data.
[0095] The specific implementation includes:
[0096] Step 1.1.1: Record the location data collected by the UWB / RTK device and use The sub-Lagrangian interpolation function eliminates the phenomenon of missing data values:
[0097] (3)
[0098] in , is the position coordinate data obtained from the time series. Substitute the missing position data into the function to get the approximate value of the missing value corresponding to the point , in three-dimensional space, any two-dimensional coordinates can be processed by this function.
[0099] Step 1.1.2: Use NLM (Non-Local Means) to denoise the data after missing value processing.
[0100] Suppose the set of noisy data after missing value processing is , the denoised data set is , the noise data set is , we can get:
[0101] (4)
[0102] For the point that needs to be denoised, calculate its similarity with the neighboring points, calculate the weight based on the similarity and sum it up to get the denoised value of the point, such as Figure 3 As shown in Figure 2, the data denoising process includes:
[0103] (1) Select the target denoising point and search point ;
[0104] (2) Select Centered search window and search point The neighborhood window centered on .
[0105] (3) Through weight factor To express and Similarity of neighborhood:
[0106] (5)
[0107] in:
[0108] (6)
[0109] (7)
[0110] is the normalization coefficient, is the smoothing parameter, is the neighborhood window range, is the Gaussian weighted Euclidean distance, is the standard deviation of the Gaussian kernel.
[0111] pass The similarity of the point neighborhood is calculated by weighting its value :
[0112] (8)
[0113] (4) Slide the second neighborhood window to get the next similarity measurement point , and similarly calculate and Similarity and weighted values of neighborhood windows;
[0114] (5) Search all in the window Point, traversal is completed;
[0115] (6) The value obtained after denoising the target denoising point using the NLM method:
[0116] (9)
[0117] Once all points in this dimension have been traversed, the data denoising process ends.
[0118] Step 1.2: Acquire RGB character image data captured by multiple cameras, and perform data preprocessing on the RGB character image data to obtain human body position and posture key point data.
[0119] The gesture recognition part needs to input the collected RGB image data into a trained neural network, which can be any multi-view 3D gesture recognition network that outputs multiple key point spatial coordinates.
[0120] In view of the problem that the existing test simulation system is difficult to simulate nonlinear complex models and behavior processes, the present invention adopts VoxelPose (VoxelPose is a method for multi-view 3D human posture estimation. This method operates directly in 3D space, avoiding the wrong decision that may occur when establishing cross-view correspondence based on 2D posture estimation in each camera view; VoxelPose aggregates the features in all camera views into 3D voxel space, and then inputs them into the voxel proposal network (Cuboid Proposal Network, CPN) to locate all people, and then uses the pose regression network (Pose Regression Network, PRN) to estimate the detailed 3D posture of each proposal.) network for posture recognition, and the posture recognition network can better describe the posture information of the target. However, it is difficult to achieve high accuracy in predicting the spatial position of the target, while devices such as UWB can accurately locate the spatial position of the target, so the fusion of the two can better describe the position and posture of the target, providing accurate data support for mapping the real world to the virtual world.
[0121] like Figure 4 As shown in the figure, the posture recognition process includes: configuring model parameters according to information such as the number of cameras and the size of the test site. Collecting multi-target RGB image data through multi-view cameras. The image frames captured by different cameras at the same time are synchronously input into the VoxelPose network. After passing through the CPN module of the network, a rough spatial positioning representation of the target can be obtained through the cube proposal. After passing through the PRN module, the complete 3D posture prediction of each proposal can be obtained, including the precise spatial coordinates of each key point of the target:
[0122] (10)
[0123] in, , indicating the time point identified The target posture, is the spatial coordinate of the key point.
[0124] Step 1.3: Fuse the positioning data obtained in step 1.1 with the human body position and posture key point data obtained in step 1.2 to obtain the human body posture information in the real world;
[0125] Specifically, for the result of step 1.2, select the key points , assign weight . Step 1.1 corresponds to the target UWB data Assign weight and .
[0126] Calculate the fused position based on the weights:
[0127] (11)
[0128] Calculate the offset of fused position and posture data:
[0129] (12)
[0130] Correct the posture positioning of other key points according to the offset and obtain the target posture information after all preprocessing steps:
[0131] (13)
[0132] in, , etc. represent the positions of other key points of the body and the offset The sum represents the corrected positioning data, and the position data are expressed in three-dimensional space. Coordinate representation.
[0133] As shown in Figure 5, the state prediction and state correction in the present invention adopt a hybrid architecture of a double-layer LSTM neural network and an ensemble Kalman filter. Among them, the double-layer LSTM neural network model realizes posture state prediction, and the ensemble Kalman filter model realizes the correction of the predicted state. The proposed hybrid architecture can intelligently integrate new data and model predictions, and ensure the continuity and high accuracy of the prediction through dynamic sliding windows and parameter iteration, especially in noisy environments and complex time series analysis. The fusion of double-layer LSTM and ensemble Kalman filtering not only deepens the understanding of time series patterns, but also builds a prediction system with strong adaptability and robustness.
[0134] Step 2, state estimation: input the posture information of the human body in the real world into a state estimation model based on a double-layer LSTM neural network model to perform future posture state estimation to obtain predicted posture information.
[0135] The present invention realizes the state prediction function by adopting a double-layer LSTM neural network model, which is realized by two basic LSTM neural networks. The two layers of networks are marked as LSTM1 and LSTM2 respectively. First, in the initialization and training stage of LSTM1, historical observation data is used to capture short-term dependencies under supervised learning, and regularization is incorporated into the training to optimize model parameters and enhance the generalization ability of the model. Subsequently, LSTM2 is trained based on the optimal settings verified by LSTM1 to make the next prediction, and Kalman filtering is used to further improve the accuracy of the prediction. After entering the cyclic prediction stage, the system dynamically updates the sliding window, and the new observation data and the correction value adjusted by the collective Kalman filter are weighted and fused according to the confidence. This process is continuously iterated to ensure that the model parameters are continuously optimized with the new data, so as to continuously provide accurate state prediction in a dynamic environment. LSTM neural network is famous for its excellent short-term prediction ability. Through the gating mechanism, LSTM can effectively capture short-term changes and patterns in the sequence and make accurate predictions in time.
[0136] In LSTM, the input layer receives time series data , including the current observation value and historical information, and then pass it to the hidden layer A. Finally, the output layer generates the final prediction result based on the output of the hidden layer , and its cyclic network structure is shown in Figure 6.
[0137] Among them, the hidden layer is the core part of the LSTM neural network, which captures dependencies and patterns through a series of gated units and cell states, allowing the model to effectively remember and utilize past state information. The hidden layer structure of each LSTM is shown in Figure 7. The independent unit inside the LSTM is mainly composed of the input gate, forget gate, output gate, cell state update, and hidden state update.
[0138] The input gate controls the update of the cell state by new information. In pose prediction, the input gate determines which new pose data will be included in the cell state at the current moment in order to update the pose. The forget gate determines which information in the past cell state should be discarded. For pose prediction tasks, the forget gate helps the model ignore historical information that is irrelevant to the pose prediction at the current moment to maintain the effectiveness of the model. Calculate the output of the input gate and the forget gate:
[0139] ;
[0140] ;
[0141] in, represents the output of the input gate, is the input of the current time step, is the hidden state at the previous time step, is the pose information of the current time step, is the activation function, and the Sigmoid function is commonly used. , , , are the weights and biases of the input gate. represents the output of the forget gate, , , , are the weight and bias of the forget gate respectively.
[0142] The cell state update part updates and maintains the cell state based on the results of the input gate and the forget gate, as well as the current input pose information, to ensure that the model can transfer and remember information between different time steps. Calculate the internal memory method in the LSTM unit:
[0143] ;
[0144] in, Is the internal memory in the LSTM unit, representing the candidate cell state at the current time step. It is updated at time step, regulated by the input gate and the forget gate. are weights and bias terms. is the hyperbolic tangent function.
[0145] The output gate determines which cell states will be output to the hidden state and how the hidden state affects the final pose prediction result. By controlling the flow of cell states, the model can accurately capture the key features in the pose sequence. Cell hidden state update.
[0146] ;
[0147] ;
[0148] in, is the output of the output gate, is the hidden state of the previous step, is the current pose information, is the main output of the LSTM unit, which is a function of the cell state and is regulated by the output gate.
[0149] The two-layer LSTM time series prediction model adopts a nested two-layer LSTM network architecture. The two layers of the network are independently trained iteratively. The prediction scheme is as follows: Figure 8As shown. Whenever the latest perception posture state observation data is received, the prediction model will advance its time series window accordingly, that is, the window moves forward one time step. The model adopts a fixed-length time series window design, which means that whenever a new data point is added to the window, the historical observation data stored in the window will automatically discard the oldest data point to ensure that the model has the effect of accurately predicting short time series. Specifically, the window sliding mechanism is manifested as incrementing one time step forward while updating the information in the window.
[0150] The first time you use the two-layer LSTM time series prediction model to predict the next moment's posture state, you need to initialize the model, such as Fig. 9 As shown, the specific steps include:
[0151] Step 2.1: Training LSTM1: Starting from the time series window, the data in the window is fed into the underlying LSTM1 for processing. Data and Labels Train the LSTM1 model. Serves as a label for the current round.
[0152] Step 2.2: Get the optimal model weight: In each epoch, calculate The predicted value of In order to improve the generalization ability of the model and prevent overfitting, an additional regularization term L2-norm (L2 norm) is added after the loss function to calculate the loss function L:
[0153] ;
[0154] in, represents the root mean square error function, is the regularization parameter, which controls the degree of influence of regularization. Represents the weight parameter matrix The square of the L2 norm of , that is, the sum of the squares of all weight parameters. n is the number of samples, is the actual value, is the predicted value. L2 regularization does not usually cause the parameter to become zero, but it can make the parameter value relatively small, so that some parameters have less influence on the model. This effect can be regarded as a kind of "soft" feature selection, which helps to reduce the complexity of the model and improve its interpretability. After adding regularization, the loss function The rate of decline will slow down, and the rate of increase of accuracy will slow down. Regularization technology can make the model performance smoother, effectively solving the problem of model overfitting.
[0155] Step 2.3: Train LSTM2: In the validation step of LSTM1, determine the epoch number with the minimum loss, and train LSTM2 using the same epoch number, and the same weights and bias vectors. Use the trained LSTM2 to train the input sequence Make predictions, predict Pose status information for each time step :
[0156] .
[0157] Step 2.4: Data correction: convert the predicted value Input the Kalman filter module for correction to obtain the correction value For the correction process, please refer to step 3, status correction.
[0158] After completing the initialization prediction, the cyclic prediction process begins.
[0159] Step 2.5: Data weighting: Using and tags Train LSTM1, With the incoming observations Weighted combination , and put it into the sliding window as a new sequence value:
[0160] ;
[0161] in, is the confidence level of the observed value, and its range is , indicating that the model's confidence in the observed value is better than its confidence in the corrected value.
[0162] Repeat steps 2.1-2.4 to complete the sequence prediction.
[0163] Step 3, state correction: Based on the predicted posture information and the posture information of the human body in the real world, state correction is performed through ensemble Kalman filtering to obtain a corrected posture state.
[0164] When the neural network model completes the prediction of the next time step state After the prediction, due to the error of the system model, the uncertainty of the observed value and other external factors, there is a difference between the predicted value and the true value. The measurement update operation of the ensemble Kalman filter architecture can be used to adjust the predicted value according to the observed value to achieve a more accurate estimation of the system state. The state is predicted and the state is corrected through the ensemble Kalman filter. The result of the correction is The state correction process is shown in Figure 10.
[0165] The ensemble Kalman filter architecture implements state correction mainly by updating the covariance matrix , calculate the Kalman gain , update the revised estimate and revised estimates Feedback consists of four steps.
[0166] Step 3.1: Update the covariance matrix :Complete the double-layer LSTM time series prediction model After the state is predicted, the covariance matrix of the state is updated .
[0167] First, use the prediction sequence state set To estimate the mean of the predicted state :
[0168] ;
[0169] in, Represents the first m The predicted status of each member, Represents the mean of the predicted states of all ensemble members. It can be calculated by taking the average of the states of all ensemble members.
[0170] Then, calculate the deviation of the state :
[0171] ;
[0172] in, It is i The deviation between the predicted state of a ensemble member and the mean of the predicted state of the whole ensemble. It represents a vector that represents the difference between the state of a ensemble member and the predicted state of the ensemble, and its result usually has the same dimension as the state.
[0173] Finally, calculate the covariance matrix :
[0174] ;
[0175] in, Represents the estimated value of the state covariance matrix in the prediction update phase. Usually, it is limited by sampling error, approximate calculation and system model error. The calculation of is not exactly equal to the true state covariance matrix , but is estimated by the covariance of the set members. In the prediction stage, the double-layer LSTM time series prediction model is used to predict the state at the next moment, but the information of the observation value at the next moment has not been considered. Therefore, is based on the predicted state of the set members The computed estimated covariance matrix.
[0176] Step 3.2: Calculate Kalman gain :
[0177] ;
[0178] in, is the covariance matrix of the predicted state, is the observation matrix, which is used to describe the mapping relationship between the state vector and the observation vector. It projects the state space of the system into the observation space so that the system state and the observation value can be compared. is the covariance matrix of the observation noise, which describes the noise characteristics in the observation process, that is, the variance and correlation of the observation error. It represents the uncertainty of the system observation value, that is, the difference between the system observation value and the true value.
[0179] Step 3.3: Correct the prediction Status, get the corrected value :
[0180] ;
[0181] in, is the Kalman gain, The time step is When , the observation value of the pose information is collected, is the observation matrix. The ensemble Kalman filter predicts the value and measured values The difference between them (i.e., residual) is weighted to update the state estimate. By fusing the information of the two, a more accurate and reliable state estimation result can be obtained.
[0182] Step 4: Model update: The corrected posture state and the posture information of the human body in the real world are used as inputs of the state estimation model to obtain the predicted posture information at the next moment.
[0183] After completing the correction of the state at time step T+1, the correction result is fed back to the neural network. The neural network updates its loss function, which largely determines the direction and goal of model training. Through optimization algorithms such as gradient descent, the neural network will update the parameters of the model along the negative gradient direction of the loss function to reduce the value of the loss function. The gradient of the loss function indicates how the model parameters should be adjusted to reduce the loss function, thereby improving the prediction effect of the model. Through the feedback of the estimated value of each cycle correction, the training of the neural network can be guided towards the corrected result, and the prediction neural network model can be guided so that its prediction can be closer to the corrected value.
[0184] Corresponding to the data assimilation method in the above embodiment, the present invention further provides a data assimilation system for virtual-real integrated simulation, the system comprising:
[0185] A preprocessing module is used to obtain real-time position coordinate data collected by a UWB or RTK device, and perform data preprocessing on the real-time position coordinate data to obtain positioning data; obtain RGB human image data captured by multiple cameras, and perform data preprocessing on the RGB human image data to obtain human body position and posture key point data; perform data fusion on the positioning data and the human body position and posture key point data to obtain human body position and posture information in the real world;
[0186] A posture prediction module is used to input the posture information of the human body in the real world into a state estimation model based on a double-layer LSTM neural network model to estimate the future posture state and obtain predicted posture information;
[0187] A posture correction module, used for performing state correction by ensemble Kalman filtering based on the predicted posture information and the posture information of the human body in the real world to obtain a corrected posture state;
[0188] The model updating module is used to use the corrected posture state and the posture information of the human body in the real world as inputs of the state estimation model to obtain the predicted posture information at the next moment.
[0189] As for the data assimilation system in the embodiment of the present invention, since it corresponds to the data assimilation method in the above embodiment, the description is relatively simple. For the relevant similarities, please refer to the description of the data assimilation method in the above embodiment, which will not be described in detail here.
[0190] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A data assimilation method for virtual-real integrated simulation, characterized in that: The method comprises: Acquire real-time position coordinate data collected by a UWB or RTK device, and perform data preprocessing on the real-time position coordinate data to obtain positioning data; Acquire RGB character image data captured by multiple cameras, and perform data preprocessing on the RGB character image data to obtain key point data of human body position and posture; The positioning data is fused with the human body position and posture key point data to obtain the human body posture information in the real world; The posture information of the human body in the real world is input into a state estimation model based on a double-layer LSTM neural network model to estimate the future posture state, and obtain predicted posture information; the initialization prediction process of the state estimation model includes: Training the first layer of LSTM: Starting from the time series window, the data in the window is fed into the first layer of LSTM for processing, using the observed partial sequence Data and Labels Train the first layer of LSTM, where As a label for the current round; Obtaining the optimal model weights: In each training cycle, by calculating The predicted value of Verify the loss between Training the second LSTM layer: In the validation step of the first LSTM layer, determine the number of training cycles with the minimum loss, use the same number of training cycles, and the same weights and bias vectors to train the second LSTM layer; use the trained second LSTM layer to train the input sequence Make predictions, predict Pose status information for each time step ; Data correction: Change the predicted value Input the Kalman filter for correction and get the correction value ; Based on the predicted posture information and the posture information of the human body in the real world, state correction is performed through ensemble Kalman filtering to obtain a corrected posture state, including: (1) Update the covariance matrix : Using the prediction sequence state set Estimate the mean of the predicted states ; Calculate the deviation of the state : ; in, It is i The deviation between the predicted state of each set member and the mean of the predicted state; Calculate the covariance matrix : ; in, represents the estimated value of the state covariance matrix during the prediction update phase; (2) Calculate the Kalman gain : ; in, is the covariance matrix of the predicted state, is the observation matrix, which is used to describe the mapping relationship between the state vector and the observation vector; is the covariance matrix of the observation noise; (3) Correction prediction Status, get the corrected value : ; in, is the Kalman gain, The time step is When , the observation value of the pose information is collected, is the observation matrix; The corrected posture state and the posture information of the human body in the real world are used as inputs of the state estimation model to obtain the predicted posture information at the next moment; the prediction process of the state estimation model includes: Using the input sequence and label of the previous time step Train the first layer of LSTM and With the incoming observations Weighted combination , put it into the sliding window as a new sequence value, and predict the next time step.
2. The data assimilation method for virtual-real integrated simulation according to claim 1 is characterized in that: Acquire real-time position coordinate data collected by UWB or RTK equipment, and perform data preprocessing on the real-time position coordinate data to obtain positioning data, including: Record the real-time position coordinate data collected by UWB or RTK equipment, and use The sub-Lagrangian interpolation function eliminates the phenomenon of missing data values; The non-local mean interpolation method is used to denoise the data after missing value processing to obtain missing and noise-free positioning data.
3. The data assimilation method for virtual-real integrated simulation according to claim 2 is characterized in that: The non-local mean interpolation method is used to denoise the data after missing value processing, including: Select the target denoising point and search point , and Centered search window and search point The neighborhood window centered on ; By weight factor To express and Similarity of neighborhood: ; in: ; ; is the normalization coefficient, is the smoothing parameter, is the neighborhood window range, is the Gaussian weighted Euclidean distance, is the standard deviation of the Gaussian kernel; pass The similarity of the point neighborhood is calculated by weighting its value : ; Slide the second neighborhood window to get the next similarity measurement point , and similarly calculate and Similarity and weighted values of neighborhood windows; The value obtained after denoising the target denoising point using the non-local mean method NLM: ; in, It is the UWB data corresponding to the target in the positioning data.
4. The data assimilation method for virtual-real integrated simulation according to claim 1 is characterized in that: Acquire RGB character image data captured by multiple cameras, and perform data preprocessing on the RGB character image data to obtain key point data of human body position and posture, including: The collected RGB person image data is input into a trained neural network for posture recognition, and the human body position and posture key point data are output; the neural network is a multi-view 3D posture recognition network.
5. The data assimilation method for virtual-real integrated simulation according to claim 4 is characterized in that: The neural network is a VoxelPose network; the neural network performs posture recognition, including: obtaining a cube space positioning proposal of the target through a voxel proposal network CPN module of the VoxelPose network; obtaining spatial coordinate information of each key point of the target of each proposal through a posture regression network PRN module of the VoxelPose network, each key point representing a human joint or a human body part.
6. The data assimilation method for virtual-real integrated simulation according to claim 1 is characterized in that: The positioning data and the human body position and posture key point data are fused to obtain the human body posture information in the real world, including: For the human body position and posture key point data, select key points , assign weight ; The UWB data corresponding to the target in the positioning data Assign weight , and ; Calculate the fused position based on the weights: ; Calculate the offset of fused position and posture data: ; Correct the posture positioning of other key points according to the offset to obtain the posture information of the human body in the real world: ; in, , etc. represent the positions of other key points of the body and the offset The sum represents the corrected positioning data, and the position data are expressed in three-dimensional space. Coordinate representation.
7. A data assimilation system for virtual-real integrated simulation, characterized in that: A data assimilation method for virtual-real integrated simulation as described in any one of claims 1 to 6, the system comprising: A preprocessing module is used to obtain real-time position coordinate data collected by a UWB or RTK device, and perform data preprocessing on the real-time position coordinate data to obtain positioning data; obtain RGB human image data captured by multiple cameras, and perform data preprocessing on the RGB human image data to obtain human body position and posture key point data; perform data fusion on the positioning data and the human body position and posture key point data to obtain human body position and posture information in the real world; A posture prediction module is used to input the posture information of the human body in the real world into a state estimation model based on a double-layer LSTM neural network model to estimate the future posture state and obtain predicted posture information; A posture correction module, used for performing state correction by ensemble Kalman filtering based on the predicted posture information and the posture information of the human body in the real world to obtain a corrected posture state; The model updating module is used to use the corrected posture state and the posture information of the human body in the real world as inputs of the state estimation model to obtain the predicted posture information at the next moment.
