VPS and ARFoundation pose fusion positioning method

Through the VPS and ARFoundation pose fusion positioning method, the problem of inaccurate positioning and real-time delay in mobile devices in the lack of visual feature environment is solved, and a more accurate and stable augmented reality experience is achieved.

CN118362133BActive Publication Date: 2024-11-26SPEED SPACE TIME INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410796869.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-06-20
Publication Date
2024-11-26
Estimated Expiration
2044-06-20

AI Technical Summary

Technical Problem

In the existing augmented reality positioning method of mobile devices, the VPS system is inaccurate in positioning in an environment lacking visual characteristics and is delayed in real time, which affects the experience.

Method used

The VPS and ARFoundation pose fusion positioning method is adopted to improve positioning accuracy and stability through coordinate system conversion, IMU sensor data preprocessing and particle filtering algorithm.

Benefits of technology

It realizes more accurate and real-time augmented reality positioning in complex environments, reduces jitter and drift, and provides a richer and immersive user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118362133B_ABST
    Figure CN118362133B_ABST
Patent Text Reader

Abstract

The present invention discloses a method for VPS and ARFoundation posture fusion positioning, the steps are: S1 definition and conversion of coordinate system: determine the relationship between the coordinate system of VPS posture and the coordinate system of ARFoundation posture, and then convert the VPS coordinate system; S2 ARFoundation posture solution: pre-process the data obtained by the IMU sensor of the device through ARFoundation to obtain the displacement and posture estimation value of the device; S3 posture data fusion: use the particle filter algorithm to fuse the VPS posture and ARFoundation posture to obtain the fusion posture result, so as to achieve positioning. The posture fusion positioning method combines the IMU sensor posture provided by ARFoundation with the VPS posture, reduces jitter and drift, provides more accurate posture estimation, and thus improves the accuracy of positioning.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of augmented reality technology, and in particular to a method for fusion positioning of VPS and ARFoundation. Background Art

[0002] Existing methods for augmented reality positioning of mobile devices all use the VPS (Visual Positioning System) system to use image matching and tracking technology to compare the real-time captured image with the image in the pre-built map or database to determine the position and posture of the current image. However, this positioning method is limited by the availability and coverage of visual features in the environment. In an environment lacking obvious visual features, the positioning results will be inaccurate. At the same time, a large amount of image processing and feature extraction by VPS will cause delays in real time, affecting the positioning experience.

[0003] VPS is a positioning system based on computer vision technology, which aims to achieve precise positioning and navigation functions by using cameras, sensors and image processing algorithms. Unlike the traditional global positioning system (GPS), VPS uses visual information to obtain location information instead of relying on satellite signals. VPS can be positioned in indoor and outdoor environments and is suitable for a variety of application scenarios. For example, when a user is looking for a specific store or boarding gate in an unfamiliar mall or airport, VPS can provide accurate navigation guidance by identifying and navigating to specific visual representations or landmarks. In addition, VPS can also be used in augmented reality (AR) applications to provide users with an immersive interactive experience by combining virtual objects with real scenes. The concept of VPS stems from the need for positioning and navigation technology, especially in indoor environments or areas where GPS signals are limited, such as urban canyons. It provides a computer vision-based solution that overcomes the limitations of traditional GPS and provides users with more accurate and real-time location information.

[0004] VPS works by identifying and tracking visual features in the environment to determine the user's location. These visual features can be buildings, landmarks, representations, patterns, or other recognizable objects. The VPS system uses a camera or depth sensor to capture images of the environment and uses image processing algorithms for real-time analysis and feature extraction. By comparing with images in a pre-built map or database, VPS can determine the user's location and provide corresponding positioning services.

[0005] ARFoundation is an augmented reality (AR) development framework provided by the Unity engine, which aims to simplify the process of creating augmented reality applications on mobile platforms and enable developers to build cross-platform AR applications. ARFoundation provides the function of acquiring sensor data of mobile devices and tracks the dynamic posture changes of the device based on the acquired sensor data.

[0006] In order to further improve the positioning accuracy and stability, it is necessary to develop a method of VPS and ARFoundation pose fusion positioning, and introduce VPS and ARFoundation pose fusion technology, in which ARFoundation uses the sensor data on the device to track the device's pose information and align the virtual object with the real scene, and then combines the ARFoundation pose information with the VPS positioning result, thereby making up for the real-time delay and instability of VPS positioning, and achieving a more accurate and real-time augmented reality positioning experience. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide a method for VPS and ARFoundation posture fusion positioning, aiming to improve the positioning accuracy and stability of augmented reality applications on mobile devices.

[0008] In order to solve the above technical problems, the technical solution adopted by the present invention is: the method for fusion positioning of VPS and ARFoundation posture specifically includes the following steps:

[0009] S1 Definition and conversion of coordinate system: determine the relationship between the coordinate system of the virtual private server VPS pose and the coordinate system of the ARFoundation pose, and then convert the coordinate system of the virtual private server VPS pose;

[0010] S2 ARFoundation pose calculation: ARFoundation is used to pre-process the data obtained by the device's IMU sensor to obtain the device's displacement and pose estimation values;

[0011] S3 pose data fusion: Use the particle filter algorithm to fuse the virtual private server VPS pose and ARFoundation pose to obtain the fused pose result, thereby achieving positioning.

[0012] The above technical solution is adopted, through the definition of coordinate system, then the ARFoundation posture optimization processing, and finally the VPS posture fusion, so as to obtain more accurate device positioning; this technical solution proposes a positioning technology system based on VPS posture and ARFoundation posture data fusion positioning for the VPS indoor positioning accuracy problem, which mainly includes the definition and transformation of the coordinate system based on matrix transformation, and the establishment of a VPS posture and ARFoundation posture system that uses particle filter algorithm to fuse and calculate the VPS posture, which improves the VPS positioning capability of AR in multiple aspects such as positioning accuracy, positioning stability, and real-time performance. Among them, ARFoundation uses the sensor data on the device to track the posture information and align the virtual object with the real scene; then the positioning result of VPS is combined with the posture information of ARFoundation to make up for the disadvantages of VPS positioning real-time delay and instability, and realize a more accurate and real-time augmented reality positioning experience; through the fusion of posture information with ARFoundation, combined with the sensor data and posture information of the mobile device, the augmented reality positioning system of the mobile device can more accurately position and align the virtual object with the real world.

[0013] Preferably, the specific steps of step S1 are:

[0014] S11 defines the coordinate system: defines the Unity 3D world coordinate system as the reference coordinate system;

[0015] Determination of S12VPS coordinate system and ARFoundation coordinate system: In ARFoundation coordinate system, Unity3D world coordinate system is used, where x-axis points to the right, y-axis points upward, and z-axis points to the outside of the screen; in VPS, VPS world coordinate system is used, with a fixed point as the origin, x-axis points to the right, y-axis points to the outside of the screen, and z-axis points upward;

[0016] S13 defines the conversion between coordinate systems: the virtual private server VPS world coordinate system is converted to the Unity 3D world coordinate system by means of matrix transformation. In order to fuse the virtual private server VPS pose and the ARFoundation pose, it is necessary to convert the VPS world coordinate system in which they are located.

[0017] Preferably, the specific steps of coordinate system conversion in step S13 are:

[0018] S131 Matrix definition: For a 3D point or vector (x, y, z), the matrix transformation is expressed as: [x', y', z', w'] = [x, y, z, 1] * M; where M is a 4x4 matrix, representing the matrix transformation operation;

[0019] S132 Matrix multiplication operation: Perform matrix multiplication operation to realize coordinate system transformation, specifically: decompose the original matrix into the product of three matrices, which represent translation operation, rotation operation and scaling operation respectively; the formula is M=T*R*S; where T is the translation matrix, R is the rotation matrix, and S is the scaling matrix;

[0020] The translation matrix T is expressed as:

[0021] [1, 0, 0, tx]

[0022] [0, 1, 0, ty]

[0023] [0, 0, 1, tz]

[0024] [0, 0, 0, 1];

[0025] Among them, tx, ty, and tz represent the translation distance;

[0026] The rotation matrix R is expressed as:

[0027] [R11, R12, R13, 0]

[0028] [R21, R22, R23, 0]

[0029] [R31, R32, R33, 0]

[0030] [ 0, 0, 0, 1];

[0031] Wherein, R11, R12, R13, R21, R22, R23, R31, R32, R33 represent the elements of the rotation matrix R;

[0032] The scaling matrix S is expressed as:

[0033] [sx, 0, 0, 0]

[0034] [ 0, sy, 0, 0]

[0035] [ 0, 0, sz, 0]

[0036] [ 0, 0, 0, 1];

[0037] Among them, sx, sy, and sz represent the scaling ratio.

[0038] Preferably, the specific steps of step S2 are:

[0039] S21 data acquisition: ARFoundation measures the linear acceleration of the device through the accelerometer; measures the angular velocity of the device through the gyroscope; and provides the device's direction information through the magnetometer;

[0040] S22 Data preprocessing: preprocess the data obtained by the IMU sensor to improve the accuracy of the solution; the preprocessing methods include removing the influence of gravity and noise processing; removing the influence of gravity is to create a low-pass filter object by specifying the cutoff frequency of the low-pass filter, pass the data to be processed as input to the low-pass filter, and obtain the filtered output data;

[0041] The specific process of preprocessing is to remove the influence of gravity through a low-pass filter. The low-pass filter allows low-frequency signals to pass through and suppresses high-frequency signals. The influence of gravity is usually a low-frequency signal, so the low-pass filter can filter out the gravity component and extract the linear acceleration of the device; at the same time, the cutoff frequency is a parameter of the low-pass filter, which determines the frequency at which the filter starts to attenuate;

[0042] S23 Device Attitude Estimation: Use gyroscope data to estimate the rotational angular velocity of the device; obtain the rotation angle of the device through quaternion integration and angular velocity integration, which represents the rotation of the device relative to the starting attitude; specifically, first define the quaternion q (q0, q1, q2, q3), where q0 is the real part and q1, q2, q3 are all imaginary parts; then calculate the angular velocity change of the gyroscope in each time step according to the quaternion differential equation to update the attitude, the formula is:

[0043] q_dot=0.5*(w*q);

[0044] Where q_dot is the derivative of the quaternion, w is the angular velocity (from the gyroscope data), and q is the quaternion at the current moment;

[0045] Then perform numerical integration of the quaternion, i.e. Euler integration, to obtain the rotation angle of the device;

[0046] The Euler integral formula is: q(t+dt)=q(t)+q_dot*dt*w, where t is the current time; q(t) is the value of the quaternion at the current time t; dt is the time step; q(t+dt) is the value of the quaternion at the current time t after the time step dt;

[0047] S24 device displacement estimation: the accelerometer provides linear acceleration information of the device in three axes, specifically, the accelerometer data pre-processed in step S22 is subjected to Euler integration to estimate the displacement of the device; the formula for updating the displacement of the device is:

[0048] ;

[0049] Among them, f(dis+dt) is the cumulative value of the position at the next moment, f(dis) is the cumulative value of the displacement at the current moment, velocity is the cumulative value of the speed, acceleration is the acceleration, and dt is the time step.

[0050] In step S22, the data preprocessing process is specifically to remove the influence of gravity, which can be achieved through a low-pass filter. The low-pass filter allows low-frequency signals to pass through and suppresses high-frequency signals. The influence of gravity is usually a low-frequency signal, so the low-pass filter can filter out the gravity component and extract the linear acceleration of the device; at the same time, the cut-off frequency is a parameter of the low-pass filter, which determines the frequency at which the filter begins to attenuate.

[0051] Preferably, the specific steps of step S3 are:

[0052] S31 Particle filter algorithm initialization: Initialize the particle filter, that is, define the particle filter state space as a 6Dof posture, sample the particles from the prior distribution to obtain the initial state, and then assign an initial weight to each particle;

[0053] S32 state prediction: obtain the state of each particle at the current moment, obtain the data of the IMU sensor of the device (accelerometer, gyroscope, etc.) as the control input. The data of the IMU sensor reflects the motion state of the device and can be used to simulate the motion of the device; predict the position of the mobile device at the next moment through the acceleration model, which defines how the device moves in unit time; for each particle, simulate the uncertainty and error in the motion by adding random disturbances, and the mean value of the random disturbance term is 0. Add the random disturbance term to the predicted posture to obtain the predicted posture value of each particle at the next moment;

[0054] S33 Observation update: The observation update probability of each particle is calculated using the Gaussian distribution model;

[0055] S34 Weight normalization: Normalize the weights of the particles to ensure that the sum of the weights is equal to 1; the normalized weights represent the contribution of each particle in the pose estimation;

[0056] S35 Particle Resampling: Roulette wheel selection method is used to resample particles according to the normalized weights.

[0057] S36 pose estimation: For each particle, multiply its pose by the normalized weight, add the result to the pose estimate, and then calculate the weighted average, which is the pose estimate.

[0058] Preferably, the 6Dof posture in step S31 refers to the position and posture of the object in three-dimensional space, which consists of three translational degrees of freedom and three rotational degrees of freedom, a total of six degrees of freedom.

[0059] Preferably, the acceleration model formula in the three-dimensional space in step S32 is:

[0060] ;

[0061] Among them, p(t) is the position vector of the object at the current time t, p_0 is the initial position vector, v_0 is the initial velocity vector, a is a constant acceleration vector, and the acceleration model is used to predict the future position or state of the object under given initial state and acceleration conditions.

[0062] Preferably, in step S33, it is assumed that the observation error obeys a Gaussian distribution, and the observation error is assumed to be normally distributed, so the Gaussian distribution is used to describe the observation error, and the observation update probability is calculated according to the probability density function value of the observation error. The probability density function of the Gaussian distribution model is:

[0063] ;

[0064] in, f ( x ) represents the probability density function value, x represents the observation error, m represents the mean of the Gaussian distribution, s represents the standard deviation of the Gaussian distribution; the observation update probability is obtained by calculating the difference between the observation error and the actual observation value, and inputting the calculated difference into the Gaussian distribution; that is, the Gaussian distribution model calculates its observation update probability by accepting the observation error x , the mean of the Gaussian distribution m and the standard deviation of the Gaussian distribution s As input, the probability of observation update of Gaussian distribution is calculated through probability density function.

[0065] Preferably, the specific steps of step S35 are:

[0066] S351 creates a cumulative weight array: the length of the cumulative weight array is the same as the number of particles, and the cumulative weight from the first particle to the current particle is stored for each element; the cumulative weight is calculated by accumulating the normalized weights one by one; in order to perform random sampling, a cumulative weight array needs to be created;

[0067] S352 generates a random starting point and performs sampling: generates a random number between 0 and 1 as the starting point; performs sampling using a roulette wheel selection method; finds the index of the first element greater than the starting point value from the cumulative weight array, corresponding to the selected particle;

[0068] S353 Copy and update particles: copy the selected particles to a new particle set and update them as needed, the update includes adding noise or applying a motion model;

[0069] S354 Repeat sampling: Repeat steps S352 and S353 until a new particle set with the same number of original particles is obtained. The purpose of particle resampling is to provide a sampling opportunity for each particle according to the normalized weight to generate a new particle set, while retaining high-weight particles, and generating a new particle set by copying and replacing low-weight particles. The resampling process is used to maintain the number of high-weight particles and remove low-weight particles; a common particle resampling technique and process is called Roulette Wheel Selection.

[0070] Preferably, the weighted average formula in step S36 is:

[0071] ;

[0072] in, weight_i It is i The normalized weight of a particle, pose_i It is i The position of a particle, numParticles is the number of particles; the weighted average calculated f ( east ) is the estimated value of the pose.

[0073] Compared with the prior art, the present invention has the following beneficial effects: using the improved mobile augmented reality positioning technology based on VPS posture fusion ARFoundation posture, the problem of inaccurate positioning caused by possible occlusion and illumination changes in complex indoor environments is solved, and the positioning technology after integrating ARFoundation posture can combine the IMU sensor posture provided by ARFoundation with the VPS posture to reduce jitter and drift, provide more accurate posture estimation, and thus improve the accuracy of positioning; wherein the VPS positioning technology can better perceive and understand objects and spatial structures in the environment, ARFoundation provides the functions of environmental perception and 3D scene reconstruction, can identify planes, objects and spatial features, and make it possible to have a deeper interaction between virtual and reality; for example, more accurate virtual object placement, occlusion effects and physical interactions can be achieved in AR applications; the VPS positioning technology after integrating ARFoundation posture can provide a richer and more immersive augmented reality experience. Users can observe and interact with the fusion of virtual content and the real world in real time through mobile devices without the need for additional external markers or sensor devices; this interaction method is more natural and intuitive, and improves the quality of user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 It is a flowchart of step S3 of the method for VPS and ARFoundation posture fusion positioning of the present invention;

[0075] Figure 2 It is a flowchart of step S35 of the method for VPS and ARFoundation posture fusion positioning of the present invention;

[0076] Figure 3 It is an effect diagram of the posture prediction value after random perturbation processing in step S32 of the method for VPS and ARFoundation posture fusion positioning of the present invention;

[0077] Figure 4 This is a graph showing the normalized effect of particle weights observed in step S34 of the method for fusion positioning of VPS and ARFoundation poses of the present invention. DETAILED DESCRIPTION

[0078] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiment diagram of the present invention.

[0079] Embodiment: The method for fusion positioning of VPS and ARFoundation posture specifically comprises the following steps:

[0080] Definition and conversion of S1 coordinate system: Determine the relationship between the coordinate system of the virtual private server VPS pose and the coordinate system of the ARFoundation pose, and then convert the coordinate system of the virtual private server VPS pose;

[0081] The specific steps of step S1 are:

[0082] S11 Define coordinate system: Define Unity 3D world coordinate system as reference coordinate system; VPS pose is relative to the reference coordinate system of mapping service, while ARFoundation pose is relative to the coordinate system of the device itself, and the relationship between the two coordinate systems needs to be determined; the coordinate transformation process can be implemented using matrix transformation in linear algebra, such as rotation matrix and translation matrix; the coordinate transformation process can be implemented using matrix transformation in linear algebra, such as rotation matrix and translation matrix;

[0083] S12 VPS coordinate system and ARFoundation coordinate system determination: In the ARFoundation coordinate system, the Unity 3D world coordinate system is used, where the x-axis points to the right, the y-axis points upward, and the z-axis points out of the screen; in the virtual private server VPS, the virtual private server VPS world coordinate system is used, with a fixed point as the origin, the x-axis points to the right, the y-axis points out of the screen, and the z-axis points upward;

[0084] S13 defines the conversion between coordinate systems: the virtual private server VPS world coordinate system is converted to the Unity 3D world coordinate system by means of matrix transformation; in order to merge the virtual private server VPS pose and the ARFoundation pose, it is necessary to convert the VPS world coordinate system in which they are located;

[0085] The specific steps of coordinate system conversion in step S13 are:

[0086] S131 Matrix definition: For a 3D point or vector (x, y, z), the matrix transformation is expressed as: [x', y', z', w'] = [x, y, z, 1] * M; where M is a 4x4 matrix, representing the matrix transformation operation;

[0087] S132 Matrix multiplication operation: Perform matrix multiplication operation to realize coordinate system transformation, specifically: decompose the original matrix into the product of three matrices, which represent translation operation, rotation operation and scaling operation respectively; the formula is M=T*R*S; where T is the translation matrix, R is the rotation matrix, and S is the scaling matrix;

[0088] The translation matrix T is expressed as:

[0089] [1, 0, 0, tx]

[0090] [0, 1, 0, ty]

[0091] [0, 0, 1, tz]

[0092] [0, 0, 0, 1];

[0093] Among them, tx, ty, and tz represent the translation distance;

[0094] The rotation matrix R is expressed as:

[0095] [R11, R12, R13, 0]

[0096] [R21, R22, R23, 0]

[0097] [R31, R32, R33, 0]

[0098] [ 0, 0, 0, 1];

[0099] Wherein, R11, R12, R13, R21, R22, R23, R31, R32, R33 represent the elements of the rotation matrix R;

[0100] The scaling matrix S is expressed as:

[0101] [sx, 0, 0, 0]

[0102] [ 0, sy, 0, 0]

[0103] [ 0, 0, sz, 0]

[0104] [ 0, 0, 0, 1];

[0105] Among them, sx, sy, and sz represent the scaling ratio;

[0106] S2ARFoundation pose calculation: ARFoundation pre-processes the data obtained by the device's IMU sensor to obtain the device's displacement and pose estimation values; ARFoundation obtains three key data from the device's IMU sensor: accelerometer, gyroscope, and magnetometer;

[0107] The specific steps of step S2 are:

[0108] S21 data acquisition: ARFoundation measures the linear acceleration of the mobile device through the accelerometer; measures the angular velocity of the mobile device through the gyroscope; and provides the direction information of the mobile device through the magnetometer;

[0109] S22 Data preprocessing: preprocess the data obtained by the IMU sensor to improve the accuracy of the solution; the preprocessing methods include removing the influence of gravity and noise processing; removing the influence of gravity is to create a low-pass filter object by specifying the cutoff frequency of the low-pass filter, passing the data to be processed as input to the low-pass filter, and obtaining the filtered output data; the preprocessing process is specifically to remove the influence of gravity through a low-pass filter, and the low-pass filter allows low-frequency signals to pass through and suppresses high-frequency signals; the influence of gravity is usually a low-frequency signal, so the low-pass filter can filter out the gravity component and extract the linear acceleration of the mobile device; at the same time, the cutoff frequency is a parameter of the low-pass filter, which determines the frequency at which the filter starts to attenuate;

[0110] S23 Device Attitude Estimation: Use gyroscope data to estimate the rotational angular velocity of the device; obtain the rotation angle of the mobile device through quaternion integration and angular velocity integration, which represents the rotation of the device relative to the starting attitude; specifically, first define the quaternion q (q0, q1, q2, q3), where q0 is the real part and q1, q2, q3 are all imaginary parts; then calculate the angular velocity change of the gyroscope in each time step according to the quaternion differential equation to update the attitude, the formula is:

[0111] q_dot=0.5*(w*q);

[0112] Among them, q_dot is the derivative of the quaternion, w is the angular velocity (from the gyroscope data), and q is the quaternion at the current moment; the quaternion is then numerically integrated, i.e., Euler integration, to obtain the rotation angle of the mobile device;

[0113] The Euler integral formula is: q(t+dt)=q(t)+q_dot*dt*w, where t is the current time; q(t) is the value of the quaternion at the current time t; dt is the time step; q(t+dt) is the value of the quaternion at the current time t after the time step dt;

[0114] S24 Device displacement estimation: The accelerometer provides linear acceleration information of the device in three axes, specifically, the accelerometer data pre-processed in step S22 is subjected to Euler integration to estimate the displacement of the mobile device; the formula for updating the displacement of the mobile device is:

[0115] ;

[0116] Among them, f(dis+dt) is the cumulative value of the position at the next moment, f(dis) is the cumulative value of the displacement at the current moment, velocity is the cumulative value of the speed, acceleration is the acceleration, and dt is the time step;

[0117] S3 pose data fusion: Use the particle filter algorithm to fuse the virtual private server VPS pose and ARFoundation pose to obtain the fused pose result, thereby achieving positioning; particle filtering, also known as Monte Carlo filtering, is a state estimation method based on random sampling. The particle filter algorithm approximates the probability distribution of the target system by using a set of random sampling points (particles) representing possible system states; the weights of these particles are updated according to the measurement information and dynamic model to achieve the estimation of the system state; the core idea of ​​the particle filter algorithm is to sample in the state space and weight the sampled particles according to the measurement data to reflect the changes in the possibility of the system state; over time, important particles will gradually concentrate near the most likely state, thereby achieving tracking and estimation of the system state;

[0118] like Figure 1 As shown, the specific steps of step S3 are:

[0119] S31 Particle filter algorithm initialization: The particle filter algorithm is used to estimate the sequence of states. Before fusing the VPS pose and the ARFoundation pose, the particle filter needs to be initialized. To initialize the particle filter, the particle filter state space is defined as a 6Dof pose, and the particles are sampled from the prior distribution to obtain the initial state, and then each particle is assigned an initial weight. Particle filtering, also known as Monte Carlo filtering, is a state estimation method based on random sampling. The particle filter algorithm approximates the probability distribution of the target system by using a set of random sampling points (particles) representing possible system states. The weights of these particles are updated according to the measurement information and the dynamic model to estimate the system state. The core idea of ​​the particle filter algorithm is to sample in the state space and weight the sampled particles according to the measurement data to reflect the changes in the possibility of the system state. Over time, important particles will gradually concentrate near the most likely state, thereby tracking and estimating the system state.

[0120] The 6Dof posture in step S31 refers to the position and posture of the object in three-dimensional space, which consists of three translational degrees of freedom and three rotational degrees of freedom, a total of six degrees of freedom;

[0121] S32 state prediction: obtain the state of each particle at the current moment, obtain the data of the IMU sensor (accelerometer, gyroscope, etc.) of the mobile device as the control input. The IMU sensor data reflects the motion state of the device and can be used to simulate the motion of the mobile device; predict the position of the mobile device at the next moment through the acceleration model, which defines how the mobile device moves in unit time; for each particle, simulate the uncertainty and error in the motion by adding random disturbances, and the mean of the random disturbance term is 0. Add the random disturbance term to the predicted posture to obtain the predicted posture value of each particle at the next moment;

[0122] The formula of the acceleration model in the three-dimensional space in step S32 is:

[0123] ;

[0124] Wherein, p(t) is the position vector of the object at the current time t, p_0 is the initial position vector, v_0 is the initial velocity vector, a is a constant acceleration vector, and the acceleration model is used to predict the future position or state of the object under given initial state and acceleration conditions;

[0125] The particle state data in step S32 is processed by random perturbation to obtain the predicted value of the position and posture. For details, see the effect diagram of the data processing result. Figure 3 ; Figure 3 The error between the true value of the particle state and the predicted value after random perturbation is reflected in it, which shows that there are usually errors in the initial state of the particle. By introducing random perturbations, it is easier to converge to the correct pose and reduce the uncertainty during initialization. The technical effect achieved by random perturbations of particles can help reduce the jitter and instability of pose estimation. When there is noise or uncertainty in sensor data, pose estimation may be discontinuous or jittery. By introducing small random perturbations, the estimated pose changes more smoothly, thereby improving the stability of the virtual object and allowing it to interact more naturally with the real world. Random perturbations of particles help reduce estimation errors. In pose estimation, there are many factors that may lead to error accumulation, such as sensor accuracy, environmental changes, etc. By introducing random perturbations periodically, the estimated pose can approach the true value faster, thereby improving accuracy;

[0126] S33 Observation update: For each particle, the observation update probability is calculated using the Gaussian distribution model; For each particle, the observation update probability is calculated using the Gaussian distribution model, and the observation probability represents the degree of consistency between the particle's position and the actual observation value;

[0127] In step S33, it is assumed that the observation error obeys the Gaussian distribution, and the observation error is assumed to be normally distributed. Therefore, the Gaussian distribution is used to describe the observation error, and the observation update probability is calculated according to the probability density function value of the observation error. The probability density function value describes the relationship between the observation error and the mean, and is used to calculate the observation probability; the probability density function of the Gaussian distribution model is a bell-shaped curve, and the probability density function of the Gaussian distribution model is:

[0128] ;

[0129] in, f ( x ) represents the probability density function value, x represents the observation error, m represents the mean of the Gaussian distribution, s represents the standard deviation of the Gaussian distribution; the observation update probability is obtained by calculating the difference between the observation error and the actual observation value, and inputting the calculated difference into the Gaussian distribution; that is, the Gaussian distribution model calculates its observation update probability by accepting the observation error x , the mean of the Gaussian distribution m and the standard deviation of the Gaussian distribution s As input, the probability density function is used to calculate the observation update probability of the Gaussian distribution; specifically: by accepting the observation error x , the mean of the Gaussian distribution m and the standard deviation of the Gaussian distribution m The inverse matrix of the covariance matrix is ​​calculated as input, and then the observation update probability of the Gaussian distribution is calculated using the inverse matrix of the error and covariance; the Gaussian distribution model calculates its observation update probability;

[0130] S34 weight normalization: normalize the weights of the particles to ensure that the sum of the weights is equal to 1; the normalized weights represent the contribution of each particle in the pose estimation; specifically, calculate the sum of the weights of all particles, which will be used as the denominator of normalization; for each particle, divide its weight by the sum of the weights to obtain the normalized weight, and through the above steps, normalize the particle weights into a probability form to ensure that their sum is 1; in this way, in the subsequent resampling of step S35, particles with higher weights will have a higher probability of being selected, thereby better representing the current state of the system;

[0131] The particle weight normalization effect diagram observed in step S34 is shown in Figure 4; The pose particle weight normalization technology reflects the reasonable allocation of the credibility of different pose estimates, ensuring that more accurate pose estimates have a higher influence in the overall pose estimation. In this way, it is shown that the system can better adapt to noise, uncertainty, and various changes in the augmented reality environment. The technical effect achieved by particle weight normalization assigns higher weights to pose particles with higher fit, and the pose estimation system is more likely to choose an accurate pose. This helps to improve the accuracy of the position and orientation of virtual objects in the real world. It helps to reduce uncertainty in the estimation process. When the pose estimation system faces sensor noise or environmental changes, the penalty for uncertain estimates is lower, thereby improving the stability of the estimate. Errors in pose estimation are generally inevitable, but through weight normalization, the system can reduce the impact of inaccurate estimates. Even if there are errors in some pose particles, their weights are lower, so they have less impact on the overall estimate;

[0132] S35 Particle Resampling: Roulette wheel selection is used to resample particles according to normalized weights. The purpose of particle resampling is to provide sampling opportunities for each particle according to the normalized weights to generate a new set of particles, while retaining high-weight particles and generating a new set of particles by copying and replacing low-weight particles. The resampling process is used to maintain the number of high-weight particles and remove low-weight particles. The common particle resampling technique and process is called Roulette Wheel Selection.

[0133] like Figure 2 As shown, the specific steps of step S35 are:

[0134] S351 creates a cumulative weight array: the length of the cumulative weight array is the same as the number of particles, and the cumulative weight from the first particle to the current particle is stored for each element; the cumulative weight is calculated by accumulating the normalized weights one by one; in order to perform random sampling, a cumulative weight array needs to be created;

[0135] S352 generates a random starting point and performs sampling: generates a random number between 0 and 1 as the starting point; performs sampling using a roulette wheel selection method; finds the index of the first element greater than the starting point value from the cumulative weight array, corresponding to the selected particle;

[0136] S353 Copy and update particles: copy the selected particles to a new particle set and update them as needed, the update includes adding noise or applying a motion model;

[0137] S354 repeat sampling: repeat step S352 and step S353 until a new particle set with the same number of original particles is obtained;

[0138] S36 pose estimation: For each particle, multiply its pose by the normalized weight, add the result to the pose estimate, and then calculate the weighted average, which is the pose estimate;

[0139] The formula for weighted average in step S36 is:

[0140] ;

[0141] in, weight_i It is i The normalized weight of a particle, pose_i It is i The position of a particle, numParticles is the number of particles; the weighted average calculated f ( east ) is the estimated value of the pose.

[0142] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for VPS and ARFoundation posture fusion positioning, characterized in that: The specific steps include: S1 Definition and conversion of coordinate system: determine the relationship between the coordinate system of the virtual private server VPS pose and the coordinate system of the ARFoundation pose, and then convert the coordinate system of the virtual private server VPS pose; S2ARFoundation pose calculation: ARFoundation is used to pre-process the data obtained by the device's IMU sensor to obtain the device's displacement and pose estimation values; S3 pose data fusion: Use the particle filter algorithm to fuse the virtual private server VPS pose and ARFoundation pose to obtain the fused pose result, thereby achieving positioning; The specific steps of step S2 are: S21 data acquisition: ARFoundation measures the linear acceleration of the device through the accelerometer; measures the angular velocity of the device through the gyroscope; and provides the device's direction information through the magnetometer; S22 data preprocessing: preprocess the data obtained by the IMU sensor, including removing the influence of gravity and noise processing; Removing the gravity effect involves creating a low-pass filter object by specifying the cutoff frequency of the low-pass filter, passing the data to be processed as input to the low-pass filter, and obtaining the filtered output data; S23 Device Attitude Estimation: Use gyroscope data to estimate the rotational angular velocity of the device; obtain the rotation angle of the device through quaternion integration and angular velocity integration; specifically, first define the quaternion q (q0, q1, q2, q3), where q0 is the real part, and q1, q2, q3 are all imaginary parts; then calculate the angular velocity change of the gyroscope in each time step according to the quaternion differential equation to update the attitude, the formula is: q_dot = 0.5*(w*q); Among them, q_dot is the derivative of the quaternion, w is the angular velocity, and q is the quaternion at the current moment; Then perform numerical integration of the quaternion, i.e., Euler integration, to obtain the rotation angle of the device; the formula of the Euler integration is: q(t+dt)=q(t)+q_dot*dt*w, where t is the current time; q(t) is the value of the quaternion at the current time t; dt is the time step; q(t+dt) is the value of the quaternion at the current time t after the time step dt; S24 device displacement estimation: the accelerometer provides linear acceleration information of the device in three axes, specifically, performing Euler integration on the accelerometer data pre-processed in step S22 to estimate the displacement of the device; The formula for updating the displacement of the device is: f(dis+dt)=f(dis)+velocity*dt+acceleration*d t 2 ; Among them, f(dis+dt) is the cumulative value of the position at the next moment, f(dis) is the cumulative value of the displacement at the current moment, velocity is the cumulative value of the speed, acceleration is the acceleration, and dt is the time step; The specific steps of step S3 are: S31 Particle filter algorithm initialization: Initialize the particle filter, that is, define the particle filter state space as a 6Dof posture, sample the particles from the prior distribution to obtain the initial state, and then assign an initial weight to each particle; S32 state prediction: obtain the state of each particle at the current moment, obtain the data of the IMU sensor of the device as the control input, and predict the position of the device at the next moment through the acceleration model. The acceleration model defines how the device moves in a unit time; for each particle, simulate the uncertainty and error in the motion by adding random disturbances. The mean of the random disturbance term is 0. The random disturbance term is added to the predicted posture to obtain the predicted posture value of each particle at the next moment; S33 Observation update: The observation update probability of each particle is calculated using the Gaussian distribution model; S34 Weight normalization: Normalize the weights of the particles to ensure that the sum of the weights is equal to 1; S35 particle resampling: roulette wheel selection method is used to resample particles according to the normalized weights; S36 pose estimation: For each particle, multiply its pose by the normalized weight, add the result to the pose estimate, and then calculate the weighted average, which is the pose estimate.

2. The method for VPS and ARFoundation posture fusion positioning according to claim 1, characterized in that: The specific steps of step S1 are: S11 defines the coordinate system: defines the Unity 3D world coordinate system as the reference coordinate system; Determination of S12VPS coordinate system and ARFoundation coordinate system: In ARFoundation coordinate system, Unity 3D world coordinate system is used, where x-axis points to the right, y-axis points to the top, and z-axis points to the outside of the screen; in virtual private server VPS, VPS world coordinate system is used, with a fixed point as the origin, x-axis points to the right, y-axis points to the outside of the screen, and z-axis points to the top; S13 defines the conversion between coordinate systems: converting the virtual private server VPS world coordinate system into the Unity 3D world coordinate system by means of matrix transformation.

3. The method for VPS and ARFoundation posture fusion positioning according to claim 2, characterized in that: The specific steps of coordinate system conversion in step S13 are: S131 Matrix Definition: For a 3D point or vector (x, y, z), the matrix transformation is expressed as: [x′, y′, z′, w′]=[x, y, z, 1]*M; Where M is a 4x4 matrix, representing the matrix transformation operation; S132 Matrix multiplication operation: Perform matrix multiplication operation to realize coordinate system transformation, specifically: decompose the original matrix into the product of three matrices, which respectively represent translation operation, rotation operation and scaling operation; the formula is M=T*R*S; where T is the translation matrix, R is the rotation matrix, and S is the scaling matrix; The translation matrix T is expressed as: [1,0,0,tx] [0,1,0,ty] [0,0,1,tz] [0,0,0,1]; Among them, tx, ty, and tz represent the translation distance; The rotation matrix R is expressed as: [R11,R12,R13,0] [R21,R22,R23,0] [R31,R32,R33,0] [0,0,0,1]; Wherein, R11, R12, R13, R21, R22, R23, R31, R32, R33 represent the elements of the rotation matrix R; The scaling matrix S is expressed as: [sx,0,0,0] [0,sy,0,0] [0,0,sz,0] [0,0,0,1]; Among them, sx, sy, and sz represent the scaling ratio.

4. The method for VPS and ARFoundation posture fusion positioning according to claim 1, characterized in that: The 6Dof posture in step S31 refers to the position and posture of an object in three-dimensional space, which consists of three translational degrees of freedom and three rotational degrees of freedom, a total of six degrees of freedom.

5. The method for VPS and ARFoundation posture fusion positioning according to claim 1, characterized in that: The formula of the acceleration model in the three-dimensional space in step S32 is: Among them, p(t) is the position vector of the object at the current time t, p_0 is the initial position vector, v_0 is the initial velocity vector, a is a constant acceleration vector, and the acceleration model is used to predict the future position or state of the object under given initial state and acceleration conditions.

6. The method for VPS and ARFoundation posture fusion positioning according to claim 5, characterized in that: In step S33, it is assumed that the observation error obeys the Gaussian distribution, and the observation error is assumed to be normally distributed. Therefore, the Gaussian distribution is used to describe the observation error, and the observation update probability is calculated according to the probability density function value of the observation error. The probability density function of the Gaussian distribution model is: Wherein, f(x) represents the probability density function value, x represents the observation error, μ represents the mean of the Gaussian distribution, and σ represents the standard deviation of the Gaussian distribution; the observation update probability is obtained by calculating the difference between the observation error and the actual observation value, and inputting the calculated difference into the Gaussian distribution; that is, the Gaussian distribution model calculates its observation update probability by accepting the observation error x, the mean μ of the Gaussian distribution and the standard deviation σ of the Gaussian distribution as inputs, and calculating the observation update probability of the Gaussian distribution through the probability density function.

7. The method for VPS and ARFoundation posture fusion positioning according to claim 5, characterized in that: The specific steps of step S35 are: S351 creates a cumulative weight array: the length of the cumulative weight array is the same as the number of particles, and stores the cumulative weight from the first particle to the current particle for each element; wherein the cumulative weight is calculated by accumulating the normalized weights one by one; S352 generates a random starting point and performs sampling: generates a random number between 0 and 1 as a starting point; Sampling was performed using roulette wheel selection; Find the index of the first element in the cumulative weight array that is greater than the starting value, corresponding to the selected particle; S353 Copy and update particles: copy the selected particles to a new particle set and update them as needed, the update includes adding noise or applying a motion model; S354 repeat sampling: repeat step S352 and step S353 until a new particle set with the same number of original particles is obtained.

8. The method for VPS and ARFoundation posture fusion positioning according to claim 7, characterized in that: The formula for weighted average in step S36 is: Among them, weight_i is the normalized weight of the i-th particle, pose_i is the pose of the i-th particle, and numParticles is the number of particles; the calculated weighted average f(est) is the estimated value of the pose.