Indoor visual inertial positioning method oriented to large-range self-similar environment

Abstract

The invention discloses an indoor visual inertial positioning method oriented to a large-range self-similar environment, and belongs to the technical field of indoor positioning. The method comprises the following steps: firstly, constructing a spatial constraint model based on prior visual information collected by a visual sensor, and adding the spatial constraint model as prior information into system constraint; then, time sequence correlation modeling is performed by fully utilizing historical information correlation, and short-time sequence probability recursion is completed based on spatial motion constraint to obtain an optimal matching sequence so as to ensure correct matching and attitude recovery in a visual challenge scene; and finally, fusing visual information with map constraints by adopting Monte Carlo reasoning, correcting pedestrian track plotting accumulative errors, and ensuring long-term tracking performance of the system. The method supports the user side based on the smart phone to carry out low-cost, rapid deployment and continuous positioning in a large-range indoor scene, has high robustness and decimeter-level positioning precision, and is suitable for various indoor positioning scenes.

Description

Technical Field

[0001] The invention belongs to the technical field of indoor positioning, and in particular to an indoor visual inertial positioning method in a large-scale self-similar environment. Background Art

[0002] With the rapid development of the emerging field of Internet of Things information technology, location information and location-based services play an increasingly important role in all aspects of our lives. However, the accuracy of indoor positioning faces challenges due to the signal shielding effect of concrete in building structures. Although the Global Satellite Navigation System and the Beidou Satellite Navigation System are commonly used for outdoor positioning, they have the problem of signal attenuation when used indoors, which makes people turn to other effective technologies to obtain accurate indoor location information.

[0003] Common indoor positioning methods include Radio Frequency Identification (RFID) positioning technology, Bluetooth positioning technology, Wi-Fi positioning technology, Ultra-Wideband (UWB) positioning technology, inertial navigation positioning technology and geomagnetic positioning technology. Among these methods, RFID, Bluetooth, UWB and Wi-Fi estimate the position by signal strength or arrival time, which is susceptible to wall obstruction and multipath effects. In addition, these methods require the deployment of additional equipment, resulting in increased costs. Geomagnetism uses the particularity of indoor magnetic fields for positioning, but is limited to the uneven distribution of the geomagnetic field.

[0004] Compared with the above-mentioned positioning technologies, pedestrian dead reckoning (PDR) technology has been widely used because it does not require external equipment and has extremely high autonomy. This technology uses data from an inertial measurement unit as input to determine the location of pedestrians by detecting cadence, estimating step length, and calculating heading. The advantages of PDR technology include high positioning continuity and independence. However, due to hardware limitations, PDR technology faces serious error accumulation problems. Image-based relocalization (IBL) is a method for determining location by recognizing scenes in visual images. IBL is very suitable for accurately correcting positioning deviations due to its high robustness and lack of reliance on additional facilities. It can use visual data to correct errors in PDR while enhancing the consistency and real-time responsiveness of trajectory estimation. However, image-based relocalization technology is extremely sensitive to environmental factors such as lighting conditions and scene dynamic changes. In complex scenes with fewer environmental features, the accuracy of visual relocalization may be greatly reduced, which poses significant challenges. In addition, existing fusion methods have insufficient processing capabilities for abnormalities of each sensor and cannot provide effective compensation when single sensor data fails. Therefore, in order to meet the requirements of low-cost, high-precision, and strong robust positioning in a large range of self-similar environments, it is necessary to break through the limitations of existing technologies and design a multimodal fusion positioning solution that is more suitable for special indoor environments.

[0005] Publication (Announcement) No. CN 123456789B discloses a visual-inertial positioning method based on pedestrian motion feature optimization. This method uses the visual-inertial sensor data worn on the pedestrian's head to perform posture calculation, obtains the pedestrian's three-dimensional posture information, and combines the heading information to perform local and global optimization of the visual-inertial odometer. Compared with traditional positioning methods that rely on external base stations or signal sources, this technology has the advantages of low cost, no radiation, and no accumulation of errors over time. It can also achieve meter-level precision positioning in complex indoor environments without the need for additional base stations or signal sources. However, the positioning stability of this method may decrease when the visual environment quality is poor or the inertial data is interfered by noise. In addition, this method cannot obtain global coordinates and has certain limitations.

[0006] Patent Publication (Announcement) No. CN113358117B discloses a visual-inertial indoor positioning method using a map, which effectively corrects the output of the visual-inertial odometer through a three-dimensional map matching algorithm based on a conditional random field. Specifically, the method first establishes a conditional random field model of the indoor three-dimensional map and introduces it as prior information into the calculation process; secondly, the pose and trajectory information output by the visual-inertial odometer system is input into the conditional random field model as observations; finally, the positioning result of the visual-inertial odometer system is corrected through the optimal state point sequence output by the conditional random field model. However, the patent has a high reliance on the construction of three-dimensional maps, and the map update and maintenance costs are high in complex and dynamic indoor environments. In addition, the method is not robust enough to visual information, and matching errors are prone to occur in visually challenging scenarios such as lighting changes and texture loss, affecting positioning accuracy. Summary of the invention

[0007] In view of the above problems, the present invention proposes an indoor visual inertial positioning method for a large-scale self-similar environment, which solves the problems of the existing visual inertial positioning system in a large-scale, repetitive structure environment, such as easy mismatching, large cumulative error, complex deployment, and poor real-time performance. Specifically, it includes the following steps:

[0008] To achieve the above object, the technical solution adopted by the present invention is:

[0009] An indoor visual inertial positioning method for a large-scale self-similar environment includes the following steps:

[0010] Step 1: reconstruct and process the dense point cloud of the collected prior video sequence, remove the point cloud of the non-active area and filter out the outliers, project and correct the generated plane map to build the indoor obstacle feature library, and form an effective constraint for the subsequent position solution;

[0011] Step 2: Using spatiotemporal constraints and single-time state estimation probabilities, a hidden Markov estimation model of the current scene is constructed to achieve optimal scene recognition, and the camera pose of the query frame is solved based on the position hypothesis.

[0012] Step 3: Fuse multimodal information and correct the accumulated error of pedestrian trajectory estimation based on Monte Carlo reasoning to complete pedestrian motion tracking and achieve optimal state estimation.

[0013] As a further improvement of the present invention, the step 1 is specifically as follows:

[0014] (1.1) Collect the prior video sequence and perform dense point cloud reconstruction to generate point cloud data P = {(x i ,y i ,z i)}, where i represents the index of the point. The point cloud data is filtered based on the actual area of ​​interest using the regional constraint condition R to remove the clutter in the non-active area. filtered ={(x i ,y i ,z i )∈P|(x i ,y i )∈R}, then, voxel filtering is used to downsample the point cloud to reduce the amount of point cloud data and improve the processing speed. Assuming the voxel grid size is V, the filtered point cloud is P voxel =VoxelGrid(P filtered ,V);

[0015] (1.2) For the voxelized point cloud data P voxel , a method combining distance filtering and morphological filtering is used to remove outliers and obtain a clean point cloud P cleaned Next, the clean point cloud is projected onto the XY plane to generate a plane point set P XY ={(x i ,y i )∣(x i ,y i ,z i )∈P cleaned}, on the XY plane, the indoor plane contour C is generated by dilation, binarization, contour extraction and screening through image processing methods raw The error correction is performed on the contour to obtain the final indoor floor plan C final ;

[0016] (1.3) Use the line segment detection method to extract obstacle features in the indoor plan view and perform matrix processing as constraint information C in subsequent position calculation constraint .

[0017] As a further improvement of the present invention, the step 2 is specifically as follows:

[0018] (2.1) Let the image sequence be I = {I 1 ,I 2 ,...,I T}, where I t is the image at time t, and the position set in the database is represented by Each position p i Corresponding to an image, since the initial position is unknown, all positions in the database are selected as the initial hidden state set S 0 ={s 1 ,s 2 ,...,s N}, and each hidden state s i Have the same confidence probability

[0019] (2.2) At the subsequent moment, by observing image I t and motion state model, for each candidate position p i Perform confidence estimation based on the hidden state at time t-1 and its corresponding position p i,t-1 , combined with the observed motion state v t , calculate the hidden state s i The predicted position p at time t i,t , filter out the hidden state set S with the highest confidence through the confidence update formula t As the candidate position set at time t:

[0020] P t ={p 1,t ,p 2,t ,...,p N,t},P t =f(S t-1 ,v t )

[0021] Among them, f(·) is the function of hidden state transfer and motion state, P t represents the set of candidate positions predicted at the moment based on the motion state and image features;

[0022] (2.3) After obtaining the candidate position set P of the query image sequence t After that, all candidate reference key frames are traversed to perform fast feature point matching, and the EPnP solver is initialized with the matching results to perform posture solution based on Ransac.

[0023] As a further improvement of the present invention, the step 3 is specifically as follows:

[0024] (3.1) Based on the attitude solution result as the initial position of the target, the particle swarm is initialized using Gaussian distribution with the initial position of the target as the center. The particle swarm consists of particle positions and particle weights.

[0025]

[0026] (3.2) The PDR positioning result is used as the state transition quantity to update the position state of the particle swarm. The position update formula of each particle is:

[0027]

[0028] Among them, Δx, Δy are the translation amounts output by PDR;

[0029] (3.3) First, the constraint information is used to determine whether the line connecting the particle position coordinates at the previous and next moments intersects with the line connecting the wall. The particle weight is adjusted according to the intersection result. Then, the distance from the particle to the visual positioning position at that moment is determined. If the distance is greater than the threshold a, it is considered that the particle has deviated from the positioning trajectory and the weight is set to 0. If it is less than the threshold, the particle weight is adaptively adjusted according to the distance difference:

[0030]

[0031] in, is the adaptive coefficient of the ith particle at time t, is the position coordinate, is the coordinate of visual positioning;

[0032] (3.4) Regularly perform heading angle correction based on the attitude of the effective visual positioning result:

[0033] θ f =ω p θ p +ω v θ v

[0034] Among them, θ p and θ v are respectively the heading angle calculated by PDR and vision, ω p and ω v For their respective weights, calculate the sum of particle weights W sum And according to the results, the resampling step is performed, and the state conversion, weight update, heading correction and resampling are repeated to gradually optimize the particle swarm position estimation results.

[0035] The advantages of the present invention compared with the prior art are:

[0036] In view of the problems that the existing visual inertial positioning system is prone to mismatching, large cumulative errors, complex deployment and poor real-time performance in a large-scale, repetitive structure environment, the present invention proposes an indoor visual inertial positioning method for a large-scale self-similar environment. First, a spatial constraint model is constructed based on the prior visual information collected by the visual sensor, and it is added to the system constraint as prior information; then, the hidden Markov estimation model of the current scene is constructed by using the spatiotemporal constraints and the single-time state estimation probability to achieve the best scene recognition, so as to ensure the correct matching and posture recovery in the visual challenge scene; finally, Monte Carlo reasoning is used to integrate the visual information with the map constraints, correct the cumulative error of the pedestrian track calculation, and ensure the long-term tracking performance of the system. In a large-scale self-similar indoor environment, the method can achieve high-precision, low-cost and fast-response indoor positioning through the coordinated fusion of visual and inertial sensors. At the same time, the present invention supports real-time updating of position information in complex environments, and improves the positioning accuracy and system robustness through the effective combination of adaptive visual matching and inertial constraints. Compared with the existing technology, the present invention has the advantages of high computational efficiency, flexible deployment and strong environmental adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A flow chart of the method provided by the present invention;

[0038] Figure 2 Construct result plots for spatially constrained models;

[0039] Figure 3 Schematic diagram of the hidden Markov estimation model. DETAILED DESCRIPTION

[0040] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. The following embodiments are used to illustrate the present invention but are not used to limit the scope of the present invention.

[0041] like Figure 1 As shown, the present invention provides an indoor visual-inertial positioning method for a large-scale self-similar environment, which solves the problems of the existing visual-inertial positioning system in a large-scale, repetitive structure environment, such as easy mismatching, large cumulative error, complex deployment and poor real-time performance.

[0042] As a specific embodiment of the present invention, the technical solution adopted by the present invention is:

[0043] Step S1: Reconstruct and process the dense point cloud of the collected prior video sequence, remove the point cloud of the inactive area and filter out the outliers, project and correct the generated plane map to build the indoor obstacle feature library, and form an effective constraint for the subsequent position solution. Specifically include:

[0044] S1.1: Collect the prior video sequence and perform dense point cloud reconstruction to generate point cloud data P containing building structures and noise points = {(x i ,y i ,z i )}, where i represents the index of the point. According to the actual area of ​​interest, the region constraint condition R is used to remove the clutter in the non-active area and filter the point cloud data P filtered ={(x i ,y i ,z i )∈P|(x i ,y i )∈R}. Then, voxel filtering is used to downsample the point cloud to reduce the amount of point cloud data and improve the processing speed. Assuming the voxel grid size is V, the filtered point cloud is P voxel =VoxelGrid(P filtered ,V);

[0045] S1.2: voxelized point cloud data P voxel , a method combining distance filtering and morphological filtering is used to remove outliers and obtain a clean point cloud P cleaned Next, the clean point cloud is projected onto the XY plane to generate a plane point set P XY ={(x i ,y i )∣(x i ,y i ,z i )∈P cleaned On the XY plane, the indoor plane contour C is generated by dilation, binarization, contour extraction and screening through image processing methods. raw The error correction is performed on the contour to obtain the final indoor floor plan C final .deal with;

[0046] S1.3: Use the line segment detection method to extract obstacle features in the indoor plan view and perform matrix processing as constraint information C in subsequent position calculation constraint .

[0047] Step S2: Using spatiotemporal constraints and single-time state estimation probability, construct a hidden Markov estimation model of the current scene to achieve optimal scene recognition, and complete the camera pose solution of the query frame based on the position hypothesis. Specifically including:

[0048] S2.1: Let the image sequence be I = {I 1 ,I 2 ,...,I T}, where I t is the image at time t. The position set in the database is represented by Each position pi Corresponding to an image. Since the initial position is unknown, all positions in the database are selected as the initial hidden state set S 0 ={s 1 ,s 2 ,...,s N}, and each hidden state s i Have the same confidence probability

[0049] S2.2: At a subsequent time, by observing image I t and motion state model, for each candidate position p i To perform confidence estimation, the process is as follows Figure 2 As shown. According to the hidden state at time t-1 and its corresponding position p i,t-1 , since the pedestrian's walking distance per step will not exceed 2 meters, combined with the observed motion state v t , set the search range, calculate the hidden state s i The predicted position p at time t i,t . The hidden state set S with the highest confidence is selected through the confidence update formula t As the candidate position set at time t:

[0050] P t ={p 1,t ,p 2,t ,...,p N,t},P t =f(S t-1 ,v t )

[0051] Among them, f(·) is the function of hidden state transfer and motion state, P t represents the set of candidate positions predicted at the moment based on the motion state and image features;

[0052] S2.3: After obtaining the candidate position set P of the query image sequence t After that, all candidate reference keyframes are traversed, and feature points are quickly matched through BOW bags of words. The matching results are used to initialize the EPnP solver, and the posture is solved based on Ransac. At the same time, the number of internal points of the estimated posture is determined. If it is greater than the predefined threshold, the iteration is stopped in advance, which improves the system operation efficiency while meeting the calculation accuracy.

[0053] Step S3: Fusing multimodal information to correct the accumulated error of pedestrian dead reckoning based on Monte Carlo reasoning to complete pedestrian motion tracking and achieve optimal state estimation, such as Figure 3 Specifically including:

[0054] S3.1: Based on the attitude solution result as the initial position of the target, the particle swarm is initialized using Gaussian distribution with the initial position of the target as the center. The particle swarm consists of particle positions and particle weights. The particle weight is represented by w i , initially, the weights of all particles are equal and normalized to 1;

[0055] S3.2: The PDR positioning result is used as the state transition quantity to update the position state of the particle swarm. The position update formula of each particle is:

[0056]

[0057] Among them, Δx, Δy are the translation amounts output by PDR;

[0058] S3.3: First, use the constraint information to determine whether the line connecting the particle position coordinates at the previous and next moments intersects with the line connecting the wall. If so, set the weight of the particle that passes through the wall to 0. Then, determine the reliability of the visual positioning result based on the number of internal points. If the result is reliable, determine the distance from the particle to the visual positioning position at that moment. If the distance is greater than the threshold, it is considered that the particle has deviated from the positioning trajectory and the weight is set to 0; if it is less than the threshold, the particle weight is adaptively adjusted according to a:

[0059]

[0060] in, is the adaptive coefficient of the ith particle at time t, is the position coordinate, is the coordinate of visual positioning;

[0061] S3.4: Regularly perform heading angle correction based on the attitude of the valid visual positioning result:

[0062] θ f =ω p θ p +ω v θ v

[0063] Among them, θ p and θ v are respectively the heading angle calculated by PDR and vision, ω p and ω v For their respective weights, we can give them both equal weights, 1 / 2 for each. Calculate the sum of particle weights W sum And according to the results, the resampling step is performed, and the state conversion, weight update, heading correction and resampling are repeated to gradually optimize the particle swarm position estimation results.

[0064] The above description is only a preferred embodiment of the present invention and does not constitute any other form of limitation to the present invention. Any modification or equivalent change made based on the technical essence of the present invention still falls within the scope of protection required by the present invention.

Claims

1. A method for indoor visual-inertial positioning in a large-scale self-similar environment, characterized by: The following steps are involved: Step 1: reconstruct and process the dense point cloud of the collected prior video sequence, remove the point cloud of the non-active area and filter out the outliers, project and correct the generated plane map to build the indoor obstacle feature library, and form an effective constraint for the subsequent position solution; Step 2: Using spatiotemporal constraints and single-time state estimation probabilities, a hidden Markov estimation model of the current scene is constructed to achieve optimal scene recognition, and the camera pose of the query frame is solved based on the position hypothesis. Step 3: Fuse multimodal information and correct the accumulated error of pedestrian trajectory estimation based on Monte Carlo reasoning to complete pedestrian motion tracking and achieve optimal state estimation.

2. The indoor visual inertial positioning method for a large-scale self-similar environment according to claim 1, characterized in that: The step 1 is specifically as follows: (1.1) Collect the prior video sequence and perform dense point cloud reconstruction to generate point cloud data P = {(x i ,y i ,z i )}, where i represents the index of the point. The point cloud data is filtered based on the actual area of ​​interest using the regional constraint condition R to remove the clutter in the non-active area. filtered ={(x i ,y i ,z i )∈P|(x i ,y i )∈R}, then, voxel filtering is used to downsample the point cloud to reduce the amount of point cloud data and improve the processing speed. Assuming the voxel grid size is V, the filtered point cloud is P voxel =VoxelGrid(P filtered ,V); (1.2) For the voxelized point cloud data P voxel , a method combining distance filtering and morphological filtering is used to remove outliers and obtain a clean point cloud P cleaned Next, the clean point cloud is projected onto the XY plane to generate a plane point set P XY ={(x i ,y i )∣(x i ,y i ,z i )∈P cleaned }, on the XY plane, the indoor plane contour C is generated by dilation, binarization, contour extraction and screening through image processing methods raw The error correction is performed on the contour to obtain the final indoor floor plan C final ; (1.3) Use the line segment detection method to extract obstacle features in the indoor plan view and perform matrix processing as constraint information C in subsequent position calculation constraint .

3. The indoor visual inertial positioning method for a large-scale self-similar environment according to claim 1, characterized in that: The step 2 is specifically as follows: (2.1) Let the image sequence be I = {I1, I2, ..., I T }, where I t is the image at time t, and the position set in the database is represented by Each position p i For an image, since the initial position is unknown, all positions in the database are selected as the initial hidden state set S0 = {s1, s2, ..., s N }, and each hidden state s i Have the same confidence probability (2.2) At the subsequent moment, by observing image I t and motion state model, for each candidate position p i Perform confidence estimation based on the hidden state at time t-1 and its corresponding position p i,t-1 , combined with the observed motion state v t , calculate the hidden state s i The predicted position p at time t i,t , filter out the hidden state set S with the highest confidence through the confidence update formula t As the candidate position set at time t: P t ={p 1,t ,p 2,t ,...,p N,t },P t =f(S t-1 ,v t ) Among them, f(·) is the function of hidden state transfer and motion state, P t represents the set of candidate positions predicted at the moment according to the motion state and image features; (2.3) After obtaining the candidate position set P of the query image sequence t After that, all candidate reference key frames are traversed to perform fast feature point matching, and the EPnP solver is initialized with the matching results to perform posture solution based on Ransac.

4. The indoor visual inertial positioning method for a large-scale self-similar environment according to claim 1, characterized in that: The step 3 is as follows: (3.1) Based on the attitude solution result as the initial position of the target, the particle swarm is initialized using Gaussian distribution with the initial position of the target as the center. The particle swarm consists of particle positions and particle weights. (3.2) The PDR positioning result is used as the state transition quantity to update the position state of the particle swarm. The position update formula of each particle is: Among them, Δx, Δy are the translation amounts output by PDR; (3.3) First, the constraint information is used to determine whether the line connecting the particle position coordinates at the previous and next moments intersects with the line connecting the wall. The particle weight is adjusted according to the intersection result. Then, the distance from the particle to the visual positioning position at that moment is determined. If the distance is greater than the threshold a, it is considered that the particle has deviated from the positioning trajectory and the weight is set to 0. If it is less than the threshold, the particle weight is adaptively adjusted according to the distance difference: in, is the adaptive coefficient of the ith particle at time t, is the position coordinate, is the coordinate of visual positioning; (3.4) Regularly perform heading angle correction based on the attitude of the effective visual positioning result: i f =ω p i p +oh v i v Among them, θ p and θ v are respectively the heading angle calculated by PDR and vision, ω p and ω v For their respective weights, calculate the sum of particle weights W sum And according to the results, the resampling step is performed, and the state conversion, weight update, heading correction and resampling are repeated to gradually optimize the particle swarm position estimation results.

Images