Cloud visual positioning data evaluation and pose optimization method based on multi-parameter fusion

Abstract

The invention relates to a positioning technology integrating computer vision, vision positioning and IMU (Inertial Measurement Unit) sensors, in particular to a cloud vision positioning data evaluation and pose optimization method based on multi-parameter fusion. According to the pose evaluation model, the continuity of tracking data of an equipment camera or equipment, the consistency of the orientation of the equipment camera or equipment, the distance rationality of positioning feature points, the size of a re-projection error and the stability of IMU (Inertial Measurement Unit) data are comprehensively considered, and the pose of the equipment camera or equipment is dynamically adjusted. When the score is higher than a preset threshold value, updating the equipment camera or the equipment pose by adopting cloud visual positioning data; otherwise, keeping the current pose. According to the strategy, the pose of the equipment camera or the equipment is optimized, the visual positioning precision and the system stability are improved, and the user experience is improved.

Description

Technical Field

[0001] The present invention relates to the field of positioning technology, and in particular to positioning technology combining computer vision, visual positioning, and IMU sensor fusion, and in particular to a cloud-based visual positioning data evaluation and posture optimization method based on multi-parameter fusion. Background Art

[0002] Common existing visual positioning systems mostly use cloud-based and terminal-based positioning technology. Cloud-based positioning provides the absolute position of the device camera or device on the map, while the terminal uses SLAM technology to provide information about the device camera or device's movement within the map. Due to the scale difference between the device camera or device's map and the cloud map, as well as the accumulated errors of the SLAM system during movement, we need to continuously use cloud-based positioning to correct the absolute position of the device camera or device on the map. However, cloud-based positioning data is affected by factors such as map accuracy, on-site environment, and calculation errors, and its accuracy is unstable. Directly replacing the existing position of the device camera or device with cloud-based data may result in reduced positioning accuracy or system instability. Therefore, an effective evaluation mechanism is required to determine the reliability of cloud-based positioning data and optimize the real-time position of the device camera or device accordingly. Summary of the Invention

[0003] The present invention aims to provide a cloud-based visual positioning data evaluation and posture optimization method based on multi-parameter fusion. By comprehensively considering multiple sensor data and cloud-based visual positioning results, the reliability of cloud-based visual positioning data can be accurately evaluated, and the device camera or device posture can be adjusted accordingly to improve positioning accuracy and system stability.

[0004] The specific scheme of the present invention is as follows: (1) Real-time collection of tracking data (including displacement D, d) from the device camera or device, device orientation R, cloud-based visual positioning results (including R0, distance between the positioning feature point and the device camera or device, etc.), angular velocity, acceleration, and other data, and pre-processing; (2) Construct an evaluation model that uses the above parameters as input and uses a series of algorithms to determine the quality of the current pose through reprojection error, calculate the average distance between the 3D point and the device camera or device, calculate the horizontal angle, calculate the standard deviation (SD), calculate the orientation, calculate the height value, and determine whether to replace the new pose. This algorithm calculates the credibility score of the cloud-based visual positioning data. This score comprehensively considers the continuity of the tracking data from the device camera or device, the consistency of the device camera or device orientation, the reasonableness of the distance between the positioning feature points, the size of the reprojection error, and the stability of the IMU data. (3) Finally, dynamically adjust the device camera or device pose based on the cloud-based visual positioning data score obtained in (2). If the score is above a preset threshold, the device camera or device pose is updated using the cloud-based visual positioning data; otherwise, the current pose is maintained.

[0005] The present invention constructs a posture evaluation model through multi-parameter fusion and dynamic adjustment strategies, including a reprojection error model, an average distance model between a 3D point and a device camera or device, a horizontal angle model, a standard deviation SD model, an orientation model, and an altitude value model, and then calculates the credibility score of the cloud-based visual positioning data. The posture evaluation model score comprehensively considers the continuity of the tracking data of the device camera or device, the consistency of the orientation of the device camera or device, the rationality of the distance of the positioning feature points, the size of the reprojection error, and the stability of the IMU data, and dynamically adjusts the posture of the device camera or device. When the score is higher than the preset threshold, the cloud-based visual positioning data is used to update the device camera or device posture; otherwise, the current posture is maintained. This strategy optimizes the posture of the device camera or device, improves the visual positioning accuracy and system stability, and enhances the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 This is a schematic diagram of shooting under bright light and with obstruction; Figure 2 for Figure 1 Comparison of the ground truth before and after optimization in bright light with occlusion. Figure 3 This is a schematic diagram of shooting in dim light with obstructions; Figure 4 for Figure 3 Comparison of positioning before and after optimization and ground truth in dim light and with occlusion. DETAILED DESCRIPTION

[0007] The specific implementation of the present invention is as follows: Step 1: Cloud and terminal integrated positioning, computing equipment The cloud-terminal integrated positioning technology is a positioning technology that combines the absolute position of the device camera or device on the map provided by cloud positioning with the movement information of the device camera or device on the map provided by the terminal using SLAM technology.

[0008] The device camera or device can provide the image Image taken by its camera at a certain moment, and the camera internal parameter matrix K at that moment, which can be uploaded to the cloud and calculated through cloud positioning to obtain the device camera or device's current position in the map. , which is a set of low-frequency data, usually 0.3-2Hz.

[0009] The device camera or the device's SLAM system can provide us with the device camera or device's position D in the world coordinate system and its orientation R. This is a set of high-frequency data that is the same as the screen refresh rate, usually up to 60Hz. This data is combined with the cloud positioning results to obtain the device camera or device's position at every moment in the map. .

[0010] pass Already corresponding to it The position of the map in the world coordinate system can be calculated ,

[0011] The position T of the device camera or device in the map at any time ; Since cloud positioning data is affected by factors such as map accuracy, on-site environment, and calculation errors, its accuracy is not stable. Directly using cloud data to replace the existing posture may lead to a decrease in positioning accuracy or system instability. Therefore, the present invention adopts a comprehensive optimization method, which uses the above-mentioned real-time acquisition of the device camera or the device's posture information in the world coordinate system to obtain the position information of the device. (including position D and orientation R), combined with cloud-based visual positioning results The following evaluation mechanism is introduced to determine the reliability of cloud positioning data (including the orientation R0 and the distance d between the positioning feature point and the device camera or device), thereby optimizing the device camera or device's posture.

[0012] Step 2: Use reprojection error to determine the quality of the current pose The current position of the device camera or device is the position and posture of the device camera or device in the map at that moment, which is obtained by integrating the positioning result of the previous positioning and the device camera or device tracking system. , its confidence meter is .

[0013] The device camera or device pose obtained by cloud computing is used to , its confidence meter is .

[0014] The set of matched 2D-3D point pairs obtained by cloud computing is , where a 3D point P(X,Y,Z) in the map, the camera's internal matrix K can be obtained from the device, at the current pose Next, the projection in the picture is , which is calculated as follows:

[0015] The reprojection error e is the actual observed image point Compared with the theoretical projection point calculated by the camera model The distance on the two-dimensional image plane, this distance is usually the Euclidean distance:

[0016] The mean of the reprojection error of n points is:

[0017] when Less than a fixed value When we think the quality of the current pose is good enough, no update is needed. Will be discarded, continue to use , skip the following steps directly.

[0018] Step 3: Calculate the average distance between the 3D point and the object or camera The farther the distance d of a 3D point from the device camera or device, the lower its confidence The lower the value, the closer the distance d between the 3D point and the device camera or device is, and the confidence The higher, that is, when d is greater than The posterior confidence is low when d is less than When you can trust it completely.

[0019] .

[0020] Step 4: Calculate the horizontal angle Cloud-based computing is the device camera or device's position in the map, the device camera or device's position in the world It can be obtained from the device camera or device. Then the position of the map in the world is:

[0021] from The horizontal angle can be decomposed from . Since our maps are all leveled, the larger the angle, the higher the confidence level. The lower.

[0022] .

[0023] Step 5: Calculate the standard deviation SD The more dispersed the 2D points used in cloud-based visual positioning are in the visual positioning image, the more accurate the visual positioning results will be. The degree of dispersion of the data can be measured using the standard deviation (SD), which is calculated as follows:

[0024] SD and confidence The relationship is: .

[0025] Step 6: Calculate the orientation Since the map is calibrated to the same direction as the real world, the directions of east, south, west and north can be calculated from the cloud visual positioning system. Decompose the device camera or device's real-world orientation , you can get a direction value directly from the device camera or device , their difference is calculated as:

[0026] and confidence The relationship is: .

[0027] Step 7: Calculate the height Calculated from the cloud-based visual positioning system Decompose the height value from the device camera or device map , the distance between the key frame height values ​​used in the calculation The most recent keyframe height value is , their difference:

[0028] .

[0029] Step 8: Determine whether to replace the new pose The device camera or device pose confidence calculated in the cloud-based visual positioning system:

[0030] The confidence of the previous positioning will gradually decrease as the device camera or device moves. and The displacement of the device camera or device record at two moments is d, is a constant, the confidence of the previous positioning becomes

[0031] if > Adopt As the device camera or device pose, if ≤ Keep The pose of the device camera or device remains unchanged.

[0032] Step 9: The influence of light and dark In the present invention, we all use grayscale images to realize the construction of three-dimensional point cloud maps and SLAM positioning. Using grayscale images for mapping and positioning can highlight the structural information such as the edge, contour and texture of the object, so that the mapping and positioning algorithm can more attentively extract features related to the object structure, reduce the interference of irrelevant information such as color, thereby improving the precision and accuracy of point cloud mapping and visual SLAM positioning. In addition, compared with color images, grayscale images are relatively less sensitive to changes in illumination. Grayscale images mainly focus on brightness information. Although changes in illumination can also affect their grayscale values, they have less impact on the structural information of the image as a whole. Therefore, using grayscale images for three-dimensional point cloud mapping and visual SLAM positioning has better stability and robustness under different lighting environments and can adapt to a wider range of scenes.

[0033] In addition, in the present invention, the feature points are mainly selected from the edges or intersections of the contours of the object, and these structural information are less affected by illumination.

[0034] Step 10: The impact of occlusion At the same time, in this invention, we use panoramic images for 3D point cloud mapping, constructing a more accurate, comprehensive, and high-precision 3D point cloud map. To facilitate user positioning, a mobile phone is used for visual SLAM positioning. When there are obstructions, positioning mainly relies on the mobile phone terminal's SLAM. Because the panoramic grayscale image and structural information such as the edge or outline of the object are used as feature points to identify obstructions, obstructions only affect the positioning response speed and do not affect the success rate and accuracy of positioning.

[0035] Combine Figures 1 to 4 , the traditional terminal using SLAM technology in the background technology is compared with the optimization solution of the present invention, and three comparisons are made with the ground truth. The position difference and angle difference statistics are shown in the following table: Table 1. Position data of mobile phone under bright light and with occlusion

[0036] Table 2. Position data of mobile phones in dim light and with obstructions

[0037] As can be seen from the above figures and tables, the optimization solution of the present invention is closest to the ground truth, and the average and maximum differences in position and angle are significantly reduced, effectively controlling the cumulative error.

[0038] Glossary Cloud-based visual positioning system: refers to a system that uses a cloud server to process image data obtained from a device camera or device in real time, and integrates it with the movement information of the device camera or device in the map provided by the terminal using SLAM technology to determine the position and orientation of the device camera or device itself in space.

[0039] Reprojection error: The deviation between the current visual positioning feature point after reprojection based on the existing pose and the actual observed position.

[0040] Pose: includes the angular velocity and acceleration of the device's camera or device. Angular velocity refers to the speed of the device's rotation around its axis, obtained through sensors such as the device's camera or IMU. Acceleration refers to the acceleration of the device's linear motion, obtained through sensors such as the device's camera or IMU.

Claims

1. A cloud-based visual positioning data evaluation and pose optimization method based on multi-parameter fusion, characterized by The steps include: Step 1: Cloud and terminal integrated positioning, computing equipment The device camera or device provides the captured image Image, the camera internal parameter matrix K at this moment, which is uploaded to the cloud and the position of the device camera or device in the map is obtained through cloud positioning calculation. , is a set of low-frequency data, usually 0.3-2Hz; the device camera or the device's SLAM system provides the position D in the world coordinate system, and the direction R, is a set of high-frequency data with the same refresh rate as the screen, usually 60Hz; this data is combined with the cloud positioning result to obtain the device camera or device's position at each moment in the map. pass Already corresponding to it The position of the map in the world coordinate system can be calculated , The position T of the device camera or device in the map at any time By utilizing the above real-time collected device camera or device pose information in the world coordinate system , including position D and orientation R, combined with cloud-based visual positioning results , including the orientation R0, the distance d between the positioning feature point and the device camera or device, and introduce the following evaluation mechanism to judge the reliability of the cloud positioning data, so as to optimize the position of the device camera or device; Step 2: Use reprojection error to determine the quality of the current pose The current position of the device camera or device is the position and posture of the device camera or device in the map at that moment, which is obtained by integrating the positioning result of the previous positioning and the device camera or device tracking system. , its confidence meter is ; The device camera or device pose obtained by cloud computing is used to , its confidence meter is ; The set of matched 2D-3D point pairs obtained by cloud computing is , where a 3D point P(X,Y,Z) in the map, the camera's internal matrix K is obtained from the device, at the current pose Next, the projection in the picture is , which is calculated as follows: The reprojection error e is the actual observed image point Compared with the theoretical projection point calculated by the camera model The distance on the two-dimensional image plane, this distance is usually the Euclidean distance: The mean of the reprojection error of n points is: when Less than a fixed value When we think the quality of the current pose is good enough, no update is needed. Will be discarded, continue to use .

2. The cloud-based visual positioning data evaluation and posture optimization method based on multi-parameter fusion according to claim 1 is characterized in that when Greater than a fixed value The following steps are also included: Step 3: Calculate the average distance between the 3D point and the object or camera The farther the distance d of a 3D point from the device camera or device, the lower its confidence The lower the value, the closer the distance d between the 3D point and the device camera or device is, and the confidence The higher the value, that is, when d is greater than The posterior confidence is low when d is less than When you are in a hurry, you can trust it completely; Step 4: Calculate the horizontal angle Cloud-based computing is the device camera or device's position in the map, the device camera or device's position in the world It can be obtained from the device camera or device; then the position of the map in the world: from The horizontal angle angle can be decomposed from it; since our maps are all leveled, the larger the angle, the higher the confidence level. The lower; Step 5: Calculate the standard deviation SD The more dispersed the 2D points used in cloud-based visual positioning are in the visual positioning image, the more accurate the visual positioning results. The degree of data dispersion can be measured using the standard deviation (SD), calculated as follows: SD and confidence The relationship is: Step 6: Calculate the orientation Since the map is calibrated to the same direction as the real world, the directions of east, south, west and north are calculated from the cloud-based visual positioning system. Decompose the device camera or device's real-world orientation , you can get a direction value directly from the device camera or device , their difference is calculated as: o and confidence The relationship is: Step 7: Calculate the height Calculated from the cloud-based visual positioning system Decompose the height value from the device camera or device map , the distance between the key frame height values ​​used in the calculation The most recent keyframe height value is , their difference: Step 8: Determine whether to replace the new pose The device camera or device pose confidence calculated in the cloud-based visual positioning system: The confidence of the previous positioning will gradually decrease as the device camera or device moves. and The displacement of the device camera or device record at two moments is d, is a constant, the confidence of the previous positioning becomes if > Adopt As the device camera or device pose, if ≤ Keep The pose of the device camera or device remains unchanged; Step 9: The influence of light and dark Grayscale images are used to build three-dimensional point cloud maps and perform SLAM positioning.

3. The cloud-based visual positioning data evaluation and posture optimization method based on multi-parameter fusion according to claim 1 is characterized in that The following steps are also included: Step 10: The impact of occlusion Use panoramic images for 3D point cloud mapping; use mobile phones for visual SLAM positioning; when there are obstructions, rely on the SLAM of the mobile terminal for positioning, and use the panoramic grayscale image and the edge or contour structure information of the object as feature points for occlusion identification.

Images