Tower pole pile position positioning method based on mixed reality technology and related equipment

Abstract

The invention belongs to the technical field of pile position instrument positioning, and particularly relates to a tower pole pile position positioning method based on the mixed reality technology and related equipment, the method comprises the steps that detection data of pile fixing equipment and timestamps of the detection data are acquired through multiple types of acquisition equipment, and the detection data comprise image data, IMU data and GNSS data; performing time synchronization on the image data, the IMU data and the GNSS data based on the timestamps of the detection data; constructing a visual inertial odometer positioning model, and calculating first positioning of the image acquisition module through the visual inertial odometer positioning model according to the image data after time synchronization and the IMU data; and performing position correction on the first location based on the GNSS data, determining a second location of the image acquisition module, and taking the second location as a target pile position of the tower pole. According to the invention, high precision and robustness of vision are utilized, and global drifting-free performance of GNSS is combined, so that high-precision positioning can be realized in various scenes.

Description

Technical Field

[0001] The present invention belongs to the technical field of pile position meter positioning, and in particular relates to a tower pile position positioning method and related equipment based on mixed reality technology. Background Art

[0002] With the rapid development of distribution lines and the increase in intelligent demand, higher requirements are placed on the measurement accuracy and reliability of line pile positions in distribution lines. The existing pile positions are located by pile position meters. Conventional pile position meters use algorithms based on magnetic field measurement and signal processing for positioning, which mainly rely on the measurement of parameters such as magnetic field strength and direction, and determine the position of the pile position based on the analysis of the acquired data. However, the magnetic field measurement itself is affected by many factors, which may lead to inaccurate magnetic field measurement data, thereby affecting the accuracy of the positioning algorithm. For example, the magnetic field generated by the surrounding electromagnetic equipment or underground pipelines will interfere with the measurement of the pile position meter, causing the measurement results to deviate from the actual pile position. In addition, there are often various uncertain factors at the construction site, such as weather changes, construction disturbances, etc. These factors will cause fluctuations and anomalies in the magnetic field measurement data, which will lead to instability and deviation in the positioning results. That is, the positioning algorithm based on magnetic field data is not adaptable to environmental changes and abnormal conditions, which will lead to poor robustness.

[0003] In summary, the existing pile position meters are based on magnetic field data for positioning during the positioning process. However, the magnetic field is affected by environmental interference and has low measurement stability. The positioning algorithm based on magnetic field data is not adaptable to environmental changes and abnormal conditions, which will lead to low line pile position measurement accuracy and poor robustness. Summary of the invention

[0004] The purpose of the present invention is to solve the problem that the pile position meter in the prior art performs positioning based on magnetic field data during positioning, and the magnetic field is disturbed by the environment, the measurement stability is low, and the positioning algorithm based on magnetic field data is not adaptable to environmental changes and abnormal conditions, which will lead to low measurement accuracy and poor robustness of the line pile position. A tower pile position positioning method and related equipment based on mixed reality technology are proposed. The method collects image data, IMU data and GNSS data respectively by using multiple types of sensors, and synchronizes the image data, IMU data and GNSS data based on timestamps, which can reduce the error caused by time asynchrony and improve the accuracy of the data; by constructing a VIO positioning model, the image data and IMU data after time synchronization are data fused based on the VIO positioning model to determine the first positioning, which not only utilizes the high precision and strong robustness of visual acquisition, but also corrects and compensates the position drift determined based on the image data through IMU data, which is more conducive to maintaining good environmental adaptability and improving the accuracy of the first positioning; by using the global drift-free property of GNSS data to correct and compensate the first positioning, the adaptability is strong, and high-precision positioning can be achieved in various scenarios to obtain a more accurate target pile position.

[0005] In a first aspect, the present invention provides a technical solution: a tower pole pile position positioning method based on mixed reality technology, comprising the following steps:

[0006] Acquire detection data of the staking device and timestamps of each detection data through multiple types of acquisition devices, wherein the detection data includes image data, IMU data and GNSS data of the staking device;

[0007] Performing time synchronization on the image data, the IMU data and the GNSS data based on the timestamp of each of the detection data;

[0008] Constructing a visual inertial odometry positioning model, and calculating a first positioning of an image acquisition module according to the time-synchronized image data and the IMU data through the visual inertial odometry positioning model;

[0009] The first positioning is corrected based on the GNSS data, a second positioning of the image acquisition module is determined, and the second positioning is used as a target pile position of the tower pole.

[0010] Furthermore, the acquisition device includes an image acquisition module, an IMU module and a GNSS module, and the acquisition of the detection data of the staking device and the timestamp of each detection data by using multiple types of acquisition devices includes:

[0011] Collecting image data of the staking device by the image acquisition module and recording a timestamp of the collected image data;

[0012] Collecting IMU data of the staking device through the IMU module and recording the timestamp of collecting the IMU data;

[0013] The GNSS data of the staking device is collected through the GNSS module and a timestamp of collecting the GNSS data is recorded.

[0014] Furthermore, after the step of acquiring the detection data of the staking device and the timestamp of each detection data through multiple types of acquisition devices, the method further includes:

[0015] The image acquisition module, IMU module and GNSS module in the staking device are calibrated respectively to obtain calibration parameters of the image acquisition module, IMU module and GNSS module.

[0016] Furthermore, the calibrating the image acquisition module, the IMU module and the GNSS module in the staking device respectively to obtain calibration parameters of the image acquisition module, the IMU module and the GNSS module includes:

[0017] Extracting feature points based on the sample image of the image acquisition module, matching the extracted feature points with the three-dimensional coordinate points of the sample image, and calculating calibration parameters of the image acquisition module according to the matching results, wherein the calibration parameters of the image acquisition module include an intrinsic parameter matrix and an extrinsic parameter matrix;

[0018] Static calibration and dynamic calibration are performed based on the IMU sample data of the IMU module to obtain calibration parameters of the IMU module, where the calibration parameters of the IMU module include a scale factor and an offset parameter;

[0019] Based on the GNSS sample data of the GNSS module, an error analysis is performed through an error model, and calibration parameters of the GNSS module are calculated according to the error analysis results. The calibration parameters of the GNSS module include multiple error parameters.

[0020] Furthermore, the time synchronization of the image data, the IMU data and the GNSS data based on the timestamp of each of the detection data includes:

[0021] Selecting a reference timestamp from the timestamps of the image data, IMU data and GNSS data;

[0022] Based on a timestamp alignment algorithm, the other timestamps are aligned with the reference timestamp to control the time synchronization of the image data, IMU data and GNSS data.

[0023] Furthermore, the constructing of the visual inertial odometry positioning model, and calculating the first positioning of the image acquisition module according to the time-synchronized image data and the IMU data by the visual inertial odometry positioning model, includes:

[0024] Constructing the visual inertial odometry positioning model based on image training samples and IMU training samples;

[0025] Extracting key features of the image data by using a feature extraction algorithm of the visual inertial odometry positioning model, performing feature point matching according to the key features, and obtaining a matching relationship between the key features;

[0026] Performing motion estimation based on the matching relationship of each of the key features, acquiring motion information of the image acquisition module, and calculating a first relative displacement of the image acquisition module based on the motion information;

[0027] Perform pre-integration based on the IMU data to obtain a second relative displacement of the IMU module;

[0028] The first relative displacement is drift compensated by using the second relative displacement to obtain a first positioning of the image acquisition module.

[0029] Further, the IMU data includes acceleration and angular velocity of the IMU module, and the pre-integration based on the IMU data includes:

[0030] Pre-integrating the acceleration to obtain the rotation angle of the IMU module;

[0031] Integrate the acceleration to obtain position change data of the IMU module;

[0032] The second relative displacement is determined according to the posture change data and the position change data of the IMU module.

[0033] Further, the GNSS data includes satellite positioning data of the pile fixing device, and the position correction of the first positioning based on the GNSS data, determining the second positioning of the image acquisition module, and using the second positioning as the target pile position of the tower pole includes:

[0034] Based on the satellite positioning data and the first positioning, determining the positioning error of the image acquisition module by an error estimation algorithm;

[0035] The first positioning is difference compensated based on the positioning error to obtain the second positioning, and the second positioning is used as the target pile position of the tower pole.

[0036] In a second aspect, the present invention further provides a technical solution: a tower pile position positioning device based on mixed reality technology, comprising:

[0037] A data acquisition module, used for acquiring detection data of the staking device and timestamps of each detection data through multiple types of acquisition devices, wherein the detection data includes image data, IMU data and GNSS data of the staking device;

[0038] A time synchronization module, used for performing time synchronization on the image data, IMU data and GNSS data based on the timestamp of each detection data;

[0039] A position calculation module, used to construct a visual inertial odometer positioning model, and calculate a first position of the image acquisition module according to the time-synchronized image data and the IMU data through the visual inertial odometer positioning model;

[0040] A position correction module is used to perform position correction on the first positioning based on the GNSS data, determine a second positioning of the image acquisition module, and use the second positioning as a target pile position of the tower pole.

[0041] In the third aspect, the present invention also provides a technical solution: an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the tower pile position positioning method based on mixed reality technology described in the first aspect when executing the computer program.

[0042] The beneficial effects achieved by the present invention are as follows: the tower pole pile position positioning method based on mixed reality technology provided by the present invention can reduce the error caused by time asynchrony and improve the accuracy of data by respectively collecting image data, IMU data and GNSS data using multiple types of sensors, and performing time synchronization on the image data, IMU data and GNSS data based on timestamps; by constructing a VIO positioning model, data fusion is performed on the time-synchronized image data and IMU data based on the VIO positioning model to determine the first positioning, which not only utilizes the high precision and strong robustness of visual acquisition, but also corrects and compensates for the position drift determined based on the image data through the IMU data, which is more conducive to maintaining good environmental adaptability and improving the accuracy of the first positioning; by using the global drift-free property of GNSS data to correct and compensate for the first positioning, it has strong adaptability, can achieve high-precision positioning in various scenarios, and obtain a more accurate target pile position.

[0043] The above invention content is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Other features, objects and advantages of the present invention will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings. The drawings are only for the purpose of illustrating preferred embodiments and are not to be considered as limiting the present invention. Also, the same reference symbols are used throughout the drawings to represent the same parts.

[0045] Figure 1 is a flow chart of a tower pole pile position positioning method based on mixed reality technology provided by an embodiment of the present invention;

[0046] Figure 2 is a structural schematic diagram of a tower pile position positioning device based on mixed reality technology provided by an embodiment of the present invention;

[0047] Figure 3 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation method described herein is only an optimal embodiment of the present invention, which is only used to explain the present invention and does not limit the scope of protection of the present invention. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0049] Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the operations (or steps) as sequential processes, many of the operations (or steps) therein can be implemented in parallel, concurrently, or simultaneously. In addition, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the drawings; the process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0050] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. It should also be understood that in various embodiments of the present invention, the size of the sequence number of each process does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0051] It should be understood that in the present invention, "plurality" refers to two or more than two. "And / or" is merely a variable relationship that describes associated objects, indicating that three relationships may exist. For example, and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated with each other are in an "or" relationship. "Contains A, B and C", "Contains A, B, C" means that A, B, and C are all included, "Contains A, B or C" means that one of A, B, and C is included, and "Contains A, B and / or C" means that any one, any two, or any three of A, B, and C are included.

[0052] It should be understood that in the present invention, "B corresponding to A", "B corresponding to A", "A corresponds to B" or "B corresponds to A" means that B is associated with A and B can be determined based on A. Determining B based on A does not mean determining B based only on A, but B can also be determined based on A and / or other information. A and B match when the similarity between A and B is greater than or equal to a preset threshold.

[0053] Embodiment 1

[0054] Please refer to Figure 1 , Figure 1 1 is a flow chart of a tower pole pile position positioning method based on mixed reality technology provided by an embodiment of the present invention. The tower pole pile position positioning method based on mixed reality technology specifically includes the following steps:

[0055] S1. Acquire detection data of the staking device and timestamps of each detection data through multiple types of acquisition devices, wherein the detection data includes image data, IMU data and GNSS data of the staking device;

[0056] S2. Time synchronization of image data, IMU data and GNSS data based on the timestamp of each detection data;

[0057] S3, constructing a visual inertial odometer positioning model, and calculating the first positioning of the image acquisition module according to the time-synchronized image data and IMU data through the visual inertial odometer positioning model;

[0058] S4. Correct the first positioning based on the GNSS data, determine the second positioning of the image acquisition module, and use the second positioning as the target pile position of the tower.

[0059] In this embodiment, the tower pole position positioning method based on mixed reality technology is applicable to the scenario of positioning the tower pole position in the distribution line. The electronic device that executes the tower pole position positioning method based on mixed reality technology can communicate with other electronic devices through wired or wireless transmission to realize data transmission. The electronic device that executes the tower pole position positioning method based on mixed reality technology can be a management terminal, which uploads the collected detection data to the management terminal, and performs data sorting, classification, fast storage, reading, deletion, performance optimization, etc. in the background. Among them, the data types supported for transmission include vector data, pictures, video transmission, etc., to realize the background intelligent management of on-site photos and video content based on spatial coordinates.

[0060] It should be noted that the above wireless connection methods include but are not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB connection and other wireless connection methods currently known or to be developed in the future. The above management terminals include but are not limited to various electronic devices with display screens and supporting web browsing, such electronic devices include but are not limited to smart phones, tablet computers, e-book readers, laptop portable computers and desktop computers, servers, etc.

[0061] Among them, the pile-fixing device can be a pile positioner for fixing the tower pole, and the multiple types of acquisition equipment include image acquisition module, IMU (Inertial Measurement Unit) module and GNSS (Global Navigation Satellite System) module, wherein the GNSS module can be an RTK (Real Time Kinematic) device using GNSS, which can perform real-time dynamic measurement. The image acquisition module, IMU module and GNSS module can be arranged or installed in the pile-fixing device, so the image data collected by the image acquisition module within the acquisition range, the IMU data collected by the IMU module and the GNSS data collected by the GNSS module can be used as the detection data of the current pile-fixing device, and the timestamps of all data collection of each type are recorded at the same time. Serial port data output is supported in the image acquisition module, IMU module and GNSS module, so each serial port can be directly connected to the wireless transmission module, and the collected detection data can be sent to the management terminal based on the wireless transmission module.

[0062] Specifically, the timestamp of image data, the timestamp of IMU data and the timestamp of GNSS data can be used to time synchronize the image data, IMU data and GNSS data to ensure that all data are aligned and all data are time synchronized, which is conducive to reducing errors and improving the accuracy of data analysis.

[0063] More specifically, a visual inertial odometer positioning model can be constructed based on visual feature point matching and IMU pre-integration. In this embodiment, historical IMU data and historical image data can be used as sample data, and the model training is performed by pre-integrating the historical IMU data and matching the feature points of the image data, so as to construct a visual inertial odometer (Visual-Inertial Odometry, VIO) positioning model. Among them, VIO is a navigation and positioning technology that combines visual and inertial sensors, and estimates the position, speed, posture and other information of the staking device in three-dimensional space by analyzing the image sequence and fusing the inertial measurement data. Based on the constructed VIO positioning model, the image data of the staking device after time synchronization and the IMU data can be input into the VIO staking model, the image data of the staking device is matched with feature points, and the IMU data of the staking device is pre-integrated to obtain the position measurement results based on the two data, and the two position measurement results are fused, and the first positioning of the image acquisition module is calculated by the VIO staking model. The first positioning of the image acquisition module is also the first positioning of the staking device, which can also be called the initial staking position.

[0064] More specifically, since the GNSS module can accurately collect GNSS data and the GNSS data has no global drift, after calculating the first positioning, the GNSS data collected by the GNSS module can be used to globally optimize the first positioning to achieve position correction, thereby obtaining the optimal state estimation, and finally outputting the second positioning of the image acquisition module, and using the second positioning as the target pile position of the tower pole. The target pile position is the final position of the installed tower pole.

[0065] In an embodiment of the present invention, by utilizing multiple types of sensors to respectively collect image data, IMU data and GNSS data, and performing time synchronization on the image data, IMU data and GNSS data based on timestamps, it is possible to reduce errors caused by time asynchrony and improve data accuracy; by constructing a VIO positioning model, data fusion is performed on the time-synchronized image data and IMU data based on the VIO positioning model to determine the first positioning, which not only utilizes the high precision and strong robustness of visual acquisition, but also uses the IMU data to correct and compensate for the position drift determined based on the image data, which is more conducive to maintaining good environmental adaptability and improving the accuracy of the first positioning; by utilizing the global drift-free nature of GNSS data to correct and compensate for the first positioning, it has strong adaptability, can achieve high-precision positioning in various scenarios, and obtain a more accurate target position.

[0066] After the above step S1, the method further comprises the steps of:

[0067] S5, calibrating the image acquisition module, IMU module and GNSS module in the pile fixing device respectively, and obtaining the calibration parameters of the image acquisition module, IMU module and GNSS module. Step S5 specifically includes:

[0068] S51, extracting feature points based on the sample image of the image acquisition module, matching the extracted feature points with the three-dimensional coordinate points of the sample image, and calculating the calibration parameters of the image acquisition module according to the matching results. The calibration parameters of the image acquisition module include an intrinsic parameter matrix and an extrinsic parameter matrix;

[0069] S52, performing static calibration and dynamic calibration based on the IMU sample data of the IMU module to obtain calibration parameters of the IMU module, where the calibration parameters of the IMU module include a scale factor and an offset parameter;

[0070] S53. Based on the GNSS sample data of the GNSS module, an error analysis is performed through an error model, and calibration parameters of the GNSS module are calculated according to the error analysis results. The calibration parameters of the GNSS module include multiple error parameters.

[0071] In this embodiment, in order to improve the accuracy of the collected data, parameter calibration may be performed on each image acquisition module, IMU module and GNSS module to improve the precision and accuracy of the detection data.

[0072] Specifically, the parameter calibration of the image acquisition module can be used to perform processing such as dedistortion and image noise reduction on the image data to improve the image quality. The parameter calibration of the image acquisition module can first extract feature points from the sample image (calibration plate), match the extracted feature points with the three-dimensional coordinate points of the sample image, and determine the camera's intrinsic parameter matrix and extrinsic parameter matrix based on the matching results. The exact relationship between the 3D coordinate point in the world coordinate and the projection (pixel) of the corresponding 2D coordinate point in the image captured by the camera can be determined based on the intrinsic parameter matrix and the extrinsic parameter matrix, which helps to eliminate camera distortion and improve the precision and accuracy of the image. Among them, for image denoising, denoising can be performed through algorithms such as morphological noise filter, variational method, non-local mean, block matching and three-dimensional filtering. For example, when using morphological noise filter for denoising, first select the structural element matrix larger than the noise size to remove the background noise in the image, and then perform a closed operation on the image after removing the background noise to further remove other noise on the image. For example, using the total variation TV model for denoising, by determining the energy function of the image and minimizing the energy function, the image is made smooth, thereby achieving the purpose of image denoising. Of course, in addition to dedistorting and denoising the image, it can also include graying, geometric transformation and image enhancement, which will not be described one by one here.

[0073] More specifically, parameter calibration of the IMU module can be used to obtain accurate scale factors and offsets, thereby improving the accuracy and reliability of IMU data. When calibrating the IMU module, it is necessary to ensure that the IMU module has been correctly installed and connected to the data acquisition system, which includes a large number of data samples. These data samples cover the different motion states and postures of the IMU module, so as to fully reflect its performance characteristics and ensure that the IMU device is sufficiently stimulated to stimulate the dynamic range on all axes. The IMU device can obtain some data samples from the data acquisition system, and then perform static calibration based on the data samples when the IMU module is stationary. By recording the IMU data in the stationary state within a preset time period, the zero bias value (offset parameter) of the IMU on each axis can be calculated. At the same time, the scale factor of the IMU module can be estimated based on the known static acceleration and angular velocity information. At the same time, dynamic calibration is performed when the IMU module is in motion. By recording the IMU data of the IMU module under different motion trajectories and accelerations, the offset parameters and scale factors obtained by static calibration can be further optimized using kinematic models or machine learning algorithms, which can improve the accuracy and stability of the IMU module parameters. The collected IMU data can be adjusted based on the optimized data, thereby improving the positioning accuracy of the target pile position. Among them, the method for optimizing the offset parameters and scale factors is not limited to the least squares method, maximum likelihood estimation, etc.

[0074] More specifically, when calibrating the parameters of the GNSS module, it is also necessary to first obtain GNSS sample data from a known GNSS database. The GNSS sample data includes the location information of the antenna and receiver on the GNSS module. Then the GNSS sample data is processed, and the error source and size of the GNSS module are determined by the error model to provide a basis for subsequent calibration. Among them, the error model includes a system error model and a random error model. The system error model includes a multipath effect model, an atmospheric delay model, a clock error model, etc., and the random error model includes an observation noise model, etc. The key parameters of GNSS calibration can be calculated through error analysis. The key parameters include various error parameters, such as system error, hardware error, etc. The actual collected GNSS data can be corrected based on the error parameters. By calibrating the parameters of the GNSS module, the collected GNSS data can be corrected according to the calculated error parameters to eliminate / reduce the positioning deviation caused by various error sources, thereby improving the accuracy and reliability of GNSS module positioning.

[0075] In this embodiment, by calibrating the parameters of the image acquisition module and calculating the intrinsic parameter matrix and the extrinsic parameter matrix of the image acquisition module, distortion can be eliminated and denoising can be performed, which is beneficial to improving the precision and accuracy of the image; by calibrating the parameters of the IMU module, the precision and accuracy of the IMU data can be improved by adjusting the collected IMU data according to the scale factor and offset parameters obtained by the calibration; by calibrating the parameters of the GNSS module, the precision and accuracy of the GNSS data can be improved by adjusting the GNSS data based on the calculated error parameters.

[0076] Optionally, the above step S1 specifically includes:

[0077] S11, collecting image data of the pile fixing device through the image acquisition module and recording the timestamp of the collected image data;

[0078] S12, collecting IMU data of the staking device through the IMU module and recording the timestamp of collecting the IMU data;

[0079] S13, collecting GNSS data of the staking device through the GNSS module and recording the timestamp of collecting the GNSS data.

[0080] Among them, the image acquisition module may include industrial cameras, high-speed cameras, video cameras, etc. that can collect images and videos. The camera can collect images of the environment where the pile position is roughly located from multiple directions and angles within the acquisition range, and use the collected images as image data of the pile fixing device, and record the timestamp of the image data. The acceleration and angular velocity of the IMU module within a preset time period are collected through the IMU module, and the timestamp of the collected acceleration and angular velocity is recorded. In addition, the signal from the satellite is received through the GNSS module, and the three-dimensional position of the receiver on the GNSS module is calculated using the principle of triangulation. The three-dimensional position includes longitude, latitude and altitude, wherein the three-dimensional position represents the current position of the receiver of the GNSS module in the global coordinate system. The speed of the receiver is calculated based on the continuous position data, and the speed reflects the moving speed of the receiver relative to the ground. In this embodiment, the moving speed of the pile fixing device can be represented. The timestamps of the collected position and speed are recorded at the same time.

[0081] Among them, the various types of data collected can be obtained from the image acquisition module, IMU module and GNSS module based on the wireless transmission module, and the acquired detection data can be stored in the database based on different function classifications, for example: image data exists in the Image() function, IMU data is stored in the IMUData() function, and GNSS data is stored in the GNSSData() function. In addition, each detection data contains a timestamp attribute to record the timestamp.

[0082] In this embodiment, the image data of the environment where the staking equipment is located is collected by the camera and applied to the positioning. At the same time, the IMU module and the RTK device are used to collect IMU data and GNSS data respectively. The high precision and strong robustness of vision and the global drift-free nature of GNSS are utilized, and the three types of detection data are combined for data fusion, which can improve the detection accuracy and achieve high-precision positioning in various scenarios.

[0083] Optionally, the above step S2 specifically includes:

[0084] S21, selecting a reference timestamp from the timestamps of the image data, the IMU data, and the GNSS data;

[0085] S22. Based on the timestamp alignment algorithm, align other timestamps with the reference timestamp to control the time synchronization of image data, IMU data and GNSS data.

[0086] Specifically, in order to time synchronize image data, IMU data and GNSS data, the timestamp of any one of the image data, IMU data and GNSS data can be used as a reference timestamp, and then the timestamps of other types of data can be aligned with the reference timestamp to ensure the time synchronization of the three types of data: image data, IMU data and GNSS data. As a possible implementation method, since the GNSS system provides a global unified time standard, the timestamp of GNSS data can be used as a reference timestamp, and the timestamp of image data and the timestamp of IMU data can be aligned with the reference timestamp of GNSS respectively. Among them, the algorithms for timestamp alignment include interpolation, extrapolation or using the correlation between various types of data to assist in alignment.

[0087] In this embodiment, time synchronization is performed on image data, IMU data, and GNSS data based on timestamps, which can reduce errors caused by time asynchrony and improve the accuracy of detection data.

[0088] Optionally, the above step S3 specifically includes:

[0089] S31. Construct a visual inertial odometer positioning model based on image training samples and IMU training samples;

[0090] S32, extracting key features of the image data through a feature extraction algorithm of the visual inertial odometer positioning model, performing feature point matching according to the key features, and obtaining a matching relationship between the key features;

[0091] S33, performing motion estimation based on the matching relationship of each key feature, acquiring motion information of the image acquisition module, and calculating a first relative displacement of the image acquisition module based on the motion information;

[0092] S34, performing pre-integration based on the IMU data to obtain a second relative displacement of the IMU module;

[0093] S35 , performing drift compensation on the first relative displacement using the second relative displacement to obtain a first positioning of the image acquisition module.

[0094] Specifically, feature extraction and feature matching can be performed in advance based on image training samples, and pre-integration processing can be performed based on IMU training samples. The result of the pre-integration processing is used for data fusion training with the result of the feature matching output to construct a VIO positioning model. The optimal VIO positioning model obtained through training can extract features from the image data of the current pile fixing device and match feature points to obtain the matching relationship of key features. Among them, feature points include points in the image that have obvious differences and stability, such as corner points, edge points or spots. Among them, feature extraction algorithms include but are not limited to Harris corner detection algorithm, SIFT (Scale-Invariant Feature Transform) algorithm, ORB (Oriented FAST and Rotated BRIEF) algorithm, SURF algorithm, etc. In an embodiment of the present invention, the ORB algorithm is used for feature extraction. The ORB algorithm combines rotation invariance and scale invariance, and can strike a balance between real-time and robustness, thereby maintaining the strong robustness of the model. After the key features are extracted, feature point matching can be performed on the extracted key features based on a feature matching algorithm, wherein the feature matching algorithm includes but is not limited to feature descriptor matching, nearest neighbor matching, RANSAC (Random Sample Consensus) matching, neural network-based feature matching, etc.

[0095] As a possible implementation, when feature descriptor matching is used to perform feature matching on each key feature extracted, the corresponding matching point can be found by comparing the descriptors of the feature points between consecutive image frames, and the matching point can be found by calculating the similarity between the descriptors. After the matching is successful, the matching relationship between the feature points in different frames can be established. According to the matching relationship of the feature points, the motion estimation of the feature points can be performed to obtain motion information, and the first relative displacement of the camera can be calculated according to the obtained motion information of the camera. Among them, the first relative displacement can refer to the displacement generated relative to the previous position or the displacement generated relative to the initial position. Among them, the motion estimation can be implemented in combination with methods such as optical flow method, motion vector estimation or trajectory tracking, for example: the brightness change of pixels in consecutive frames is compared by the optical flow method to estimate the movement speed and direction of the pixels, or the position change of feature points between adjacent frames is compared by motion vector estimation to obtain its motion information. Optionally, in order to improve the accuracy and stability of motion estimation, the motion information can also be smoothed and denoised by the least squares method, Kalman filter algorithm or particle filter algorithm.

[0096] More specifically, pre-integration can be performed based on the IMU data to obtain a second relative displacement of the IMU module, and the drift caused by the first relative displacement can be compensated by the second relative displacement. Pre-integration based on the IMU data includes pre-integration of acceleration and angular velocity.

[0097] It should be noted that the above-mentioned first relative displacement and second relative displacement have no size difference, but only serve to distinguish them. The first relative displacement is calculated based on image data, and the second relative displacement is calculated based on IMU data.

[0098] Optionally, the above step S34 includes:

[0099] S341, pre-integrate the acceleration to obtain the rotation angle of the IMU module;

[0100] S342, integrating the acceleration to obtain position change data of the IMU module;

[0101] S343. Determine a second relative displacement according to the posture change data and position change data of the IMU module.

[0102] Specifically, the above-mentioned pre-integration is a method for estimating the displacement, velocity and posture of the IMU between two consecutive time points, and the above-mentioned pre-integration of the IMU data can refer to pre-integrating the acceleration and angular velocity collected by the IMU module. Among them, pre-integrating the angular velocity can obtain the rotation angle change data of the IMU module in the pile fixing device relative to the reference coordinate system, that is, the above-mentioned rotation angle. The rotation angle can be represented by a rotation vector, and the rotation vector can be represented by different methods, including but not limited to Euler angles, quaternions or rotation matrices, which can be selected according to needs.

[0103] More specifically, the acceleration can be pre-integrated to obtain the speed change of the IMU module in the time period. First, the acceleration can be integrated once to obtain the speed, and the speed of the IMU module at each time point can be obtained. The speed is based on the local coordinate system of the IMU module, that is, the coordinate system of the carrier staking device where the IMU module is located. Secondly, the speed is integrated again to solve the displacement, and the position change data of the IMU module in the time period can be obtained. The above-mentioned second relative displacement can be calculated based on the position change data.

[0104] As a possible implementation, since the acceleration is integrated twice, the original noise in the acceleration will also be accumulated, which will further affect the accuracy of calculating the second relative displacement. Therefore, after the acceleration is obtained, the acceleration can be subjected to noise reduction processing.

[0105] In this embodiment, a VIO positioning model is constructed by using visual feature point matching and pre-integration of IMU data, taking advantage of the high precision and robustness of vision, and drift compensation is performed on the first relative displacement calculated based on feature point matching based on the second relative displacement calculated based on IMU data, which can improve the accuracy and precision of the first positioning of the camera, thereby improving the positioning accuracy of the target stake.

[0106] Optionally, the subsidy S4 specifically includes:

[0107] S41, determining a positioning error of the image acquisition module by an error estimation algorithm based on the satellite positioning data and the first positioning;

[0108] S42: Perform difference compensation on the first positioning based on the positioning error to obtain a second positioning, and use the second positioning as the target pile position of the tower pole.

[0109] Among them, satellite positioning data is obtained through the CNSS system, and absolute position information can be extracted from the satellite positioning data. The absolute position information has high accuracy, and the absolute position information provided by the GNSS system can be used to globally optimize the first positioning to obtain the optimal state estimation. Among them, the satellite positioning data can include pseudorange and Doppler shift data. The pseudorange is the distance from the GPS observation station to the satellite obtained by GPS observation, and the Doppler shift refers to the phenomenon that the received signal frequency will change when the transmitting signal source moves relative to the receiver in the GNSS module. When the signal source is close to the receiver, the signal frequency will increase; when the signal source is far away from the receiver, the signal frequency will decrease. The pseudorange can provide information about the relative position between the receiver and the satellite in the GNSS module, and the Doppler shift can provide information about the relative motion between the receiver and the satellite in the GNSS module. By combining the relative position information and relative motion information provided by the pseudorange and Doppler shift data, more accurate and reliable navigation and positioning can be achieved, and the position of the pile-fixing equipment measured based on the satellite positioning data can be obtained.

[0110] More specifically, the position obtained based on the satellite positioning data is compared with the first positioning, and the positioning error can be calculated based on the error estimation algorithm, and the first positioning is differentially compensated based on the calculated positioning error, and finally the second positioning is obtained, and the second positioning is used as the target pile position of the tower. In this embodiment, the state estimation covariance matrix can be used as an error estimation algorithm to calculate the positioning error of the camera. The state estimation covariance matrix is ​​an n×n matrix, where n represents the dimension of the state vector, and the state estimation covariance matrix is ​​used to describe the variance and covariance of each state vector. Because the collected detection data is a plurality of groups, the first positioning calculated based on each type of detection data can be represented by an n×n matrix, and the position information represented by each element in the matrix of the first positioning and the position information calculated based on the satellite positioning data can be calculated. The variance and covariance between each position information in the matrix of the first positioning and the position information calculated based on the satellite positioning data can be calculated, and then the state estimation covariance matrix can be determined. The final positioning error can be obtained by calculating based on the state estimation covariance matrix, and finally the first positioning is differentially compensated according to the calculated positioning error, and the positioning with centimeter-level accuracy can be achieved.

[0111] In this embodiment, the absolute position information provided by the GNSS system is integrated into the visual-IMU joint positioning result, and the global drift-free property of the GNSS system is utilized to ensure that high-precision positioning can be achieved in various scenarios. At the same time, the positioning error is calculated using the state estimation covariance matrix for positioning compensation, which can make a more accurate estimate of the system state, thereby improving the stability and reliability of the system.

[0112] In an embodiment of the present invention, multiple types of sensors are used to respectively collect image data, IMU data and GNSS data, and the image data, IMU data and GNSS data are time synchronized based on the timestamp, which can reduce the error caused by time asynchrony and improve the accuracy of the data; by constructing a VIO positioning model, the image data and IMU data after time synchronization are data fused based on the VIO positioning model to determine the first positioning, which not only utilizes the high precision and strong robustness of visual acquisition, but also uses the IMU data to correct and compensate for the position drift determined based on the image data, which is more conducive to maintaining good environmental adaptability and improving the accuracy of the first positioning; by using the global drift-free property of GNSS data to correct and compensate for the first positioning, it has strong adaptability, can achieve high-precision positioning in various scenarios, and obtain a more accurate target pile position.

[0113] Embodiment 2

[0114] Please refer to Figure 2 , Figure 2: is a schematic diagram of the structure of a tower pole pile position positioning device based on mixed reality technology provided by an embodiment of the present invention. The tower pole pile position positioning device M20 based on mixed reality technology includes:

[0115] The data acquisition module M201 is used to acquire detection data of the staking device and the timestamp of each detection data through multiple types of acquisition devices, wherein the detection data includes image data, IMU data and GNSS data of the staking device;

[0116] A time synchronization module M202 is used to perform time synchronization on the image data, IMU data and GNSS data based on the timestamp of each detection data;

[0117] A position calculation module M203 is used to construct a visual inertial odometer positioning model, and calculate the first positioning of the image acquisition module according to the time-synchronized image data and the IMU data through the visual inertial odometer positioning model;

[0118] The position correction module M204 is used to perform position correction on the first positioning based on the GNSS data, determine the second positioning of the image acquisition module, and use the second positioning as the target pile position of the tower.

[0119] The tower pole pile position positioning device based on mixed reality technology provided in the embodiment of the present invention can implement each process in the tower pole pile position positioning method based on mixed reality technology and can achieve the same beneficial effects. To avoid repetition, it will not be described here.

[0120] The embodiment of the present invention also provides a computer device, please refer to Figure 3 , Figure 3 : is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention, and the computer device D30 includes: a processor D301, a memory D302, and a computer program stored in the memory D302 and executable on the processor D301. The processor D301 calls the computer program stored in the memory D302 to execute each step of the tower pile position positioning method based on mixed reality technology provided in an embodiment of the present invention, and can achieve the same technical effect, refer to the description in the above embodiment, and will not be repeated here.

[0121] It should be pointed out that those skilled in the art can understand that the electronic device in the embodiment of the present invention is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), digital signal processors (DSP), embedded devices, etc. The electronic device can be a computing device such as a desktop computer, a notebook, a handheld computer, and a cloud server. The electronic device can perform human-computer interaction through a keyboard, a mouse, a remote control, a touchpad, or a voice-controlled device.

[0122] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes and steps in the tower pile position positioning method based on mixed reality technology provided in the embodiment of the present invention are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0123] It should be noted that the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory may be an internal storage unit of an electronic device, such as a hard disk or memory of the electronic device. In other embodiments, the memory may also be an external storage device of an electronic device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Of course, the memory may also include both the internal storage unit of the electronic device and its external storage device. In this embodiment, the memory is generally used to store operating devices and various application software installed in the electronic device, such as program codes of a tower pile position positioning method based on mixed reality technology, etc. In addition, the memory may also be used to temporarily store various types of data that have been output or are to be output.

[0124] Those skilled in the art can understand that all or part of the processes in the tower pole pile position positioning method based on mixed reality technology in the above-mentioned embodiment can be completed by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of various embodiments of the tower pole pile position positioning method based on mixed reality technology. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

[0125] The above specific implementation is a preferred implementation of the tower pile position positioning method based on mixed reality technology of the present invention, and is not intended to limit the specific implementation scope of the present invention. The scope of the present invention includes but is not limited to this specific implementation. All equivalent changes made in accordance with the shape and structure of the present invention are within the protection scope of the present invention.

Claims

1. A tower pole position positioning method based on mixed reality technology, characterized in that: The following steps are involved: Acquire detection data of the staking device and timestamps of each detection data through multiple types of acquisition devices, wherein the detection data includes image data, IMU data and GNSS data of the staking device; Performing time synchronization on the image data, the IMU data and the GNSS data based on the timestamp of each of the detection data; Constructing a visual inertial odometry positioning model, and calculating a first positioning of an image acquisition module according to the time-synchronized image data and the IMU data through the visual inertial odometry positioning model; The first positioning is corrected based on the GNSS data, a second positioning of the image acquisition module is determined, and the second positioning is used as a target pile position of the tower pole.

2. The tower pole position positioning method based on mixed reality technology according to claim 1, characterized in that: The acquisition device includes an image acquisition module, an IMU module and a GNSS module. The acquisition of detection data of the staking device and the timestamp of each detection data by multiple types of acquisition devices includes: Collecting image data of the staking device by the image acquisition module and recording a timestamp of the collected image data; Collecting IMU data of the staking device through the IMU module and recording the timestamp of collecting the IMU data; The GNSS data of the staking device is collected through the GNSS module and a timestamp of collecting the GNSS data is recorded.

3. The tower pole position positioning method based on mixed reality technology according to claim 2, characterized in that: After the step of acquiring the detection data of the staking device and the timestamp of each detection data through multiple types of acquisition devices, the method further includes: The image acquisition module, IMU module and GNSS module in the staking device are calibrated respectively to obtain calibration parameters of the image acquisition module, IMU module and GNSS module.

4. The tower pole position positioning method based on mixed reality technology according to claim 3, characterized in that: The step of calibrating the image acquisition module, the IMU module and the GNSS module in the staking device respectively to obtain calibration parameters of the image acquisition module, the IMU module and the GNSS module includes: Extracting feature points based on the sample image of the image acquisition module, matching the extracted feature points with the three-dimensional coordinate points of the sample image, and calculating calibration parameters of the image acquisition module according to the matching results, wherein the calibration parameters of the image acquisition module include an intrinsic parameter matrix and an extrinsic parameter matrix; Static calibration and dynamic calibration are performed based on the IMU sample data of the IMU module to obtain calibration parameters of the IMU module, where the calibration parameters of the IMU module include a scale factor and an offset parameter; Based on the GNSS sample data of the GNSS module, an error analysis is performed through an error model, and calibration parameters of the GNSS module are calculated according to the error analysis results. The calibration parameters of the GNSS module include multiple error parameters.

5. The tower pole position positioning method based on mixed reality technology according to claim 1, characterized in that: The time synchronization of the image data, the IMU data and the GNSS data based on the timestamp of each of the detection data includes: Selecting a reference timestamp from the timestamps of the image data, IMU data and GNSS data; Based on a timestamp alignment algorithm, the other timestamps are aligned with the reference timestamp to control the time synchronization of the image data, IMU data and GNSS data.

6. The tower pole position positioning method based on mixed reality technology according to claim 1, characterized in that: The constructing of the visual inertial odometer positioning model, and calculating the first positioning of the image acquisition module according to the time-synchronized image data and the IMU data by using the visual inertial odometer positioning model, comprises: Constructing the visual inertial odometry positioning model based on image training samples and IMU training samples; Extracting key features of the image data by using a feature extraction algorithm of the visual inertial odometry positioning model, performing feature point matching according to the key features, and obtaining a matching relationship between the key features; Performing motion estimation based on the matching relationship of each of the key features, acquiring motion information of the image acquisition module, and calculating a first relative displacement of the image acquisition module based on the motion information; Perform pre-integration based on the IMU data to obtain a second relative displacement of the IMU module; The first relative displacement is drift compensated by using the second relative displacement to obtain a first positioning of the image acquisition module.

7. The tower pole position positioning method based on mixed reality technology according to claim 6, characterized in that: The IMU data includes acceleration and angular velocity of the IMU module, and the pre-integration based on the IMU data includes: Pre-integrating the acceleration to obtain the rotation angle of the IMU module; Integrate the acceleration to obtain position change data of the IMU module; The second relative displacement is determined according to the posture change data and the position change data of the IMU module.

8. The tower pole pile position positioning method based on mixed reality technology according to any one of claims 1 to 7, characterized in that: The GNSS data includes satellite positioning data of the pile fixing device, and the step of correcting the first positioning based on the GNSS data, determining the second positioning of the image acquisition module, and using the second positioning as the target pile position of the tower pole includes: Based on the satellite positioning data and the first positioning, determining the positioning error of the image acquisition module by an error estimation algorithm; The first positioning is difference compensated based on the positioning error to obtain the second positioning, and the second positioning is used as the target pile position of the tower pole.

9. The tower pile position positioning device based on mixed reality technology is characterized in that: The device includes: A data acquisition module, used for acquiring detection data of the staking device and timestamps of each detection data through multiple types of acquisition devices, wherein the detection data includes image data, IMU data and GNSS data of the staking device; A time synchronization module, used for performing time synchronization on the image data, IMU data and GNSS data based on the timestamp of each detection data; A position calculation module, used to construct a visual inertial odometer positioning model, and calculate a first position of the image acquisition module according to the time-synchronized image data and the IMU data through the visual inertial odometer positioning model; A position correction module is used to perform position correction on the first positioning based on the GNSS data, determine a second positioning of the image acquisition module, and use the second positioning as a target pile position of the tower pole.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the tower pile position positioning method based on mixed reality technology are implemented as described in any one of claims 1 to 8.

Images